diff --git a/experiments/mcp-tool-search/.gitignore b/experiments/mcp-tool-search/.gitignore new file mode 100644 index 00000000..fafa4ae4 --- /dev/null +++ b/experiments/mcp-tool-search/.gitignore @@ -0,0 +1,7 @@ +.venv/ +__pycache__/ +*.pyc +bin/codedb +catalog_index/ +.codedb/ +*.snapshot diff --git a/experiments/mcp-tool-search/GAPS.md b/experiments/mcp-tool-search/GAPS.md new file mode 100644 index 00000000..1bf85a22 --- /dev/null +++ b/experiments/mcp-tool-search/GAPS.md @@ -0,0 +1,171 @@ +# Gaps + +This experiment is a **reproduction**. Zed's tweet of 20 Oct 2026 +([2112588120558338261](https://x.com/zeddotdev/status/2112588120558338261)) +is the only primary source. The linked `t.co` asset is a midwit-meme image +(`fixtures/zed-tweet-2026-10-20.png`), not a paper, table, or fixture dump. +Replies asking for a blog, paper, or the 150 tests went unanswered at the +time of this run. + +## Zed setup that is not public + +| Item | Status | +|---|---| +| The 150 queries | Not public | +| The 536-tool catalog | Not public (which servers, which names, which schemas) | +| Gold labels / acceptable alternatives | Not public | +| The list of 25 approaches | Not public — tweet only says "decision models, LLMs, and embeddings" | +| Field weights, k1/b, verb list, boost values | Not public | +| Identifier-splitting / normalization code | Not in `zed-industries/zed` (searched BM25, tool_search, 536, weighted) | +| Per-approach scores, latency, CIs | Not public | +| Train/test split or significance tests | Not public | +| Whether IDF is per-user catalog or global | Asked in-thread; unanswered | + +The only BM25 in the Zed repo is `crates/edit_prediction_context/src/bm25_context.rs` +(edit-prediction context, June 2026). That is a different problem. + +## Catalog / query differences vs Zed + +Our catalog (`data/catalog.json`): + +- 536 tools, 51 servers +- 366 tools hand-authored from public MCP names/descriptions (GitHub official + MCP, modelcontextprotocol filesystem/memory/fetch/git/postgres/time, + Slack/Linear/Jira/Notion-style servers, Playwright, Docker, Kubernetes, + Stripe, …) +- 170 `expanded` tools generated from the same servers' resource/action + families so the size matches Zed's 536 +- 150 hand-labelled queries (`data/queries.json`) in five kinds: + verb+product, paraphrase, identifier, resource, ambiguous + +This is **easier than StackOne's published suite** (270 tools / 2,700 +queries, BM25 top-1 14%). Our descriptions almost always contain the +product token (`jira`, `slack`, `postgres`), so lexical overlap is a +strong signal. That is why `boolean_or` and plain BM25 look extremely +strong here. Do **not** treat the absolute percentages as Zed's numbers. + +Scaling distractors (`acme_*`, `northstar_*`, …) reuse create/get/list +verbs but use vendor names that never appear in the queries. They test +"does IDF get polluted?" more than "does `create` steal the rank from +`jira`?" — so BM25 does **not** fall off at 2k/5k the way StackOne/Stacklok +reported on lookalike tools. A harder distractor set (many issue-trackers +named `create_issue`) would be a better stress test; we did not have Zed's. + +## Approaches not reproduced + +Zed said they tested 25 approaches. They did not publish the list. We ran +every class they named that can be done locally, plus codedb. + +### Could not run + +| Approach | Why | +|---|---| +| LLM tool router / LLM query rewrite | No API key in this environment; Zed's prompt and model unknown | +| Live agent loop (`live_agent.py`) | Harness shipped; skipped — no OPENAI_API_KEY / ANTHROPIC_API_KEY / other common LLM keys. Command: `OPENAI_API_KEY=... python live_agent.py` | +| Trained decision model (LightGBM / classifier) | No Zed training labels; a 20KB LightGBM was mentioned only in a reply, not by Zed | +| Cross-encoder reranker (ms-marco MiniLM, bge-reranker, Jev) | Not installed; would need a second model download and a candidate generator we do not know they used | +| API embedding (`text-embedding-3-small`, Voyage, Jina, …) | No provider key | +| Second local dense model (gte-small, nomic-embed) | Only `BAAI/bge-small-en-v1.5` via fastembed was run, to keep the one-command path small | +| Zed's exact weighted-BM25 binary | Weights and tokenizer unpublished | +| Orama / SQLite FTS5 / ripgrep-as-Zed-used-it | Implementation details unknown; we used in-process Python BM25/TF-IDF instead | + +### Ran as stand-ins (not claimed to be Zed's) + +| Our name | What it stands in for | +|---|---| +| `bm25_weighted` | Zed "weighted BM25 + normalization": identifier split, verb down-weight 0.25, field weights name 3.0 / server 2.2 / description 1.0 / params 0.45, +1.35 namespace boost if the query names the server | +| `decision_heuristic` | Decision model: identifier-shaped query → name match, else weighted BM25 | +| `query_rewrite_weighted_bm25` | LLM rewrite: a 6-entry synonym table | +| `lexical_rerank_weighted_top20` | Neural reranker: overlap rerank of the weighted-BM25 top 20 | +| `hashed_ngram_embed` | Extra local embedding if fastembed is missing (not a neural model) | +| `char_trigram` | In-process analog of codedb's trigram index | + +## codedb primitives vs tool search + +codedb is a **code-intelligence** engine. We indexed one Python file per +tool (`server__name.py` with a `def name(...):` and a docstring). That is +off-label. + +| Primitive | What it did here | Why it is a poor fit for MCP tool search | +|---|---|---| +| `codedb_search` (multi-word → `searchContentRanked` BM25+) | **91.3% R@1** — competitive with plain BM25 | Ranking includes code-graph centrality / test-file priors that do nothing on a flat catalog | +| `codedb_trigram_tokens` | 60.7% R@1 | Single-token trigram path; fusing tokens loses phrase/product context | +| `codedb_word` | 67.3% R@1 | Exact identifier postings; NL queries are not identifiers | +| `codedb_symbol` | 30.7% R@1 | Looks up a guessed symbol; `create`/`issue` collide across servers | +| `codedb_find` | 48.7% R@1 | Fuzzy **filename** search, not content | + +codedb already has identifier splitting and BM25+ (`explore.zig` +`searchContentRanked`). It does **not** have: + +1. Fielded BM25 (name vs description vs params vs server) +2. Verb down-weighting +3. Namespace / server-alias boosts +4. A first-class "document is a tool" schema (MCP name, JSON-schema params) +5. A tool-search MCP surface (by design: codedb does not edit, and it is + not a tool router) + +### What codedb would need to win at tool search + +If codedb wanted to be the thing Zed shipped: + +1. **Index tools as structured docs**, not source files: `{server, name, description, params}`. +2. **Field weights + verb down-weight + ident split** — the three rules + everyone (Zed tweet, StackOne, thread replies) agrees matter. codedb + already splits identifiers for code; it does not down-weight `create`. +3. **Keep the trigram/word indexes** as a cheap first-stage or hybrid + (RRF with BM25). On this set, `codedb_search` already matches plain + BM25; `hybrid_codedb_weighted_bm25` is the headline winner (R@1 93.3%, + tied with BM25+bge, **no model**). +4. **Do not route NL tool queries through `codedb_symbol` / `codedb_word`.** + Those are the right tools for code, the wrong tools for "create a Jira + ticket". +5. Optional: a small local embedding only as a hybrid, not a default. + bge-small won some points here (+2 R@1 vs plain BM25) but the CI + overlaps, it needs a model, and it is 15× slower than in-process BM25. + +## Measurement caveats + +- codedb query latency includes MCP JSON-RPC in a warm process (~10 ms + p50 for `codedb_search`). In-process Python BM25 is ~0.14 ms. Do not + compare those as algorithm time. +- `hybrid_codedb_*` p50 ≈ 0.01 ms is **RRF fusion only** on precomputed + rankings, not end-to-end. +- `embed_bge_small` index time (~2 s) is first-build of 536 short docs + after the ONNX weights were already cached. +- Bootstrap CIs (2000 resamples) on 150 queries: a 4–5 point R@1 gap is + often inside the interval (`bm25_plain` 86–95%, `bm25_weighted` 79–91%). +- No Zed number can be confirmed or refuted from this run. + +## Agent cost model caveats + +Recall@k is not the agent metric. The follow-up measures search-call +count and tokens burned to surface the right tool schema. + +| Item | Status | +|---|---| +| Offline paging | Deterministic offset slices of the **saved top-10 ranking**. Call *n* returns ranks `[(n-1)k : nk]` as compact MCP tool schemas (name, description, synthesized `inputSchema`, `_meta.server`). | +| Miss tail | Tools not in the saved top-10 are appended in **catalog order**. That is pessimistic: a real re-search with a rewritten query could do better; this harness does not re-rank. | +| Always-found | Because the tail includes every catalog id, gold is always found. `tokens_per_successful_query` equals mean tokens. A ranker that misses the top-10 still "succeeds" after many pages. | +| Tokeniser | `tiktoken` `o200k_base` over the compact JSON payload only. No chat-template, no system prompt, no tool-call envelope. Live-loop tokens (when a key exists) come from the API usage fields and *do* include those wrappers. | +| Input schemas | Synthesized from the catalog `params` list (`type: string` + a short description). Zed's unpublished schemas may be larger or smaller. | +| Mean vs p95 | A few deep-tail misses pull the mean above p95. p95 calls = 1.0 on most good rankers because 95%+ finish on page 1. | +| Live agent | **Not run.** No `OPENAI_API_KEY` / `ANTHROPIC_API_KEY` / other common LLM keys in this environment. `live_agent.py` is ready: `OPENAI_API_KEY=... python live_agent.py`. | +| Live rewrite | If the model rewrites the query, the harness still pages the **original** saved ranking (documented limitation; keeps the loop cheap and deterministic). A production loop would re-run the chosen index. | +| All-schemas baseline | 536 compact schemas, one JSON array. 38,003 o200k tokens / 170,223 bytes. Real MCP `tools/list` payloads with vendor JSON Schema are often larger. | + +Cost ranking ≠ recall ranking. `boolean_or` is first on R@1 and 19th on +tokens/success at k=5: first-page hit rate is already 99.3% for several +methods, so leftover cost is payload fat plus the rare catalog-tail miss. +Weighted / RRF methods that park gold at rank 2–5 look worse on R@1 and +cheaper to an agent that reads a page of five. + +## Sources checked + +- Tweet + photo + 50 replies (no official follow-up with numbers) +- https://zed.dev/blog (no MCP tool-search post as of 21 Oct 2026) +- https://delta.dev/blog (none) +- `zed-industries/zed` code search: BM25, tool_search, ToolSearch, 536 +- StackOne's public BM25/TF-IDF write-up (different catalog; used only + as a known 20/80 hybrid formula) +- FastMCP `BM25SearchTransform` (generic Okapi; not Zed) +- GitHub official MCP `AllTools` list and public MCP server schemas diff --git a/experiments/mcp-tool-search/README.md b/experiments/mcp-tool-search/README.md new file mode 100644 index 00000000..b799d895 --- /dev/null +++ b/experiments/mcp-tool-search/README.md @@ -0,0 +1,91 @@ +# experiments/mcp-tool-search + +Reproduction of [Zed's 20 Oct 2026 MCP tool-search tweet](https://x.com/zeddotdev/status/2112588120558338261). + +**This does not change codedb's shipped behaviour.** Everything lives under this directory. + +## Status: reproduction, not Zed's fixtures + +Zed said they tested 25 approaches (decision models, LLMs, embeddings) on +150 queries against 536 tools and shipped weighted BM25 with a few +normalization rules. Their tweet links a meme, not a write-up. The Zed +blog has no post. `zed-industries/zed` has no public tool-search +benchmark (the only BM25 code is an edit-prediction retriever). + +This harness builds the closest faithful public reproduction: + +- 536-tool catalog from real public MCP servers (GitHub, filesystem, + memory, postgres, Slack, Linear, Jira, Notion, Playwright, Docker, + Kubernetes, Stripe, …) plus expanded tools from those families +- 150 hand-labelled queries +- Local implementations of the approaches that can be run without Zed's + unpublished weights or an LLM API + +See [`GAPS.md`](GAPS.md) for everything that is not public or was not run. + +## Run + +```bash +./run.sh +``` + +That creates a venv, optionally installs `fastembed` (bge-small), +downloads a codedb release binary if needed, freezes the catalog, runs +the recall harness, then the offline agent-cost model (`cost_eval.py`) +and the live-agent loop (`live_agent.py`, skipped if no LLM key). + +Useful flags (passed through to `harness.py`): + +```bash +./run.sh --skip-dense +./run.sh --skip-codedb +./run.sh --skip-scaling +``` + +Cost-only (reuses saved rankings, no search rerun): + +```bash +python cost_eval.py +OPENAI_API_KEY=... python live_agent.py # skipped without a key +``` + +## Approaches + +Local / Zed-like: + +- plain BM25, identifier-split BM25, name-only / description-only BM25 +- **weighted field BM25** (name/namespace boosts, verb down-weighting, + identifier splitting) — our stand-in for what Zed shipped +- TF-IDF, StackOne-style BM25+TF-IDF (20/80) +- boolean AND/OR, name exact/substring +- character trigram cosine +- hashed n-gram embedding (fallback, not neural) +- RRF hybrids (BM25↔trigram, weighted↔trigram, weighted↔embed) +- rule-based decision heuristic and tiny synonym rewrite (not an LLM) +- lexical rerank of weighted-BM25 top-20 + +If `fastembed` installs: + +- `BAAI/bge-small-en-v1.5` plus RRF hybrids with BM25 / weighted / trigram + +codedb primitives (one Python file per tool: name, description, params, server): + +- `codedb_search` (auto: ranked BM25+ on multi-word, trigram substring on one token) +- `codedb_trigram_tokens` (per-token trigram path + RRF) +- `codedb_word`, `codedb_symbol`, `codedb_find` +- hybrids: codedb + weighted BM25, codedb trigram + BM25, codedb + embedding + +Not run (see GAPS.md): LLM routing, trained LightGBM/decision models, +cross-encoder rerankers, API embedding models, Zed's exact 25 list. + +## Metrics + +Recall@1/3/5, MRR@10, nDCG@10, p50/p95 query latency, index build time, +RSS where available, model/API need. Bootstrap 95% CIs (2000 resamples). +Scaling: 536, ~2k, ~5k tools by adding verb-heavy distractors. + +Agent cost (the metric that matters for a tool-using model): search-call +count and o200k_base tokens returned as full MCP schemas, paging at +k ∈ {3, 5, 10}, plus an all-536-schemas baseline. Approaches are ranked +by tokens burned per successful query. Live loop records real API usage +when `OPENAI_API_KEY` or `ANTHROPIC_API_KEY` is set. diff --git a/experiments/mcp-tool-search/cost_eval.py b/experiments/mcp-tool-search/cost_eval.py new file mode 100644 index 00000000..472ec5f8 --- /dev/null +++ b/experiments/mcp-tool-search/cost_eval.py @@ -0,0 +1,381 @@ +#!/usr/bin/env python3 +"""Compute the offline agent cost model from saved rankings (or a live run).""" + +from __future__ import annotations + +import csv +import json +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parent +sys.path.insert(0, str(ROOT)) + +from lib.catalog import load_frozen # noqa: E402 +from lib.cost import ( # noqa: E402 + PAGE_KS, + all_schemas_baseline, + query_cost, + summarize_cost, +) + +RESULTS = ROOT / "results" + + +def load_method_rankings() -> dict[str, dict]: + out = {} + for p in sorted(RESULTS.glob("*.json")): + if p.name in { + "summary.json", + "scaling.json", + "cost.json", + "cost_per_query.json", + "live_agent.json", + }: + continue + data = json.loads(p.read_text(encoding="utf-8")) + if "rows" not in data or "meta" not in data: + continue + name = data["meta"]["name"] + rankings = {r["id"]: r.get("ranking") or [] for r in data["rows"]} + out[name] = { + "path": p, + "data": data, + "rankings": rankings, + "recall@1": data.get("summary", {}).get("recall@1"), + } + return out + + +def main() -> int: + tools, queries = load_frozen() + by_id = {t["id"]: t for t in tools} + catalog_ids = [t["id"] for t in tools] + methods = load_method_rankings() + if not methods: + print("no saved rankings in results/; run harness.py first", file=sys.stderr) + return 1 + + baseline = all_schemas_baseline(tools) + print( + f"all-schemas baseline: {baseline['tokens_per_turn']} o200k tokens " + f"({baseline['n_tools']} tools)" + ) + + cost_methods = [] + for name, bundle in methods.items(): + rows = [query_cost(q, bundle["rankings"].get(q["id"], []), catalog_ids, by_id) for q in queries] + by_k = {k: summarize_cost(rows, k) for k in PAGE_KS} + rec = bundle["recall@1"] + cost_methods.append( + { + "name": name, + "recall@1": rec, + "by_k": by_k, + "rows": rows, + } + ) + + # Merge cost fields into the per-query JSON/CSV. + data = bundle["data"] + row_by_id = {r["id"]: r for r in rows} + for r in data["rows"]: + extra = row_by_id.get(r["id"]) + if not extra: + continue + for key in extra: + if key in {"id", "query"}: + continue + r[key] = extra[key] + data["cost"] = {str(k): by_k[k] for k in PAGE_KS} + bundle["path"].write_text(json.dumps(data, indent=2) + "\n", encoding="utf-8") + + csv_path = RESULTS / f"{name}.csv" + fieldnames = [ + "id", + "query", + "kind", + "hit1", + "recall@1", + "recall@3", + "recall@5", + "mrr@10", + "ndcg@10", + "hit_rank", + "k3_calls", + "k3_tokens", + "k3_success1", + "k5_calls", + "k5_tokens", + "k5_success1", + "k10_calls", + "k10_tokens", + "k10_success1", + "ranking", + ] + with csv_path.open("w", newline="", encoding="utf-8") as f: + w = csv.DictWriter(f, fieldnames=fieldnames, extrasaction="ignore") + w.writeheader() + for r in data["rows"]: + w.writerow( + { + **r, + "mrr@10": f"{float(r.get('mrr@10') or 0):.4f}", + "ndcg@10": f"{float(r.get('ndcg@10') or 0):.4f}", + "ranking": " ".join(r.get("ranking") or []), + } + ) + k5 = by_k[5] + print( + f"{name:32s} k=5 calls={k5['mean_calls']:.2f} (p95 {k5['p95_calls']:.1f}) " + f"tok/q={k5['mean_tokens']:.0f} success@1={k5['success_within_1']:.3f}" + ) + + # Rank by tokens per successful query at k=5 (FastMCP-style page size). + ranked = sorted( + cost_methods, + key=lambda m: (m["by_k"][5]["tokens_per_successful_query"], -(m["recall@1"] or 0)), + ) + recall_order = sorted( + [m for m in cost_methods if m["recall@1"] is not None], + key=lambda m: -(m["recall@1"] or 0), + ) + recall_rank = {m["name"]: i + 1 for i, m in enumerate(recall_order)} + + ranking_table = [] + for i, m in enumerate(ranked, start=1): + ranking_table.append( + { + "cost_rank": i, + "recall_rank": recall_rank.get(m["name"]), + "name": m["name"], + "recall@1": m["recall@1"], + "k5_tokens_per_success": m["by_k"][5]["tokens_per_successful_query"], + "k5_mean_calls": m["by_k"][5]["mean_calls"], + "k5_success_within_1": m["by_k"][5]["success_within_1"], + "rank_diff": (None if recall_rank.get(m["name"]) is None else recall_rank[m["name"]] - i), + } + ) + + payload = { + "encoding": "o200k_base", + "page_ks": list(PAGE_KS), + "n_queries": len(queries), + "n_tools": len(tools), + "all_schemas_baseline": baseline, + "methods": [ + { + "name": m["name"], + "recall@1": m["recall@1"], + "by_k": m["by_k"], + } + for m in cost_methods + ], + "ranking_by_tokens_per_success_k5": ranking_table, + "model": ( + "Deterministic offset paging of the saved ranking. Call n returns " + "ranks[(n-1)k : n k] as full MCP tool schemas. Stops when any gold/alt " + "appears. Tools not in the saved top-10 are appended in catalog order " + "(pessimistic tail for misses)." + ), + } + (RESULTS / "cost.json").write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8") + + with (RESULTS / "cost.csv").open("w", newline="", encoding="utf-8") as f: + w = csv.DictWriter( + f, + fieldnames=[ + "name", + "recall@1", + "k", + "mean_calls", + "p95_calls", + "success_within_1", + "success_within_2", + "success_within_3", + "mean_tokens", + "p95_tokens", + "total_tokens", + "tokens_per_successful_query", + ], + ) + w.writeheader() + for m in cost_methods: + for k in PAGE_KS: + s = m["by_k"][k] + w.writerow( + { + "name": m["name"], + "recall@1": "" if m["recall@1"] is None else f"{m['recall@1']:.4f}", + "k": k, + "mean_calls": f"{s['mean_calls']:.4f}", + "p95_calls": f"{s['p95_calls']:.4f}", + "success_within_1": f"{s['success_within_1']:.4f}", + "success_within_2": f"{s['success_within_2']:.4f}", + "success_within_3": f"{s['success_within_3']:.4f}", + "mean_tokens": f"{s['mean_tokens']:.1f}", + "p95_tokens": f"{s['p95_tokens']:.1f}", + "total_tokens": s["total_tokens"], + "tokens_per_successful_query": f"{s['tokens_per_successful_query']:.1f}", + } + ) + + with (RESULTS / "cost_per_query.csv").open("w", newline="", encoding="utf-8") as f: + fields = ["method", "id", "query", "hit_rank"] + for k in PAGE_KS: + fields += [f"k{k}_calls", f"k{k}_tokens", f"k{k}_success1", f"k{k}_success2", f"k{k}_success3"] + w = csv.DictWriter(f, fieldnames=fields) + w.writeheader() + for m in cost_methods: + for r in m["rows"]: + w.writerow({"method": m["name"], **{k: r[k] for k in fields if k != "method"}}) + + (RESULTS / "cost_per_query.json").write_text( + json.dumps({m["name"]: m["rows"] for m in cost_methods}, indent=2) + "\n", + encoding="utf-8", + ) + section = render_cost_md(payload, ranking_table) + splice_cost_section(ROOT / "results.md", section) + splice_cost_section(RESULTS / "results.md", section) + print(f"wrote {RESULTS / 'cost.json'}") + return 0 + + +def _pct(x: float) -> str: + return f"{100.0 * x:.1f}%" + + +def _f(x: float, n: int = 2) -> str: + return f"{x:.{n}f}" + + +def render_cost_md(payload: dict, ranking_table: list[dict]) -> str: + base = payload["all_schemas_baseline"] + by = {m["name"]: m for m in payload["methods"]} + lines = [ + "## Agent cost (search calls + tokens, not recall@k)", + "", + "Recall@k is not the agent metric. An agent calls `search(query)`, gets", + "the top-k **full tool schemas** (name, description, inputSchema, as an", + "MCP `tools/list` / `search_tools` hit), and pages until the gold/alt", + "tool appears. Tokens are `tiktoken` **o200k_base** over the compact JSON", + "payload actually returned to the model.", + "", + "Paging model: offset slices of the saved ranking.", + "Call *n* returns ranks `[(n-1)k : n k]`. Tools missing from the saved", + "top-10 are appended in catalog order (pessimistic tail).", + "", + f"**All-schemas baseline:** `{base['tokens_per_turn']:,}` o200k tokens", + f"per turn to load all {base['n_tools']} tool schemas up front", + f"({base['payload_bytes']:,} bytes). No search calls.", + "", + "### Ranking by tokens burned per successful query (k=5)", + "", + "Primary page size is 5 (FastMCP-style). Sorted by mean o200k tokens", + "returned until gold appears. **This ranking is not the recall@1 ranking.**", + "29 of 30 approaches change place. `boolean_or` wins recall@1 (94.7%)", + "and lands 19th on cost: its first page is fatter, and the rare miss", + "pages deep into the catalog tail. Methods that put gold in the first", + "page (even at rank 2–5) are cheaper even when they lose rank-1.", + "", + "| Cost # | Recall # | Δ | Approach | R@1 | tok/success | mean calls | success@1 |", + "|---:|---:|---:|---|---:|---:|---:|---:|", + ] + for row in ranking_table: + delta = row["rank_diff"] + d = "—" if delta is None else f"{delta:+d}" + r1 = "" if row["recall@1"] is None else _pct(row["recall@1"]) + lines.append( + f"| {row['cost_rank']} | {row['recall_rank']} | {d} | {row['name']} | " + f"{r1} | {row['k5_tokens_per_success']:.1f} | " + f"{row['k5_mean_calls']:.2f} | {_pct(row['k5_success_within_1'])} |" + ) + + top = ranking_table[0] + top_s = by[top["name"]]["by_k"][5] + save = base["tokens_per_turn"] - top_s["mean_tokens"] + lines += [ + "", + f"Cheapest: **{top['name']}** at {top['k5_tokens_per_success']:.0f} tokens/query " + f"({save:,.0f} less than the all-schemas dump, " + f"{100.0 * save / base['tokens_per_turn']:.1f}% saved). " + f"Mean search calls {top['k5_mean_calls']:.2f}, first-page hit " + f"{_pct(top['k5_success_within_1'])}.", + "", + "Mean tokens can exceed p95: a few deep-tail misses pull the mean up", + "while 95% of queries finish in one page.", + "", + ] + + for k in PAGE_KS: + lines += [ + f"### Offline cost, k={k}", + "", + "| Approach | mean calls | p95 calls | ≤1 | ≤2 | ≤3 | mean tok | p95 tok | total tok |", + "|---|---:|---:|---:|---:|---:|---:|---:|---:|", + ] + for row in ranking_table: + s = by[row["name"]]["by_k"][k] + lines.append( + f"| {row['name']} | {_f(s['mean_calls'])} | {_f(s['p95_calls'], 1)} | " + f"{_pct(s['success_within_1'])} | {_pct(s['success_within_2'])} | " + f"{_pct(s['success_within_3'])} | {s['mean_tokens']:.0f} | " + f"{s['p95_tokens']:.0f} | {s['total_tokens']:,} |" + ) + lines.append("") + + lines += [ + "### Live agent loop", + "", + "Skipped in this environment: no LLM API key", + "(`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `AZURE_OPENAI_API_KEY`,", + "`GOOGLE_API_KEY`, `GEMINI_API_KEY`, `COHERE_API_KEY`, `TOGETHER_API_KEY`,", + "`GROQ_API_KEY`, `MISTRAL_API_KEY`, `OPENROUTER_API_KEY`).", + "", + "Harness is ready:", + "", + "```bash", + "OPENAI_API_KEY=... python live_agent.py", + "# or", + "ANTHROPIC_API_KEY=... python live_agent.py", + "```", + "", + "Default approaches: `bm25_plain`, `bm25_weighted`, `embed_bge_small`,", + "`hybrid_rrf_bm25_embed`, `codedb_search`, `char_trigram`. Records real", + "search-call counts, API input/output tokens, whether the named tool is", + "gold/alt, and wall time. Writes `results/live_agent.json`.", + "", + "Raw cost outputs: `results/cost.json`, `results/cost.csv`,", + "`results/cost_per_query.json`, `results/cost_per_query.csv`.", + "Per-approach JSON/CSV now include `hit_rank` and `k{3,5,10}_*` fields.", + "", + ] + return "\n".join(lines) + + +COST_BEGIN = "" +COST_END = "" + + +def splice_cost_section(path: Path, section: str) -> None: + block = f"{COST_BEGIN}\n{section}{COST_END}\n" + if path.exists(): + text = path.read_text(encoding="utf-8") + if COST_BEGIN in text and COST_END in text: + pre, rest = text.split(COST_BEGIN, 1) + _, post = rest.split(COST_END, 1) + path.write_text(pre + block + post.lstrip("\n"), encoding="utf-8") + return + # Insert before the recall headline table if present, else append. + needle = "## Headline table" + if needle in text: + pre, post = text.split(needle, 1) + path.write_text(pre.rstrip() + "\n\n" + block + "\n" + needle + post, encoding="utf-8") + return + path.write_text(text.rstrip() + "\n\n" + block, encoding="utf-8") + return + path.write_text(block, encoding="utf-8") + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/experiments/mcp-tool-search/data/catalog.json b/experiments/mcp-tool-search/data/catalog.json new file mode 100644 index 00000000..c8018ffa --- /dev/null +++ b/experiments/mcp-tool-search/data/catalog.json @@ -0,0 +1,8054 @@ +[ + { + "id": "github.get_me", + "server": "github", + "name": "get_me", + "description": "Get details of the authenticated GitHub user and their context.", + "params": [ + "include_auth" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.get_teams", + "server": "github", + "name": "get_teams", + "description": "Get teams for the authenticated GitHub user.", + "params": [], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.get_team_members", + "server": "github", + "name": "get_team_members", + "description": "List members of a GitHub team.", + "params": [ + "org", + "team_slug" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.search_repositories", + "server": "github", + "name": "search_repositories", + "description": "Search GitHub repositories by name, description, or topic.", + "params": [ + "query", + "perPage", + "sort" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.get_file_contents", + "server": "github", + "name": "get_file_contents", + "description": "Get file or directory contents from a GitHub repository.", + "params": [ + "owner", + "repo", + "path", + "ref" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.list_commits", + "server": "github", + "name": "list_commits", + "description": "List commits on a GitHub repository branch.", + "params": [ + "owner", + "repo", + "sha", + "page" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.search_code", + "server": "github", + "name": "search_code", + "description": "Search code across GitHub repositories.", + "params": [ + "query", + "perPage" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.search_commits", + "server": "github", + "name": "search_commits", + "description": "Search commits across GitHub.", + "params": [ + "query" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.get_commit", + "server": "github", + "name": "get_commit", + "description": "Get a single GitHub commit by SHA.", + "params": [ + "owner", + "repo", + "sha" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.get_file_blame", + "server": "github", + "name": "get_file_blame", + "description": "Get git blame for a file in a GitHub repository.", + "params": [ + "owner", + "repo", + "path" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.list_branches", + "server": "github", + "name": "list_branches", + "description": "List branches in a GitHub repository.", + "params": [ + "owner", + "repo" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.list_tags", + "server": "github", + "name": "list_tags", + "description": "List git tags in a GitHub repository.", + "params": [ + "owner", + "repo" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.get_tag", + "server": "github", + "name": "get_tag", + "description": "Get a git tag from a GitHub repository.", + "params": [ + "owner", + "repo", + "tag" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.list_releases", + "server": "github", + "name": "list_releases", + "description": "List GitHub releases for a repository.", + "params": [ + "owner", + "repo" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.get_latest_release", + "server": "github", + "name": "get_latest_release", + "description": "Get the latest GitHub release.", + "params": [ + "owner", + "repo" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.get_release_by_tag", + "server": "github", + "name": "get_release_by_tag", + "description": "Get a GitHub release by tag name.", + "params": [ + "owner", + "repo", + "tag" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.create_or_update_file", + "server": "github", + "name": "create_or_update_file", + "description": "Create or update a single file in a GitHub repository.", + "params": [ + "owner", + "repo", + "path", + "content", + "message", + "branch" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.create_repository", + "server": "github", + "name": "create_repository", + "description": "Create a new GitHub repository.", + "params": [ + "name", + "description", + "private" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.delete_repository", + "server": "github", + "name": "delete_repository", + "description": "Delete a GitHub repository.", + "params": [ + "owner", + "repo" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.fork_repository", + "server": "github", + "name": "fork_repository", + "description": "Fork a GitHub repository.", + "params": [ + "owner", + "repo", + "organization" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.create_branch", + "server": "github", + "name": "create_branch", + "description": "Create a branch in a GitHub repository.", + "params": [ + "owner", + "repo", + "branch", + "from_branch" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.push_files", + "server": "github", + "name": "push_files", + "description": "Push multiple files to a GitHub repository in one commit.", + "params": [ + "owner", + "repo", + "branch", + "files", + "message" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.delete_file", + "server": "github", + "name": "delete_file", + "description": "Delete a file from a GitHub repository.", + "params": [ + "owner", + "repo", + "path", + "message", + "branch" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.list_starred_repositories", + "server": "github", + "name": "list_starred_repositories", + "description": "List repositories starred by a GitHub user.", + "params": [ + "username" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.star_repository", + "server": "github", + "name": "star_repository", + "description": "Star a GitHub repository.", + "params": [ + "owner", + "repo" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.unstar_repository", + "server": "github", + "name": "unstar_repository", + "description": "Unstar a GitHub repository.", + "params": [ + "owner", + "repo" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.list_repository_collaborators", + "server": "github", + "name": "list_repository_collaborators", + "description": "List collaborators on a GitHub repository.", + "params": [ + "owner", + "repo" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.get_repository_tree", + "server": "github", + "name": "get_repository_tree", + "description": "Get the git tree of a GitHub repository.", + "params": [ + "owner", + "repo", + "tree_sha", + "recursive" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.create_issue", + "server": "github", + "name": "create_issue", + "description": "Create an issue in a GitHub repository.", + "params": [ + "owner", + "repo", + "title", + "body", + "labels", + "assignees" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.issue_write", + "server": "github", + "name": "issue_write", + "description": "Create or update a GitHub issue (consolidated write tool).", + "params": [ + "method", + "owner", + "repo", + "title", + "body", + "issue_number" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.issue_read", + "server": "github", + "name": "issue_read", + "description": "Read a GitHub issue by number.", + "params": [ + "method", + "owner", + "repo", + "issue_number" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.search_issues", + "server": "github", + "name": "search_issues", + "description": "Search GitHub issues.", + "params": [ + "query", + "owner", + "repo" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.list_issues", + "server": "github", + "name": "list_issues", + "description": "List issues in a GitHub repository.", + "params": [ + "owner", + "repo", + "state", + "labels" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.add_issue_comment", + "server": "github", + "name": "add_issue_comment", + "description": "Add a comment to a GitHub issue.", + "params": [ + "owner", + "repo", + "issue_number", + "body" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.update_issue_comment", + "server": "github", + "name": "update_issue_comment", + "description": "Update a comment on a GitHub issue.", + "params": [ + "owner", + "repo", + "comment_id", + "body" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.create_pull_request", + "server": "github", + "name": "create_pull_request", + "description": "Open a new pull request on GitHub.", + "params": [ + "owner", + "repo", + "title", + "head", + "base", + "body", + "draft" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.list_pull_requests", + "server": "github", + "name": "list_pull_requests", + "description": "List pull requests in a GitHub repository.", + "params": [ + "owner", + "repo", + "state" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.search_pull_requests", + "server": "github", + "name": "search_pull_requests", + "description": "Search GitHub pull requests.", + "params": [ + "query" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.pull_request_read", + "server": "github", + "name": "pull_request_read", + "description": "Read a GitHub pull request by number.", + "params": [ + "owner", + "repo", + "pull_number" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.update_pull_request", + "server": "github", + "name": "update_pull_request", + "description": "Update title, body, or state of a GitHub pull request.", + "params": [ + "owner", + "repo", + "pull_number", + "title", + "body", + "state" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.merge_pull_request", + "server": "github", + "name": "merge_pull_request", + "description": "Merge a GitHub pull request.", + "params": [ + "owner", + "repo", + "pull_number", + "merge_method" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.update_pull_request_branch", + "server": "github", + "name": "update_pull_request_branch", + "description": "Update a pull request branch with the base branch.", + "params": [ + "owner", + "repo", + "pull_number" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.pull_request_review_write", + "server": "github", + "name": "pull_request_review_write", + "description": "Create or submit a GitHub pull request review.", + "params": [ + "method", + "owner", + "repo", + "pull_number", + "body", + "event" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.add_comment_to_pending_review", + "server": "github", + "name": "add_comment_to_pending_review", + "description": "Add a comment to a pending GitHub pull request review.", + "params": [ + "owner", + "repo", + "pull_number", + "path", + "body", + "line" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.assign_copilot_to_issue", + "server": "github", + "name": "assign_copilot_to_issue", + "description": "Assign GitHub Copilot to work on an issue.", + "params": [ + "owner", + "repo", + "issue_number" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.request_copilot_review", + "server": "github", + "name": "request_copilot_review", + "description": "Request a Copilot review on a GitHub pull request.", + "params": [ + "owner", + "repo", + "pull_number" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.get_code_scanning_alert", + "server": "github", + "name": "get_code_scanning_alert", + "description": "Get a GitHub code scanning alert.", + "params": [ + "owner", + "repo", + "alert_number" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.list_code_scanning_alerts", + "server": "github", + "name": "list_code_scanning_alerts", + "description": "List GitHub code scanning alerts.", + "params": [ + "owner", + "repo", + "state" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.get_secret_scanning_alert", + "server": "github", + "name": "get_secret_scanning_alert", + "description": "Get a GitHub secret scanning alert.", + "params": [ + "owner", + "repo", + "alert_number" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.list_secret_scanning_alerts", + "server": "github", + "name": "list_secret_scanning_alerts", + "description": "List GitHub secret scanning alerts.", + "params": [ + "owner", + "repo" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.get_dependabot_alert", + "server": "github", + "name": "get_dependabot_alert", + "description": "Get a Dependabot alert.", + "params": [ + "owner", + "repo", + "alert_number" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.list_dependabot_alerts", + "server": "github", + "name": "list_dependabot_alerts", + "description": "List Dependabot alerts for a repository.", + "params": [ + "owner", + "repo" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.list_notifications", + "server": "github", + "name": "list_notifications", + "description": "List GitHub notifications for the authenticated user.", + "params": [ + "all", + "participating" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.get_notification_details", + "server": "github", + "name": "get_notification_details", + "description": "Get details of a GitHub notification thread.", + "params": [ + "thread_id" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.dismiss_notification", + "server": "github", + "name": "dismiss_notification", + "description": "Dismiss or mark a GitHub notification as done.", + "params": [ + "thread_id" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.mark_all_notifications_read", + "server": "github", + "name": "mark_all_notifications_read", + "description": "Mark all GitHub notifications as read.", + "params": [], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.list_discussions", + "server": "github", + "name": "list_discussions", + "description": "List GitHub discussions in a repository.", + "params": [ + "owner", + "repo" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.get_discussion", + "server": "github", + "name": "get_discussion", + "description": "Get a GitHub discussion.", + "params": [ + "owner", + "repo", + "discussion_number" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.get_discussion_comments", + "server": "github", + "name": "get_discussion_comments", + "description": "Get comments on a GitHub discussion.", + "params": [ + "owner", + "repo", + "discussion_number" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.actions_list", + "server": "github", + "name": "actions_list", + "description": "List GitHub Actions workflows.", + "params": [ + "owner", + "repo" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.actions_get", + "server": "github", + "name": "actions_get", + "description": "Get a GitHub Actions workflow or run.", + "params": [ + "owner", + "repo", + "workflow_id" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.actions_run_trigger", + "server": "github", + "name": "actions_run_trigger", + "description": "Trigger a GitHub Actions workflow dispatch.", + "params": [ + "owner", + "repo", + "workflow_id", + "ref", + "inputs" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.actions_get_job_logs", + "server": "github", + "name": "actions_get_job_logs", + "description": "Get logs for a GitHub Actions job.", + "params": [ + "owner", + "repo", + "job_id" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.list_gists", + "server": "github", + "name": "list_gists", + "description": "List GitHub gists for a user.", + "params": [ + "username" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.get_gist", + "server": "github", + "name": "get_gist", + "description": "Get a GitHub gist by id.", + "params": [ + "gist_id" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.create_gist", + "server": "github", + "name": "create_gist", + "description": "Create a GitHub gist.", + "params": [ + "description", + "files", + "public" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.update_gist", + "server": "github", + "name": "update_gist", + "description": "Update a GitHub gist.", + "params": [ + "gist_id", + "description", + "files" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.projects_list", + "server": "github", + "name": "projects_list", + "description": "List GitHub Projects.", + "params": [ + "owner", + "owner_type" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.projects_get", + "server": "github", + "name": "projects_get", + "description": "Get a GitHub Project.", + "params": [ + "project_number", + "owner" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.projects_write", + "server": "github", + "name": "projects_write", + "description": "Create or update a GitHub Project item.", + "params": [ + "method", + "project_id", + "title" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.get_label", + "server": "github", + "name": "get_label", + "description": "Get a GitHub label.", + "params": [ + "owner", + "repo", + "name" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.list_labels", + "server": "github", + "name": "list_labels", + "description": "List labels in a GitHub repository.", + "params": [ + "owner", + "repo" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.label_write", + "server": "github", + "name": "label_write", + "description": "Create, update, or delete a GitHub label.", + "params": [ + "method", + "owner", + "repo", + "name", + "color" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.search_users", + "server": "github", + "name": "search_users", + "description": "Search GitHub users.", + "params": [ + "query" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.search_orgs", + "server": "github", + "name": "search_orgs", + "description": "Search GitHub organizations.", + "params": [ + "query" + ], + "source": "public", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "filesystem.read_file", + "server": "filesystem", + "name": "read_file", + "description": "Read the complete contents of a file from the local filesystem.", + "params": [ + "path" + ], + "source": "public", + "aliases": [ + "filesystem", + "files", + "disk", + "folder", + "directory" + ] + }, + { + "id": "filesystem.read_text_file", + "server": "filesystem", + "name": "read_text_file", + "description": "Read a text file from the local filesystem.", + "params": [ + "path", + "head", + "tail" + ], + "source": "public", + "aliases": [ + "filesystem", + "files", + "disk", + "folder", + "directory" + ] + }, + { + "id": "filesystem.read_media_file", + "server": "filesystem", + "name": "read_media_file", + "description": "Read an image or audio file from the local filesystem.", + "params": [ + "path" + ], + "source": "public", + "aliases": [ + "filesystem", + "files", + "disk", + "folder", + "directory" + ] + }, + { + "id": "filesystem.write_file", + "server": "filesystem", + "name": "write_file", + "description": "Create a new file or completely overwrite an existing file.", + "params": [ + "path", + "content" + ], + "source": "public", + "aliases": [ + "filesystem", + "files", + "disk", + "folder", + "directory" + ] + }, + { + "id": "filesystem.edit_file", + "server": "filesystem", + "name": "edit_file", + "description": "Make a line-based edit to a text file using old and new strings.", + "params": [ + "path", + "old_string", + "new_string" + ], + "source": "public", + "aliases": [ + "filesystem", + "files", + "disk", + "folder", + "directory" + ] + }, + { + "id": "filesystem.create_directory", + "server": "filesystem", + "name": "create_directory", + "description": "Create a new directory or ensure a directory exists.", + "params": [ + "path" + ], + "source": "public", + "aliases": [ + "filesystem", + "files", + "disk", + "folder", + "directory" + ] + }, + { + "id": "filesystem.list_directory", + "server": "filesystem", + "name": "list_directory", + "description": "Get a detailed listing of files and directories.", + "params": [ + "path" + ], + "source": "public", + "aliases": [ + "filesystem", + "files", + "disk", + "folder", + "directory" + ] + }, + { + "id": "filesystem.list_directory_with_sizes", + "server": "filesystem", + "name": "list_directory_with_sizes", + "description": "Get a listing of files and directories with sizes.", + "params": [ + "path" + ], + "source": "public", + "aliases": [ + "filesystem", + "files", + "disk", + "folder", + "directory" + ] + }, + { + "id": "filesystem.directory_tree", + "server": "filesystem", + "name": "directory_tree", + "description": "Get a recursive tree view of files and directories.", + "params": [ + "path" + ], + "source": "public", + "aliases": [ + "filesystem", + "files", + "disk", + "folder", + "directory" + ] + }, + { + "id": "filesystem.move_file", + "server": "filesystem", + "name": "move_file", + "description": "Move or rename files and directories.", + "params": [ + "source", + "destination" + ], + "source": "public", + "aliases": [ + "filesystem", + "files", + "disk", + "folder", + "directory" + ] + }, + { + "id": "filesystem.search_files", + "server": "filesystem", + "name": "search_files", + "description": "Recursively search for files matching a pattern.", + "params": [ + "path", + "pattern" + ], + "source": "public", + "aliases": [ + "filesystem", + "files", + "disk", + "folder", + "directory" + ] + }, + { + "id": "filesystem.get_file_info", + "server": "filesystem", + "name": "get_file_info", + "description": "Retrieve metadata about a file or directory.", + "params": [ + "path" + ], + "source": "public", + "aliases": [ + "filesystem", + "files", + "disk", + "folder", + "directory" + ] + }, + { + "id": "filesystem.list_allowed_directories", + "server": "filesystem", + "name": "list_allowed_directories", + "description": "List directories the filesystem server is allowed to access.", + "params": [], + "source": "public", + "aliases": [ + "filesystem", + "files", + "disk", + "folder", + "directory" + ] + }, + { + "id": "memory.create_entities", + "server": "memory", + "name": "create_entities", + "description": "Create multiple entities in the knowledge graph.", + "params": [ + "entities" + ], + "source": "public", + "aliases": [ + "memory", + "knowledge graph", + "remember" + ] + }, + { + "id": "memory.create_relations", + "server": "memory", + "name": "create_relations", + "description": "Create relations between entities in the knowledge graph.", + "params": [ + "relations" + ], + "source": "public", + "aliases": [ + "memory", + "knowledge graph", + "remember" + ] + }, + { + "id": "memory.add_observations", + "server": "memory", + "name": "add_observations", + "description": "Add observations to existing entities in the knowledge graph.", + "params": [ + "observations" + ], + "source": "public", + "aliases": [ + "memory", + "knowledge graph", + "remember" + ] + }, + { + "id": "memory.delete_entities", + "server": "memory", + "name": "delete_entities", + "description": "Delete entities and their relations from the knowledge graph.", + "params": [ + "entityNames" + ], + "source": "public", + "aliases": [ + "memory", + "knowledge graph", + "remember" + ] + }, + { + "id": "memory.delete_observations", + "server": "memory", + "name": "delete_observations", + "description": "Delete specific observations from entities.", + "params": [ + "deletions" + ], + "source": "public", + "aliases": [ + "memory", + "knowledge graph", + "remember" + ] + }, + { + "id": "memory.delete_relations", + "server": "memory", + "name": "delete_relations", + "description": "Delete relations from the knowledge graph.", + "params": [ + "relations" + ], + "source": "public", + "aliases": [ + "memory", + "knowledge graph", + "remember" + ] + }, + { + "id": "memory.read_graph", + "server": "memory", + "name": "read_graph", + "description": "Read the entire knowledge graph.", + "params": [], + "source": "public", + "aliases": [ + "memory", + "knowledge graph", + "remember" + ] + }, + { + "id": "memory.search_nodes", + "server": "memory", + "name": "search_nodes", + "description": "Search for nodes in the knowledge graph based on a query.", + "params": [ + "query" + ], + "source": "public", + "aliases": [ + "memory", + "knowledge graph", + "remember" + ] + }, + { + "id": "memory.open_nodes", + "server": "memory", + "name": "open_nodes", + "description": "Open specific nodes in the knowledge graph by name.", + "params": [ + "names" + ], + "source": "public", + "aliases": [ + "memory", + "knowledge graph", + "remember" + ] + }, + { + "id": "postgres.query", + "server": "postgres", + "name": "query", + "description": "Run a read-only SQL query against PostgreSQL.", + "params": [ + "sql" + ], + "source": "public", + "aliases": [ + "postgres", + "postgresql", + "sql", + "database" + ] + }, + { + "id": "postgres.list_tables", + "server": "postgres", + "name": "list_tables", + "description": "List tables in the connected PostgreSQL database.", + "params": [ + "schema" + ], + "source": "public", + "aliases": [ + "postgres", + "postgresql", + "sql", + "database" + ] + }, + { + "id": "postgres.describe_table", + "server": "postgres", + "name": "describe_table", + "description": "Describe columns and types of a PostgreSQL table.", + "params": [ + "schema", + "table" + ], + "source": "public", + "aliases": [ + "postgres", + "postgresql", + "sql", + "database" + ] + }, + { + "id": "postgres.list_schemas", + "server": "postgres", + "name": "list_schemas", + "description": "List schemas in the PostgreSQL database.", + "params": [], + "source": "public", + "aliases": [ + "postgres", + "postgresql", + "sql", + "database" + ] + }, + { + "id": "postgres.explain_query", + "server": "postgres", + "name": "explain_query", + "description": "EXPLAIN a PostgreSQL query plan.", + "params": [ + "sql" + ], + "source": "public", + "aliases": [ + "postgres", + "postgresql", + "sql", + "database" + ] + }, + { + "id": "fetch.fetch", + "server": "fetch", + "name": "fetch", + "description": "Fetch a URL and return the contents as markdown or HTML.", + "params": [ + "url", + "max_length", + "raw" + ], + "source": "public", + "aliases": [ + "fetch", + "http", + "url", + "webpage" + ] + }, + { + "id": "git.git_status", + "server": "git", + "name": "git_status", + "description": "Show the working tree status of a git repository.", + "params": [ + "repo_path" + ], + "source": "public", + "aliases": [ + "git" + ] + }, + { + "id": "git.git_diff_unstaged", + "server": "git", + "name": "git_diff_unstaged", + "description": "Show unstaged git differences.", + "params": [ + "repo_path" + ], + "source": "public", + "aliases": [ + "git" + ] + }, + { + "id": "git.git_diff_staged", + "server": "git", + "name": "git_diff_staged", + "description": "Show staged git differences.", + "params": [ + "repo_path" + ], + "source": "public", + "aliases": [ + "git" + ] + }, + { + "id": "git.git_diff", + "server": "git", + "name": "git_diff", + "description": "Show git diff between branches or commits.", + "params": [ + "repo_path", + "target" + ], + "source": "public", + "aliases": [ + "git" + ] + }, + { + "id": "git.git_commit", + "server": "git", + "name": "git_commit", + "description": "Record staged changes to the git repository.", + "params": [ + "repo_path", + "message" + ], + "source": "public", + "aliases": [ + "git" + ] + }, + { + "id": "git.git_add", + "server": "git", + "name": "git_add", + "description": "Add file contents to the git staging area.", + "params": [ + "repo_path", + "files" + ], + "source": "public", + "aliases": [ + "git" + ] + }, + { + "id": "git.git_reset", + "server": "git", + "name": "git_reset", + "description": "Unstage all staged changes.", + "params": [ + "repo_path" + ], + "source": "public", + "aliases": [ + "git" + ] + }, + { + "id": "git.git_log", + "server": "git", + "name": "git_log", + "description": "Show commit logs.", + "params": [ + "repo_path", + "max_count" + ], + "source": "public", + "aliases": [ + "git" + ] + }, + { + "id": "git.git_create_branch", + "server": "git", + "name": "git_create_branch", + "description": "Create a new git branch.", + "params": [ + "repo_path", + "branch_name", + "base_branch" + ], + "source": "public", + "aliases": [ + "git" + ] + }, + { + "id": "git.git_checkout", + "server": "git", + "name": "git_checkout", + "description": "Switch git branches.", + "params": [ + "repo_path", + "branch_name" + ], + "source": "public", + "aliases": [ + "git" + ] + }, + { + "id": "git.git_show", + "server": "git", + "name": "git_show", + "description": "Show a git commit.", + "params": [ + "repo_path", + "revision" + ], + "source": "public", + "aliases": [ + "git" + ] + }, + { + "id": "git.git_branch", + "server": "git", + "name": "git_branch", + "description": "List git branches.", + "params": [ + "repo_path" + ], + "source": "public", + "aliases": [ + "git" + ] + }, + { + "id": "sequential_thinking.sequentialthinking", + "server": "sequential_thinking", + "name": "sequentialthinking", + "description": "A detailed tool for dynamic and reflective problem-solving through thoughts.", + "params": [ + "thought", + "nextThoughtNeeded", + "thoughtNumber", + "totalThoughts" + ], + "source": "public", + "aliases": [ + "sequential thinking", + "think step" + ] + }, + { + "id": "time.get_current_time", + "server": "time", + "name": "get_current_time", + "description": "Get the current time in a timezone.", + "params": [ + "timezone" + ], + "source": "public", + "aliases": [ + "time", + "timezone", + "clock" + ] + }, + { + "id": "time.convert_time", + "server": "time", + "name": "convert_time", + "description": "Convert time between timezones.", + "params": [ + "source_timezone", + "time", + "target_timezone" + ], + "source": "public", + "aliases": [ + "time", + "timezone", + "clock" + ] + }, + { + "id": "brave_search.brave_web_search", + "server": "brave_search", + "name": "brave_web_search", + "description": "Search the web using Brave Search.", + "params": [ + "query", + "count", + "offset" + ], + "source": "public", + "aliases": [ + "brave", + "web search" + ] + }, + { + "id": "brave_search.brave_local_search", + "server": "brave_search", + "name": "brave_local_search", + "description": "Search for local businesses and places using Brave.", + "params": [ + "query", + "count" + ], + "source": "public", + "aliases": [ + "brave", + "web search" + ] + }, + { + "id": "slack.slack_post_message", + "server": "slack", + "name": "slack_post_message", + "description": "Post a message to a Slack channel.", + "params": [ + "channel_id", + "text" + ], + "source": "public", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.slack_reply_to_thread", + "server": "slack", + "name": "slack_reply_to_thread", + "description": "Reply to a Slack thread.", + "params": [ + "channel_id", + "thread_ts", + "text" + ], + "source": "public", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.slack_add_reaction", + "server": "slack", + "name": "slack_add_reaction", + "description": "Add an emoji reaction to a Slack message.", + "params": [ + "channel_id", + "timestamp", + "emoji" + ], + "source": "public", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.slack_get_channel_history", + "server": "slack", + "name": "slack_get_channel_history", + "description": "Get recent messages from a Slack channel.", + "params": [ + "channel_id", + "limit" + ], + "source": "public", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.slack_get_thread_replies", + "server": "slack", + "name": "slack_get_thread_replies", + "description": "Get replies in a Slack thread.", + "params": [ + "channel_id", + "thread_ts" + ], + "source": "public", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.slack_list_channels", + "server": "slack", + "name": "slack_list_channels", + "description": "List Slack channels in the workspace.", + "params": [ + "limit", + "cursor" + ], + "source": "public", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.slack_get_users", + "server": "slack", + "name": "slack_get_users", + "description": "Get a list of Slack users.", + "params": [ + "cursor", + "limit" + ], + "source": "public", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.slack_get_user_profile", + "server": "slack", + "name": "slack_get_user_profile", + "description": "Get a Slack user's profile.", + "params": [ + "user_id" + ], + "source": "public", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.slack_join_channel", + "server": "slack", + "name": "slack_join_channel", + "description": "Join a Slack channel.", + "params": [ + "channel_id" + ], + "source": "public", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.slack_leave_channel", + "server": "slack", + "name": "slack_leave_channel", + "description": "Leave a Slack channel.", + "params": [ + "channel_id" + ], + "source": "public", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.slack_create_channel", + "server": "slack", + "name": "slack_create_channel", + "description": "Create a Slack channel.", + "params": [ + "name", + "is_private" + ], + "source": "public", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.slack_set_channel_topic", + "server": "slack", + "name": "slack_set_channel_topic", + "description": "Set the topic of a Slack channel.", + "params": [ + "channel_id", + "topic" + ], + "source": "public", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.slack_upload_file", + "server": "slack", + "name": "slack_upload_file", + "description": "Upload a file to Slack.", + "params": [ + "channel_id", + "filename", + "content" + ], + "source": "public", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.slack_search_messages", + "server": "slack", + "name": "slack_search_messages", + "description": "Search Slack messages.", + "params": [ + "query", + "count" + ], + "source": "public", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.slack_list_users", + "server": "slack", + "name": "slack_list_users", + "description": "List Slack workspace users.", + "params": [ + "limit", + "cursor" + ], + "source": "public", + "aliases": [ + "slack" + ] + }, + { + "id": "linear.linear_create_issue", + "server": "linear", + "name": "linear_create_issue", + "description": "Create a Linear issue in a team.", + "params": [ + "title", + "description", + "team", + "teamId", + "priority", + "labels" + ], + "source": "public", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.linear_update_issue", + "server": "linear", + "name": "linear_update_issue", + "description": "Update a Linear issue.", + "params": [ + "id", + "title", + "description", + "state", + "priority" + ], + "source": "public", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.linear_list_issues", + "server": "linear", + "name": "linear_list_issues", + "description": "List Linear issues, optionally filtered.", + "params": [ + "team", + "state", + "assignee" + ], + "source": "public", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.linear_get_issue", + "server": "linear", + "name": "linear_get_issue", + "description": "Get a Linear issue by id or identifier.", + "params": [ + "id" + ], + "source": "public", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.linear_search_issues", + "server": "linear", + "name": "linear_search_issues", + "description": "Search Linear issues by text.", + "params": [ + "query" + ], + "source": "public", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.linear_create_comment", + "server": "linear", + "name": "linear_create_comment", + "description": "Add a comment to a Linear issue.", + "params": [ + "issueId", + "body" + ], + "source": "public", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.linear_list_teams", + "server": "linear", + "name": "linear_list_teams", + "description": "List Linear teams.", + "params": [], + "source": "public", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.linear_list_projects", + "server": "linear", + "name": "linear_list_projects", + "description": "List Linear projects.", + "params": [ + "team" + ], + "source": "public", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.linear_create_project", + "server": "linear", + "name": "linear_create_project", + "description": "Create a Linear project.", + "params": [ + "name", + "team", + "description" + ], + "source": "public", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.linear_assign_issue", + "server": "linear", + "name": "linear_assign_issue", + "description": "Assign a Linear issue to a user.", + "params": [ + "id", + "assigneeId" + ], + "source": "public", + "aliases": [ + "linear" + ] + }, + { + "id": "jira.jira_create_issue", + "server": "jira", + "name": "jira_create_issue", + "description": "Create a Jira issue or ticket in a project.", + "params": [ + "project", + "key", + "summary", + "description", + "issuetype" + ], + "source": "public", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.jira_update_issue", + "server": "jira", + "name": "jira_update_issue", + "description": "Update an existing Jira issue.", + "params": [ + "issue_key", + "fields" + ], + "source": "public", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.jira_get_issue", + "server": "jira", + "name": "jira_get_issue", + "description": "Get a Jira issue by key.", + "params": [ + "issue_key" + ], + "source": "public", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.jira_search_issues", + "server": "jira", + "name": "jira_search_issues", + "description": "Search Jira issues with JQL.", + "params": [ + "jql", + "maxResults" + ], + "source": "public", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.jira_add_comment", + "server": "jira", + "name": "jira_add_comment", + "description": "Add a comment to a Jira issue.", + "params": [ + "issue_key", + "body" + ], + "source": "public", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.jira_transition_issue", + "server": "jira", + "name": "jira_transition_issue", + "description": "Transition a Jira issue to a new status.", + "params": [ + "issue_key", + "transition" + ], + "source": "public", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.jira_assign_issue", + "server": "jira", + "name": "jira_assign_issue", + "description": "Assign a Jira issue.", + "params": [ + "issue_key", + "account_id" + ], + "source": "public", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.jira_list_projects", + "server": "jira", + "name": "jira_list_projects", + "description": "List Jira projects.", + "params": [], + "source": "public", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.jira_create_sprint", + "server": "jira", + "name": "jira_create_sprint", + "description": "Create a Jira Software sprint.", + "params": [ + "board_id", + "name", + "startDate", + "endDate" + ], + "source": "public", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.jira_add_worklog", + "server": "jira", + "name": "jira_add_worklog", + "description": "Add a worklog to a Jira issue.", + "params": [ + "issue_key", + "timeSpent", + "comment" + ], + "source": "public", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "notion.notion_search", + "server": "notion", + "name": "notion_search", + "description": "Search Notion pages and databases.", + "params": [ + "query", + "filter" + ], + "source": "public", + "aliases": [ + "notion" + ] + }, + { + "id": "notion.notion_get_page", + "server": "notion", + "name": "notion_get_page", + "description": "Retrieve a Notion page.", + "params": [ + "page_id" + ], + "source": "public", + "aliases": [ + "notion" + ] + }, + { + "id": "notion.notion_create_page", + "server": "notion", + "name": "notion_create_page", + "description": "Create a Notion page in a parent page or database.", + "params": [ + "parent", + "properties", + "children" + ], + "source": "public", + "aliases": [ + "notion" + ] + }, + { + "id": "notion.notion_update_page", + "server": "notion", + "name": "notion_update_page", + "description": "Update a Notion page properties.", + "params": [ + "page_id", + "properties" + ], + "source": "public", + "aliases": [ + "notion" + ] + }, + { + "id": "notion.notion_get_block", + "server": "notion", + "name": "notion_get_block", + "description": "Get a Notion block.", + "params": [ + "block_id" + ], + "source": "public", + "aliases": [ + "notion" + ] + }, + { + "id": "notion.notion_append_block", + "server": "notion", + "name": "notion_append_block", + "description": "Append children blocks to a Notion block.", + "params": [ + "block_id", + "children" + ], + "source": "public", + "aliases": [ + "notion" + ] + }, + { + "id": "notion.notion_query_database", + "server": "notion", + "name": "notion_query_database", + "description": "Query a Notion database.", + "params": [ + "database_id", + "filter", + "sorts" + ], + "source": "public", + "aliases": [ + "notion" + ] + }, + { + "id": "notion.notion_create_database", + "server": "notion", + "name": "notion_create_database", + "description": "Create a Notion database.", + "params": [ + "parent", + "title", + "properties" + ], + "source": "public", + "aliases": [ + "notion" + ] + }, + { + "id": "notion.notion_list_users", + "server": "notion", + "name": "notion_list_users", + "description": "List Notion workspace users.", + "params": [], + "source": "public", + "aliases": [ + "notion" + ] + }, + { + "id": "notion.notion_get_comments", + "server": "notion", + "name": "notion_get_comments", + "description": "Get comments on a Notion page.", + "params": [ + "block_id" + ], + "source": "public", + "aliases": [ + "notion" + ] + }, + { + "id": "playwright.browser_navigate", + "server": "playwright", + "name": "browser_navigate", + "description": "Navigate the browser to a URL.", + "params": [ + "url" + ], + "source": "public", + "aliases": [ + "playwright", + "browser" + ] + }, + { + "id": "playwright.browser_snapshot", + "server": "playwright", + "name": "browser_snapshot", + "description": "Capture an accessibility snapshot of the current page.", + "params": [], + "source": "public", + "aliases": [ + "playwright", + "browser" + ] + }, + { + "id": "playwright.browser_click", + "server": "playwright", + "name": "browser_click", + "description": "Click an element on the page.", + "params": [ + "element", + "ref" + ], + "source": "public", + "aliases": [ + "playwright", + "browser" + ] + }, + { + "id": "playwright.browser_type", + "server": "playwright", + "name": "browser_type", + "description": "Type text into an editable element.", + "params": [ + "element", + "ref", + "text" + ], + "source": "public", + "aliases": [ + "playwright", + "browser" + ] + }, + { + "id": "playwright.browser_fill_form", + "server": "playwright", + "name": "browser_fill_form", + "description": "Fill multiple form fields.", + "params": [ + "fields" + ], + "source": "public", + "aliases": [ + "playwright", + "browser" + ] + }, + { + "id": "playwright.browser_take_screenshot", + "server": "playwright", + "name": "browser_take_screenshot", + "description": "Take a screenshot of the page or an element.", + "params": [ + "filename", + "fullPage" + ], + "source": "public", + "aliases": [ + "playwright", + "browser" + ] + }, + { + "id": "playwright.browser_evaluate", + "server": "playwright", + "name": "browser_evaluate", + "description": "Evaluate JavaScript on the page.", + "params": [ + "function" + ], + "source": "public", + "aliases": [ + "playwright", + "browser" + ] + }, + { + "id": "playwright.browser_tab_list", + "server": "playwright", + "name": "browser_tab_list", + "description": "List open browser tabs.", + "params": [], + "source": "public", + "aliases": [ + "playwright", + "browser" + ] + }, + { + "id": "playwright.browser_tab_new", + "server": "playwright", + "name": "browser_tab_new", + "description": "Open a new browser tab.", + "params": [ + "url" + ], + "source": "public", + "aliases": [ + "playwright", + "browser" + ] + }, + { + "id": "playwright.browser_tab_close", + "server": "playwright", + "name": "browser_tab_close", + "description": "Close a browser tab.", + "params": [ + "index" + ], + "source": "public", + "aliases": [ + "playwright", + "browser" + ] + }, + { + "id": "playwright.browser_wait_for", + "server": "playwright", + "name": "browser_wait_for", + "description": "Wait for text or time to pass.", + "params": [ + "text", + "time" + ], + "source": "public", + "aliases": [ + "playwright", + "browser" + ] + }, + { + "id": "playwright.browser_press_key", + "server": "playwright", + "name": "browser_press_key", + "description": "Press a key on the page.", + "params": [ + "key" + ], + "source": "public", + "aliases": [ + "playwright", + "browser" + ] + }, + { + "id": "puppeteer.puppeteer_navigate", + "server": "puppeteer", + "name": "puppeteer_navigate", + "description": "Navigate to a URL with Puppeteer.", + "params": [ + "url" + ], + "source": "public", + "aliases": [ + "puppeteer" + ] + }, + { + "id": "puppeteer.puppeteer_screenshot", + "server": "puppeteer", + "name": "puppeteer_screenshot", + "description": "Take a screenshot with Puppeteer.", + "params": [ + "name", + "selector" + ], + "source": "public", + "aliases": [ + "puppeteer" + ] + }, + { + "id": "puppeteer.puppeteer_click", + "server": "puppeteer", + "name": "puppeteer_click", + "description": "Click an element with Puppeteer.", + "params": [ + "selector" + ], + "source": "public", + "aliases": [ + "puppeteer" + ] + }, + { + "id": "puppeteer.puppeteer_fill", + "server": "puppeteer", + "name": "puppeteer_fill", + "description": "Fill an input with Puppeteer.", + "params": [ + "selector", + "value" + ], + "source": "public", + "aliases": [ + "puppeteer" + ] + }, + { + "id": "puppeteer.puppeteer_evaluate", + "server": "puppeteer", + "name": "puppeteer_evaluate", + "description": "Execute JavaScript with Puppeteer.", + "params": [ + "script" + ], + "source": "public", + "aliases": [ + "puppeteer" + ] + }, + { + "id": "docker.docker_ps", + "server": "docker", + "name": "docker_ps", + "description": "List running Docker containers.", + "params": [ + "all" + ], + "source": "public", + "aliases": [ + "docker", + "container" + ] + }, + { + "id": "docker.docker_logs", + "server": "docker", + "name": "docker_logs", + "description": "Fetch logs from a Docker container.", + "params": [ + "container", + "tail" + ], + "source": "public", + "aliases": [ + "docker", + "container" + ] + }, + { + "id": "docker.docker_run", + "server": "docker", + "name": "docker_run", + "description": "Run a Docker container.", + "params": [ + "image", + "name", + "ports", + "env" + ], + "source": "public", + "aliases": [ + "docker", + "container" + ] + }, + { + "id": "docker.docker_stop", + "server": "docker", + "name": "docker_stop", + "description": "Stop a Docker container.", + "params": [ + "container" + ], + "source": "public", + "aliases": [ + "docker", + "container" + ] + }, + { + "id": "docker.docker_rm", + "server": "docker", + "name": "docker_rm", + "description": "Remove a Docker container.", + "params": [ + "container", + "force" + ], + "source": "public", + "aliases": [ + "docker", + "container" + ] + }, + { + "id": "docker.docker_images", + "server": "docker", + "name": "docker_images", + "description": "List Docker images.", + "params": [], + "source": "public", + "aliases": [ + "docker", + "container" + ] + }, + { + "id": "docker.docker_build", + "server": "docker", + "name": "docker_build", + "description": "Build a Docker image.", + "params": [ + "path", + "tag" + ], + "source": "public", + "aliases": [ + "docker", + "container" + ] + }, + { + "id": "docker.docker_exec", + "server": "docker", + "name": "docker_exec", + "description": "Run a command in a running Docker container.", + "params": [ + "container", + "command" + ], + "source": "public", + "aliases": [ + "docker", + "container" + ] + }, + { + "id": "kubernetes.k8s_list_pods", + "server": "kubernetes", + "name": "k8s_list_pods", + "description": "List Kubernetes pods in a namespace.", + "params": [ + "namespace" + ], + "source": "public", + "aliases": [ + "kubernetes", + "k8s" + ] + }, + { + "id": "kubernetes.k8s_get_pod", + "server": "kubernetes", + "name": "k8s_get_pod", + "description": "Get a Kubernetes pod.", + "params": [ + "namespace", + "name" + ], + "source": "public", + "aliases": [ + "kubernetes", + "k8s" + ] + }, + { + "id": "kubernetes.k8s_pod_logs", + "server": "kubernetes", + "name": "k8s_pod_logs", + "description": "Get logs from a Kubernetes pod.", + "params": [ + "namespace", + "name", + "container" + ], + "source": "public", + "aliases": [ + "kubernetes", + "k8s" + ] + }, + { + "id": "kubernetes.k8s_apply", + "server": "kubernetes", + "name": "k8s_apply", + "description": "Apply a Kubernetes manifest.", + "params": [ + "manifest", + "namespace" + ], + "source": "public", + "aliases": [ + "kubernetes", + "k8s" + ] + }, + { + "id": "kubernetes.k8s_delete", + "server": "kubernetes", + "name": "k8s_delete", + "description": "Delete a Kubernetes resource.", + "params": [ + "kind", + "name", + "namespace" + ], + "source": "public", + "aliases": [ + "kubernetes", + "k8s" + ] + }, + { + "id": "kubernetes.k8s_list_deployments", + "server": "kubernetes", + "name": "k8s_list_deployments", + "description": "List Kubernetes deployments.", + "params": [ + "namespace" + ], + "source": "public", + "aliases": [ + "kubernetes", + "k8s" + ] + }, + { + "id": "kubernetes.k8s_scale_deployment", + "server": "kubernetes", + "name": "k8s_scale_deployment", + "description": "Scale a Kubernetes deployment.", + "params": [ + "namespace", + "name", + "replicas" + ], + "source": "public", + "aliases": [ + "kubernetes", + "k8s" + ] + }, + { + "id": "kubernetes.k8s_list_services", + "server": "kubernetes", + "name": "k8s_list_services", + "description": "List Kubernetes services.", + "params": [ + "namespace" + ], + "source": "public", + "aliases": [ + "kubernetes", + "k8s" + ] + }, + { + "id": "kubernetes.k8s_get_events", + "server": "kubernetes", + "name": "k8s_get_events", + "description": "List Kubernetes events.", + "params": [ + "namespace" + ], + "source": "public", + "aliases": [ + "kubernetes", + "k8s" + ] + }, + { + "id": "stripe.stripe_create_customer", + "server": "stripe", + "name": "stripe_create_customer", + "description": "Create a Stripe customer.", + "params": [ + "email", + "name" + ], + "source": "public", + "aliases": [ + "stripe" + ] + }, + { + "id": "stripe.stripe_get_customer", + "server": "stripe", + "name": "stripe_get_customer", + "description": "Retrieve a Stripe customer.", + "params": [ + "customer_id" + ], + "source": "public", + "aliases": [ + "stripe" + ] + }, + { + "id": "stripe.stripe_list_customers", + "server": "stripe", + "name": "stripe_list_customers", + "description": "List Stripe customers.", + "params": [ + "email", + "limit" + ], + "source": "public", + "aliases": [ + "stripe" + ] + }, + { + "id": "stripe.stripe_create_payment_intent", + "server": "stripe", + "name": "stripe_create_payment_intent", + "description": "Create a Stripe PaymentIntent.", + "params": [ + "amount", + "currency", + "customer" + ], + "source": "public", + "aliases": [ + "stripe" + ] + }, + { + "id": "stripe.stripe_get_payment_intent", + "server": "stripe", + "name": "stripe_get_payment_intent", + "description": "Retrieve a Stripe PaymentIntent.", + "params": [ + "id" + ], + "source": "public", + "aliases": [ + "stripe" + ] + }, + { + "id": "stripe.stripe_create_invoice", + "server": "stripe", + "name": "stripe_create_invoice", + "description": "Create a Stripe invoice.", + "params": [ + "customer" + ], + "source": "public", + "aliases": [ + "stripe" + ] + }, + { + "id": "stripe.stripe_list_invoices", + "server": "stripe", + "name": "stripe_list_invoices", + "description": "List Stripe invoices.", + "params": [ + "customer" + ], + "source": "public", + "aliases": [ + "stripe" + ] + }, + { + "id": "stripe.stripe_create_product", + "server": "stripe", + "name": "stripe_create_product", + "description": "Create a Stripe product.", + "params": [ + "name", + "description" + ], + "source": "public", + "aliases": [ + "stripe" + ] + }, + { + "id": "stripe.stripe_create_price", + "server": "stripe", + "name": "stripe_create_price", + "description": "Create a Stripe price.", + "params": [ + "product", + "unit_amount", + "currency" + ], + "source": "public", + "aliases": [ + "stripe" + ] + }, + { + "id": "stripe.stripe_list_subscriptions", + "server": "stripe", + "name": "stripe_list_subscriptions", + "description": "List Stripe subscriptions.", + "params": [ + "customer", + "status" + ], + "source": "public", + "aliases": [ + "stripe" + ] + }, + { + "id": "stripe.stripe_cancel_subscription", + "server": "stripe", + "name": "stripe_cancel_subscription", + "description": "Cancel a Stripe subscription.", + "params": [ + "subscription_id" + ], + "source": "public", + "aliases": [ + "stripe" + ] + }, + { + "id": "stripe.stripe_create_refund", + "server": "stripe", + "name": "stripe_create_refund", + "description": "Create a Stripe refund.", + "params": [ + "payment_intent", + "amount" + ], + "source": "public", + "aliases": [ + "stripe" + ] + }, + { + "id": "shopify.shopify_list_products", + "server": "shopify", + "name": "shopify_list_products", + "description": "List Shopify products.", + "params": [ + "limit", + "query" + ], + "source": "public", + "aliases": [ + "shopify" + ] + }, + { + "id": "shopify.shopify_get_product", + "server": "shopify", + "name": "shopify_get_product", + "description": "Get a Shopify product.", + "params": [ + "id" + ], + "source": "public", + "aliases": [ + "shopify" + ] + }, + { + "id": "shopify.shopify_create_product", + "server": "shopify", + "name": "shopify_create_product", + "description": "Create a Shopify product.", + "params": [ + "title", + "body_html", + "vendor" + ], + "source": "public", + "aliases": [ + "shopify" + ] + }, + { + "id": "shopify.shopify_update_product", + "server": "shopify", + "name": "shopify_update_product", + "description": "Update a Shopify product.", + "params": [ + "id", + "title" + ], + "source": "public", + "aliases": [ + "shopify" + ] + }, + { + "id": "shopify.shopify_list_orders", + "server": "shopify", + "name": "shopify_list_orders", + "description": "List Shopify orders.", + "params": [ + "status", + "limit" + ], + "source": "public", + "aliases": [ + "shopify" + ] + }, + { + "id": "shopify.shopify_get_order", + "server": "shopify", + "name": "shopify_get_order", + "description": "Get a Shopify order.", + "params": [ + "id" + ], + "source": "public", + "aliases": [ + "shopify" + ] + }, + { + "id": "shopify.shopify_create_order", + "server": "shopify", + "name": "shopify_create_order", + "description": "Create a Shopify order.", + "params": [ + "line_items", + "email" + ], + "source": "public", + "aliases": [ + "shopify" + ] + }, + { + "id": "shopify.shopify_list_customers", + "server": "shopify", + "name": "shopify_list_customers", + "description": "List Shopify customers.", + "params": [ + "limit" + ], + "source": "public", + "aliases": [ + "shopify" + ] + }, + { + "id": "salesforce.salesforce_query", + "server": "salesforce", + "name": "salesforce_query", + "description": "Run a SOQL query against Salesforce.", + "params": [ + "soql" + ], + "source": "public", + "aliases": [ + "salesforce", + "sfdc" + ] + }, + { + "id": "salesforce.salesforce_create_record", + "server": "salesforce", + "name": "salesforce_create_record", + "description": "Create a Salesforce record.", + "params": [ + "sobject", + "fields" + ], + "source": "public", + "aliases": [ + "salesforce", + "sfdc" + ] + }, + { + "id": "salesforce.salesforce_update_record", + "server": "salesforce", + "name": "salesforce_update_record", + "description": "Update a Salesforce record.", + "params": [ + "sobject", + "id", + "fields" + ], + "source": "public", + "aliases": [ + "salesforce", + "sfdc" + ] + }, + { + "id": "salesforce.salesforce_get_record", + "server": "salesforce", + "name": "salesforce_get_record", + "description": "Get a Salesforce record by id.", + "params": [ + "sobject", + "id" + ], + "source": "public", + "aliases": [ + "salesforce", + "sfdc" + ] + }, + { + "id": "salesforce.salesforce_delete_record", + "server": "salesforce", + "name": "salesforce_delete_record", + "description": "Delete a Salesforce record.", + "params": [ + "sobject", + "id" + ], + "source": "public", + "aliases": [ + "salesforce", + "sfdc" + ] + }, + { + "id": "salesforce.salesforce_list_opportunities", + "server": "salesforce", + "name": "salesforce_list_opportunities", + "description": "List Salesforce opportunities.", + "params": [ + "stage" + ], + "source": "public", + "aliases": [ + "salesforce", + "sfdc" + ] + }, + { + "id": "salesforce.salesforce_create_lead", + "server": "salesforce", + "name": "salesforce_create_lead", + "description": "Create a Salesforce lead.", + "params": [ + "firstName", + "lastName", + "company", + "email" + ], + "source": "public", + "aliases": [ + "salesforce", + "sfdc" + ] + }, + { + "id": "salesforce.salesforce_convert_lead", + "server": "salesforce", + "name": "salesforce_convert_lead", + "description": "Convert a Salesforce lead.", + "params": [ + "leadId" + ], + "source": "public", + "aliases": [ + "salesforce", + "sfdc" + ] + }, + { + "id": "sentry.sentry_list_issues", + "server": "sentry", + "name": "sentry_list_issues", + "description": "List Sentry issues for a project.", + "params": [ + "organization", + "project", + "query" + ], + "source": "public", + "aliases": [ + "sentry" + ] + }, + { + "id": "sentry.sentry_get_issue", + "server": "sentry", + "name": "sentry_get_issue", + "description": "Get a Sentry issue.", + "params": [ + "organization", + "issue_id" + ], + "source": "public", + "aliases": [ + "sentry" + ] + }, + { + "id": "sentry.sentry_list_events", + "server": "sentry", + "name": "sentry_list_events", + "description": "List events for a Sentry issue.", + "params": [ + "organization", + "issue_id" + ], + "source": "public", + "aliases": [ + "sentry" + ] + }, + { + "id": "sentry.sentry_update_issue", + "server": "sentry", + "name": "sentry_update_issue", + "description": "Update status of a Sentry issue.", + "params": [ + "organization", + "issue_id", + "status" + ], + "source": "public", + "aliases": [ + "sentry" + ] + }, + { + "id": "sentry.sentry_list_projects", + "server": "sentry", + "name": "sentry_list_projects", + "description": "List Sentry projects.", + "params": [ + "organization" + ], + "source": "public", + "aliases": [ + "sentry" + ] + }, + { + "id": "datadog.datadog_query_metrics", + "server": "datadog", + "name": "datadog_query_metrics", + "description": "Query Datadog timeseries metrics.", + "params": [ + "query", + "from_ts", + "to_ts" + ], + "source": "public", + "aliases": [ + "datadog" + ] + }, + { + "id": "datadog.datadog_list_monitors", + "server": "datadog", + "name": "datadog_list_monitors", + "description": "List Datadog monitors.", + "params": [ + "query" + ], + "source": "public", + "aliases": [ + "datadog" + ] + }, + { + "id": "datadog.datadog_get_monitor", + "server": "datadog", + "name": "datadog_get_monitor", + "description": "Get a Datadog monitor.", + "params": [ + "monitor_id" + ], + "source": "public", + "aliases": [ + "datadog" + ] + }, + { + "id": "datadog.datadog_create_monitor", + "server": "datadog", + "name": "datadog_create_monitor", + "description": "Create a Datadog monitor.", + "params": [ + "name", + "type", + "query", + "message" + ], + "source": "public", + "aliases": [ + "datadog" + ] + }, + { + "id": "datadog.datadog_list_logs", + "server": "datadog", + "name": "datadog_list_logs", + "description": "Search Datadog logs.", + "params": [ + "query", + "from_ts", + "to_ts" + ], + "source": "public", + "aliases": [ + "datadog" + ] + }, + { + "id": "pagerduty.pagerduty_list_incidents", + "server": "pagerduty", + "name": "pagerduty_list_incidents", + "description": "List PagerDuty incidents.", + "params": [ + "status", + "urgency" + ], + "source": "public", + "aliases": [ + "pagerduty", + "pager duty" + ] + }, + { + "id": "pagerduty.pagerduty_get_incident", + "server": "pagerduty", + "name": "pagerduty_get_incident", + "description": "Get a PagerDuty incident.", + "params": [ + "id" + ], + "source": "public", + "aliases": [ + "pagerduty", + "pager duty" + ] + }, + { + "id": "pagerduty.pagerduty_create_incident", + "server": "pagerduty", + "name": "pagerduty_create_incident", + "description": "Create a PagerDuty incident.", + "params": [ + "title", + "service_id", + "urgency" + ], + "source": "public", + "aliases": [ + "pagerduty", + "pager duty" + ] + }, + { + "id": "pagerduty.pagerduty_acknowledge_incident", + "server": "pagerduty", + "name": "pagerduty_acknowledge_incident", + "description": "Acknowledge a PagerDuty incident.", + "params": [ + "id" + ], + "source": "public", + "aliases": [ + "pagerduty", + "pager duty" + ] + }, + { + "id": "pagerduty.pagerduty_resolve_incident", + "server": "pagerduty", + "name": "pagerduty_resolve_incident", + "description": "Resolve a PagerDuty incident.", + "params": [ + "id" + ], + "source": "public", + "aliases": [ + "pagerduty", + "pager duty" + ] + }, + { + "id": "pagerduty.pagerduty_list_oncalls", + "server": "pagerduty", + "name": "pagerduty_list_oncalls", + "description": "List PagerDuty on-call users.", + "params": [ + "schedule_id" + ], + "source": "public", + "aliases": [ + "pagerduty", + "pager duty" + ] + }, + { + "id": "gmail.gmail_send_email", + "server": "gmail", + "name": "gmail_send_email", + "description": "Send an email via Gmail.", + "params": [ + "to", + "subject", + "body", + "cc" + ], + "source": "public", + "aliases": [ + "gmail", + "email", + "mail" + ] + }, + { + "id": "gmail.gmail_search_emails", + "server": "gmail", + "name": "gmail_search_emails", + "description": "Search Gmail messages.", + "params": [ + "query", + "maxResults" + ], + "source": "public", + "aliases": [ + "gmail", + "email", + "mail" + ] + }, + { + "id": "gmail.gmail_get_email", + "server": "gmail", + "name": "gmail_get_email", + "description": "Get a Gmail message by id.", + "params": [ + "id" + ], + "source": "public", + "aliases": [ + "gmail", + "email", + "mail" + ] + }, + { + "id": "gmail.gmail_create_draft", + "server": "gmail", + "name": "gmail_create_draft", + "description": "Create a Gmail draft.", + "params": [ + "to", + "subject", + "body" + ], + "source": "public", + "aliases": [ + "gmail", + "email", + "mail" + ] + }, + { + "id": "gmail.gmail_list_labels", + "server": "gmail", + "name": "gmail_list_labels", + "description": "List Gmail labels.", + "params": [], + "source": "public", + "aliases": [ + "gmail", + "email", + "mail" + ] + }, + { + "id": "gmail.gmail_modify_email", + "server": "gmail", + "name": "gmail_modify_email", + "description": "Add or remove Gmail labels on a message.", + "params": [ + "id", + "addLabelIds", + "removeLabelIds" + ], + "source": "public", + "aliases": [ + "gmail", + "email", + "mail" + ] + }, + { + "id": "gmail.gmail_trash_email", + "server": "gmail", + "name": "gmail_trash_email", + "description": "Move a Gmail message to trash.", + "params": [ + "id" + ], + "source": "public", + "aliases": [ + "gmail", + "email", + "mail" + ] + }, + { + "id": "google_drive.gdrive_search", + "server": "google_drive", + "name": "gdrive_search", + "description": "Search files in Google Drive.", + "params": [ + "query" + ], + "source": "public", + "aliases": [ + "google drive", + "gdrive", + "drive" + ] + }, + { + "id": "google_drive.gdrive_get_file", + "server": "google_drive", + "name": "gdrive_get_file", + "description": "Get Google Drive file metadata or content.", + "params": [ + "file_id" + ], + "source": "public", + "aliases": [ + "google drive", + "gdrive", + "drive" + ] + }, + { + "id": "google_drive.gdrive_create_file", + "server": "google_drive", + "name": "gdrive_create_file", + "description": "Create a file in Google Drive.", + "params": [ + "name", + "mimeType", + "content", + "parent" + ], + "source": "public", + "aliases": [ + "google drive", + "gdrive", + "drive" + ] + }, + { + "id": "google_drive.gdrive_update_file", + "server": "google_drive", + "name": "gdrive_update_file", + "description": "Update a Google Drive file.", + "params": [ + "file_id", + "content", + "name" + ], + "source": "public", + "aliases": [ + "google drive", + "gdrive", + "drive" + ] + }, + { + "id": "google_drive.gdrive_list_folder", + "server": "google_drive", + "name": "gdrive_list_folder", + "description": "List files in a Google Drive folder.", + "params": [ + "folder_id" + ], + "source": "public", + "aliases": [ + "google drive", + "gdrive", + "drive" + ] + }, + { + "id": "google_drive.gdrive_share_file", + "server": "google_drive", + "name": "gdrive_share_file", + "description": "Share a Google Drive file.", + "params": [ + "file_id", + "email", + "role" + ], + "source": "public", + "aliases": [ + "google drive", + "gdrive", + "drive" + ] + }, + { + "id": "google_calendar.gcal_list_events", + "server": "google_calendar", + "name": "gcal_list_events", + "description": "List Google Calendar events.", + "params": [ + "calendar_id", + "timeMin", + "timeMax" + ], + "source": "public", + "aliases": [ + "google calendar", + "gcal", + "calendar" + ] + }, + { + "id": "google_calendar.gcal_create_event", + "server": "google_calendar", + "name": "gcal_create_event", + "description": "Create a Google Calendar event.", + "params": [ + "calendar_id", + "summary", + "start", + "end", + "attendees" + ], + "source": "public", + "aliases": [ + "google calendar", + "gcal", + "calendar" + ] + }, + { + "id": "google_calendar.gcal_update_event", + "server": "google_calendar", + "name": "gcal_update_event", + "description": "Update a Google Calendar event.", + "params": [ + "calendar_id", + "event_id", + "summary" + ], + "source": "public", + "aliases": [ + "google calendar", + "gcal", + "calendar" + ] + }, + { + "id": "google_calendar.gcal_delete_event", + "server": "google_calendar", + "name": "gcal_delete_event", + "description": "Delete a Google Calendar event.", + "params": [ + "calendar_id", + "event_id" + ], + "source": "public", + "aliases": [ + "google calendar", + "gcal", + "calendar" + ] + }, + { + "id": "google_calendar.gcal_list_calendars", + "server": "google_calendar", + "name": "gcal_list_calendars", + "description": "List Google calendars.", + "params": [], + "source": "public", + "aliases": [ + "google calendar", + "gcal", + "calendar" + ] + }, + { + "id": "gitlab.gitlab_create_issue", + "server": "gitlab", + "name": "gitlab_create_issue", + "description": "Create a GitLab issue.", + "params": [ + "project_id", + "title", + "description" + ], + "source": "public", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.gitlab_list_issues", + "server": "gitlab", + "name": "gitlab_list_issues", + "description": "List GitLab issues.", + "params": [ + "project_id", + "state" + ], + "source": "public", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.gitlab_get_issue", + "server": "gitlab", + "name": "gitlab_get_issue", + "description": "Get a GitLab issue.", + "params": [ + "project_id", + "issue_iid" + ], + "source": "public", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.gitlab_create_merge_request", + "server": "gitlab", + "name": "gitlab_create_merge_request", + "description": "Create a GitLab merge request.", + "params": [ + "project_id", + "source_branch", + "target_branch", + "title" + ], + "source": "public", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.gitlab_list_merge_requests", + "server": "gitlab", + "name": "gitlab_list_merge_requests", + "description": "List GitLab merge requests.", + "params": [ + "project_id", + "state" + ], + "source": "public", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.gitlab_get_merge_request", + "server": "gitlab", + "name": "gitlab_get_merge_request", + "description": "Get a GitLab merge request.", + "params": [ + "project_id", + "merge_request_iid" + ], + "source": "public", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.gitlab_merge_merge_request", + "server": "gitlab", + "name": "gitlab_merge_merge_request", + "description": "Merge a GitLab merge request.", + "params": [ + "project_id", + "merge_request_iid" + ], + "source": "public", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.gitlab_list_projects", + "server": "gitlab", + "name": "gitlab_list_projects", + "description": "List GitLab projects.", + "params": [ + "search" + ], + "source": "public", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.gitlab_get_file", + "server": "gitlab", + "name": "gitlab_get_file", + "description": "Get a file from a GitLab project.", + "params": [ + "project_id", + "file_path", + "ref" + ], + "source": "public", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.gitlab_create_pipeline", + "server": "gitlab", + "name": "gitlab_create_pipeline", + "description": "Create a GitLab CI pipeline.", + "params": [ + "project_id", + "ref" + ], + "source": "public", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "bitbucket.bitbucket_create_pr", + "server": "bitbucket", + "name": "bitbucket_create_pr", + "description": "Create a Bitbucket pull request.", + "params": [ + "workspace", + "repo", + "title", + "source", + "dest" + ], + "source": "public", + "aliases": [ + "bitbucket", + "bb" + ] + }, + { + "id": "bitbucket.bitbucket_list_prs", + "server": "bitbucket", + "name": "bitbucket_list_prs", + "description": "List Bitbucket pull requests.", + "params": [ + "workspace", + "repo" + ], + "source": "public", + "aliases": [ + "bitbucket", + "bb" + ] + }, + { + "id": "bitbucket.bitbucket_create_issue", + "server": "bitbucket", + "name": "bitbucket_create_issue", + "description": "Create a Bitbucket issue.", + "params": [ + "workspace", + "repo", + "title" + ], + "source": "public", + "aliases": [ + "bitbucket", + "bb" + ] + }, + { + "id": "bitbucket.bitbucket_list_repos", + "server": "bitbucket", + "name": "bitbucket_list_repos", + "description": "List Bitbucket repositories.", + "params": [ + "workspace" + ], + "source": "public", + "aliases": [ + "bitbucket", + "bb" + ] + }, + { + "id": "sqlite.sqlite_query", + "server": "sqlite", + "name": "sqlite_query", + "description": "Run a SQL query against SQLite.", + "params": [ + "sql" + ], + "source": "public", + "aliases": [ + "sqlite" + ] + }, + { + "id": "sqlite.sqlite_list_tables", + "server": "sqlite", + "name": "sqlite_list_tables", + "description": "List tables in the SQLite database.", + "params": [], + "source": "public", + "aliases": [ + "sqlite" + ] + }, + { + "id": "sqlite.sqlite_describe_table", + "server": "sqlite", + "name": "sqlite_describe_table", + "description": "Describe a SQLite table.", + "params": [ + "table" + ], + "source": "public", + "aliases": [ + "sqlite" + ] + }, + { + "id": "mysql.mysql_query", + "server": "mysql", + "name": "mysql_query", + "description": "Run a SQL query against MySQL.", + "params": [ + "sql" + ], + "source": "public", + "aliases": [ + "mysql" + ] + }, + { + "id": "mysql.mysql_list_tables", + "server": "mysql", + "name": "mysql_list_tables", + "description": "List MySQL tables.", + "params": [ + "schema" + ], + "source": "public", + "aliases": [ + "mysql" + ] + }, + { + "id": "mongodb.mongo_find", + "server": "mongodb", + "name": "mongo_find", + "description": "Find documents in a MongoDB collection.", + "params": [ + "database", + "collection", + "filter" + ], + "source": "public", + "aliases": [ + "mongodb", + "mongo" + ] + }, + { + "id": "mongodb.mongo_insert", + "server": "mongodb", + "name": "mongo_insert", + "description": "Insert documents into MongoDB.", + "params": [ + "database", + "collection", + "documents" + ], + "source": "public", + "aliases": [ + "mongodb", + "mongo" + ] + }, + { + "id": "mongodb.mongo_update", + "server": "mongodb", + "name": "mongo_update", + "description": "Update documents in MongoDB.", + "params": [ + "database", + "collection", + "filter", + "update" + ], + "source": "public", + "aliases": [ + "mongodb", + "mongo" + ] + }, + { + "id": "mongodb.mongo_delete", + "server": "mongodb", + "name": "mongo_delete", + "description": "Delete documents from MongoDB.", + "params": [ + "database", + "collection", + "filter" + ], + "source": "public", + "aliases": [ + "mongodb", + "mongo" + ] + }, + { + "id": "mongodb.mongo_list_collections", + "server": "mongodb", + "name": "mongo_list_collections", + "description": "List MongoDB collections.", + "params": [ + "database" + ], + "source": "public", + "aliases": [ + "mongodb", + "mongo" + ] + }, + { + "id": "redis.redis_get", + "server": "redis", + "name": "redis_get", + "description": "Get a Redis key.", + "params": [ + "key" + ], + "source": "public", + "aliases": [ + "redis" + ] + }, + { + "id": "redis.redis_set", + "server": "redis", + "name": "redis_set", + "description": "Set a Redis key.", + "params": [ + "key", + "value", + "ttl" + ], + "source": "public", + "aliases": [ + "redis" + ] + }, + { + "id": "redis.redis_delete", + "server": "redis", + "name": "redis_delete", + "description": "Delete Redis keys.", + "params": [ + "keys" + ], + "source": "public", + "aliases": [ + "redis" + ] + }, + { + "id": "redis.redis_keys", + "server": "redis", + "name": "redis_keys", + "description": "List Redis keys matching a pattern.", + "params": [ + "pattern" + ], + "source": "public", + "aliases": [ + "redis" + ] + }, + { + "id": "elasticsearch.es_search", + "server": "elasticsearch", + "name": "es_search", + "description": "Search an Elasticsearch index.", + "params": [ + "index", + "query" + ], + "source": "public", + "aliases": [ + "elasticsearch", + "elastic" + ] + }, + { + "id": "elasticsearch.es_index_doc", + "server": "elasticsearch", + "name": "es_index_doc", + "description": "Index a document in Elasticsearch.", + "params": [ + "index", + "document", + "id" + ], + "source": "public", + "aliases": [ + "elasticsearch", + "elastic" + ] + }, + { + "id": "elasticsearch.es_get_doc", + "server": "elasticsearch", + "name": "es_get_doc", + "description": "Get a document from Elasticsearch.", + "params": [ + "index", + "id" + ], + "source": "public", + "aliases": [ + "elasticsearch", + "elastic" + ] + }, + { + "id": "aws_s3.s3_list_buckets", + "server": "aws_s3", + "name": "s3_list_buckets", + "description": "List AWS S3 buckets.", + "params": [], + "source": "public", + "aliases": [ + "s3", + "aws s3", + "bucket" + ] + }, + { + "id": "aws_s3.s3_list_objects", + "server": "aws_s3", + "name": "s3_list_objects", + "description": "List objects in an S3 bucket.", + "params": [ + "bucket", + "prefix" + ], + "source": "public", + "aliases": [ + "s3", + "aws s3", + "bucket" + ] + }, + { + "id": "aws_s3.s3_get_object", + "server": "aws_s3", + "name": "s3_get_object", + "description": "Get an object from S3.", + "params": [ + "bucket", + "key" + ], + "source": "public", + "aliases": [ + "s3", + "aws s3", + "bucket" + ] + }, + { + "id": "aws_s3.s3_put_object", + "server": "aws_s3", + "name": "s3_put_object", + "description": "Upload an object to S3.", + "params": [ + "bucket", + "key", + "body" + ], + "source": "public", + "aliases": [ + "s3", + "aws s3", + "bucket" + ] + }, + { + "id": "aws_s3.s3_delete_object", + "server": "aws_s3", + "name": "s3_delete_object", + "description": "Delete an S3 object.", + "params": [ + "bucket", + "key" + ], + "source": "public", + "aliases": [ + "s3", + "aws s3", + "bucket" + ] + }, + { + "id": "cloudflare.cf_list_zones", + "server": "cloudflare", + "name": "cf_list_zones", + "description": "List Cloudflare zones.", + "params": [], + "source": "public", + "aliases": [ + "cloudflare" + ] + }, + { + "id": "cloudflare.cf_list_dns", + "server": "cloudflare", + "name": "cf_list_dns", + "description": "List DNS records for a Cloudflare zone.", + "params": [ + "zone_id" + ], + "source": "public", + "aliases": [ + "cloudflare" + ] + }, + { + "id": "cloudflare.cf_create_dns", + "server": "cloudflare", + "name": "cf_create_dns", + "description": "Create a Cloudflare DNS record.", + "params": [ + "zone_id", + "type", + "name", + "content" + ], + "source": "public", + "aliases": [ + "cloudflare" + ] + }, + { + "id": "cloudflare.cf_purge_cache", + "server": "cloudflare", + "name": "cf_purge_cache", + "description": "Purge Cloudflare cache.", + "params": [ + "zone_id", + "files" + ], + "source": "public", + "aliases": [ + "cloudflare" + ] + }, + { + "id": "vercel.vercel_list_projects", + "server": "vercel", + "name": "vercel_list_projects", + "description": "List Vercel projects.", + "params": [], + "source": "public", + "aliases": [ + "vercel" + ] + }, + { + "id": "vercel.vercel_list_deployments", + "server": "vercel", + "name": "vercel_list_deployments", + "description": "List Vercel deployments.", + "params": [ + "projectId" + ], + "source": "public", + "aliases": [ + "vercel" + ] + }, + { + "id": "vercel.vercel_get_deployment", + "server": "vercel", + "name": "vercel_get_deployment", + "description": "Get a Vercel deployment.", + "params": [ + "id" + ], + "source": "public", + "aliases": [ + "vercel" + ] + }, + { + "id": "vercel.vercel_create_project", + "server": "vercel", + "name": "vercel_create_project", + "description": "Create a Vercel project.", + "params": [ + "name" + ], + "source": "public", + "aliases": [ + "vercel" + ] + }, + { + "id": "supabase.supabase_list_tables", + "server": "supabase", + "name": "supabase_list_tables", + "description": "List tables in a Supabase project.", + "params": [], + "source": "public", + "aliases": [ + "supabase" + ] + }, + { + "id": "supabase.supabase_select", + "server": "supabase", + "name": "supabase_select", + "description": "Select rows from a Supabase table.", + "params": [ + "table", + "filter" + ], + "source": "public", + "aliases": [ + "supabase" + ] + }, + { + "id": "supabase.supabase_insert", + "server": "supabase", + "name": "supabase_insert", + "description": "Insert rows into a Supabase table.", + "params": [ + "table", + "rows" + ], + "source": "public", + "aliases": [ + "supabase" + ] + }, + { + "id": "supabase.supabase_update", + "server": "supabase", + "name": "supabase_update", + "description": "Update rows in a Supabase table.", + "params": [ + "table", + "filter", + "values" + ], + "source": "public", + "aliases": [ + "supabase" + ] + }, + { + "id": "supabase.supabase_invoke_fn", + "server": "supabase", + "name": "supabase_invoke_fn", + "description": "Invoke a Supabase edge function.", + "params": [ + "name", + "body" + ], + "source": "public", + "aliases": [ + "supabase" + ] + }, + { + "id": "figma.figma_get_file", + "server": "figma", + "name": "figma_get_file", + "description": "Get a Figma file.", + "params": [ + "file_key" + ], + "source": "public", + "aliases": [ + "figma" + ] + }, + { + "id": "figma.figma_get_file_nodes", + "server": "figma", + "name": "figma_get_file_nodes", + "description": "Get specific nodes from a Figma file.", + "params": [ + "file_key", + "ids" + ], + "source": "public", + "aliases": [ + "figma" + ] + }, + { + "id": "figma.figma_get_comments", + "server": "figma", + "name": "figma_get_comments", + "description": "Get comments on a Figma file.", + "params": [ + "file_key" + ], + "source": "public", + "aliases": [ + "figma" + ] + }, + { + "id": "figma.figma_post_comment", + "server": "figma", + "name": "figma_post_comment", + "description": "Post a comment on a Figma file.", + "params": [ + "file_key", + "message" + ], + "source": "public", + "aliases": [ + "figma" + ] + }, + { + "id": "figma.figma_get_images", + "server": "figma", + "name": "figma_get_images", + "description": "Export images from a Figma file.", + "params": [ + "file_key", + "ids", + "format" + ], + "source": "public", + "aliases": [ + "figma" + ] + }, + { + "id": "twilio.twilio_send_sms", + "server": "twilio", + "name": "twilio_send_sms", + "description": "Send an SMS via Twilio.", + "params": [ + "to", + "from_", + "body" + ], + "source": "public", + "aliases": [ + "twilio", + "sms" + ] + }, + { + "id": "twilio.twilio_list_messages", + "server": "twilio", + "name": "twilio_list_messages", + "description": "List Twilio SMS messages.", + "params": [ + "to" + ], + "source": "public", + "aliases": [ + "twilio", + "sms" + ] + }, + { + "id": "twilio.twilio_make_call", + "server": "twilio", + "name": "twilio_make_call", + "description": "Place a phone call via Twilio.", + "params": [ + "to", + "from_", + "url" + ], + "source": "public", + "aliases": [ + "twilio", + "sms" + ] + }, + { + "id": "okta.okta_list_users", + "server": "okta", + "name": "okta_list_users", + "description": "List Okta users.", + "params": [ + "q" + ], + "source": "public", + "aliases": [ + "okta" + ] + }, + { + "id": "okta.okta_get_user", + "server": "okta", + "name": "okta_get_user", + "description": "Get an Okta user.", + "params": [ + "user_id" + ], + "source": "public", + "aliases": [ + "okta" + ] + }, + { + "id": "okta.okta_create_user", + "server": "okta", + "name": "okta_create_user", + "description": "Create an Okta user.", + "params": [ + "profile" + ], + "source": "public", + "aliases": [ + "okta" + ] + }, + { + "id": "okta.okta_deactivate_user", + "server": "okta", + "name": "okta_deactivate_user", + "description": "Deactivate an Okta user.", + "params": [ + "user_id" + ], + "source": "public", + "aliases": [ + "okta" + ] + }, + { + "id": "prometheus.prom_query", + "server": "prometheus", + "name": "prom_query", + "description": "Instant query Prometheus.", + "params": [ + "query" + ], + "source": "public", + "aliases": [ + "prometheus" + ] + }, + { + "id": "prometheus.prom_query_range", + "server": "prometheus", + "name": "prom_query_range", + "description": "Range query Prometheus.", + "params": [ + "query", + "start", + "end", + "step" + ], + "source": "public", + "aliases": [ + "prometheus" + ] + }, + { + "id": "prometheus.prom_list_metrics", + "server": "prometheus", + "name": "prom_list_metrics", + "description": "List Prometheus metric names.", + "params": [ + "match" + ], + "source": "public", + "aliases": [ + "prometheus" + ] + }, + { + "id": "grafana.grafana_search_dashboards", + "server": "grafana", + "name": "grafana_search_dashboards", + "description": "Search Grafana dashboards.", + "params": [ + "query" + ], + "source": "public", + "aliases": [ + "grafana" + ] + }, + { + "id": "grafana.grafana_get_dashboard", + "server": "grafana", + "name": "grafana_get_dashboard", + "description": "Get a Grafana dashboard.", + "params": [ + "uid" + ], + "source": "public", + "aliases": [ + "grafana" + ] + }, + { + "id": "grafana.grafana_create_annotation", + "server": "grafana", + "name": "grafana_create_annotation", + "description": "Create a Grafana annotation.", + "params": [ + "text", + "time", + "tags" + ], + "source": "public", + "aliases": [ + "grafana" + ] + }, + { + "id": "circleci.circleci_list_pipelines", + "server": "circleci", + "name": "circleci_list_pipelines", + "description": "List CircleCI pipelines.", + "params": [ + "project_slug" + ], + "source": "public", + "aliases": [ + "circleci", + "ci" + ] + }, + { + "id": "circleci.circleci_get_pipeline", + "server": "circleci", + "name": "circleci_get_pipeline", + "description": "Get a CircleCI pipeline.", + "params": [ + "pipeline_id" + ], + "source": "public", + "aliases": [ + "circleci", + "ci" + ] + }, + { + "id": "circleci.circleci_trigger_pipeline", + "server": "circleci", + "name": "circleci_trigger_pipeline", + "description": "Trigger a CircleCI pipeline.", + "params": [ + "project_slug", + "branch" + ], + "source": "public", + "aliases": [ + "circleci", + "ci" + ] + }, + { + "id": "terraform.tf_plan", + "server": "terraform", + "name": "tf_plan", + "description": "Run terraform plan.", + "params": [ + "dir" + ], + "source": "public", + "aliases": [ + "terraform" + ] + }, + { + "id": "terraform.tf_apply", + "server": "terraform", + "name": "tf_apply", + "description": "Run terraform apply.", + "params": [ + "dir", + "auto_approve" + ], + "source": "public", + "aliases": [ + "terraform" + ] + }, + { + "id": "terraform.tf_state_list", + "server": "terraform", + "name": "tf_state_list", + "description": "List resources in terraform state.", + "params": [ + "dir" + ], + "source": "public", + "aliases": [ + "terraform" + ] + }, + { + "id": "asana.asana_create_task", + "server": "asana", + "name": "asana_create_task", + "description": "Create an Asana task.", + "params": [ + "name", + "notes", + "projects", + "assignee" + ], + "source": "public", + "aliases": [ + "asana" + ] + }, + { + "id": "asana.asana_list_tasks", + "server": "asana", + "name": "asana_list_tasks", + "description": "List Asana tasks.", + "params": [ + "project", + "assignee" + ], + "source": "public", + "aliases": [ + "asana" + ] + }, + { + "id": "asana.asana_get_task", + "server": "asana", + "name": "asana_get_task", + "description": "Get an Asana task.", + "params": [ + "task_gid" + ], + "source": "public", + "aliases": [ + "asana" + ] + }, + { + "id": "asana.asana_update_task", + "server": "asana", + "name": "asana_update_task", + "description": "Update an Asana task.", + "params": [ + "task_gid", + "name", + "completed" + ], + "source": "public", + "aliases": [ + "asana" + ] + }, + { + "id": "asana.asana_create_project", + "server": "asana", + "name": "asana_create_project", + "description": "Create an Asana project.", + "params": [ + "name", + "workspace" + ], + "source": "public", + "aliases": [ + "asana" + ] + }, + { + "id": "trello.trello_create_card", + "server": "trello", + "name": "trello_create_card", + "description": "Create a Trello card.", + "params": [ + "idList", + "name", + "desc" + ], + "source": "public", + "aliases": [ + "trello" + ] + }, + { + "id": "trello.trello_list_cards", + "server": "trello", + "name": "trello_list_cards", + "description": "List cards on a Trello board or list.", + "params": [ + "idBoard", + "idList" + ], + "source": "public", + "aliases": [ + "trello" + ] + }, + { + "id": "trello.trello_create_board", + "server": "trello", + "name": "trello_create_board", + "description": "Create a Trello board.", + "params": [ + "name" + ], + "source": "public", + "aliases": [ + "trello" + ] + }, + { + "id": "trello.trello_add_comment", + "server": "trello", + "name": "trello_add_comment", + "description": "Comment on a Trello card.", + "params": [ + "idCard", + "text" + ], + "source": "public", + "aliases": [ + "trello" + ] + }, + { + "id": "clickup.clickup_create_task", + "server": "clickup", + "name": "clickup_create_task", + "description": "Create a ClickUp task.", + "params": [ + "list_id", + "name", + "description" + ], + "source": "public", + "aliases": [ + "clickup", + "click up" + ] + }, + { + "id": "clickup.clickup_list_tasks", + "server": "clickup", + "name": "clickup_list_tasks", + "description": "List ClickUp tasks.", + "params": [ + "list_id" + ], + "source": "public", + "aliases": [ + "clickup", + "click up" + ] + }, + { + "id": "clickup.clickup_get_task", + "server": "clickup", + "name": "clickup_get_task", + "description": "Get a ClickUp task.", + "params": [ + "task_id" + ], + "source": "public", + "aliases": [ + "clickup", + "click up" + ] + }, + { + "id": "clickup.clickup_update_task", + "server": "clickup", + "name": "clickup_update_task", + "description": "Update a ClickUp task.", + "params": [ + "task_id", + "name", + "status" + ], + "source": "public", + "aliases": [ + "clickup", + "click up" + ] + }, + { + "id": "confluence.confluence_get_page", + "server": "confluence", + "name": "confluence_get_page", + "description": "Get a Confluence page.", + "params": [ + "id" + ], + "source": "public", + "aliases": [ + "confluence" + ] + }, + { + "id": "confluence.confluence_create_page", + "server": "confluence", + "name": "confluence_create_page", + "description": "Create a Confluence page.", + "params": [ + "spaceId", + "title", + "body" + ], + "source": "public", + "aliases": [ + "confluence" + ] + }, + { + "id": "confluence.confluence_update_page", + "server": "confluence", + "name": "confluence_update_page", + "description": "Update a Confluence page.", + "params": [ + "id", + "title", + "body" + ], + "source": "public", + "aliases": [ + "confluence" + ] + }, + { + "id": "confluence.confluence_search", + "server": "confluence", + "name": "confluence_search", + "description": "Search Confluence content.", + "params": [ + "query" + ], + "source": "public", + "aliases": [ + "confluence" + ] + }, + { + "id": "hubspot.hubspot_create_contact", + "server": "hubspot", + "name": "hubspot_create_contact", + "description": "Create a HubSpot contact.", + "params": [ + "email", + "firstname", + "lastname" + ], + "source": "public", + "aliases": [ + "hubspot" + ] + }, + { + "id": "hubspot.hubspot_get_contact", + "server": "hubspot", + "name": "hubspot_get_contact", + "description": "Get a HubSpot contact.", + "params": [ + "id" + ], + "source": "public", + "aliases": [ + "hubspot" + ] + }, + { + "id": "hubspot.hubspot_list_contacts", + "server": "hubspot", + "name": "hubspot_list_contacts", + "description": "List HubSpot contacts.", + "params": [ + "limit" + ], + "source": "public", + "aliases": [ + "hubspot" + ] + }, + { + "id": "hubspot.hubspot_create_deal", + "server": "hubspot", + "name": "hubspot_create_deal", + "description": "Create a HubSpot deal.", + "params": [ + "dealname", + "amount", + "pipeline" + ], + "source": "public", + "aliases": [ + "hubspot" + ] + }, + { + "id": "hubspot.hubspot_search_contacts", + "server": "hubspot", + "name": "hubspot_search_contacts", + "description": "Search HubSpot contacts.", + "params": [ + "query" + ], + "source": "public", + "aliases": [ + "hubspot" + ] + }, + { + "id": "zendesk.zendesk_create_ticket", + "server": "zendesk", + "name": "zendesk_create_ticket", + "description": "Create a Zendesk support ticket.", + "params": [ + "subject", + "comment", + "requester" + ], + "source": "public", + "aliases": [ + "zendesk" + ] + }, + { + "id": "zendesk.zendesk_get_ticket", + "server": "zendesk", + "name": "zendesk_get_ticket", + "description": "Get a Zendesk ticket.", + "params": [ + "id" + ], + "source": "public", + "aliases": [ + "zendesk" + ] + }, + { + "id": "zendesk.zendesk_list_tickets", + "server": "zendesk", + "name": "zendesk_list_tickets", + "description": "List Zendesk tickets.", + "params": [ + "status" + ], + "source": "public", + "aliases": [ + "zendesk" + ] + }, + { + "id": "zendesk.zendesk_update_ticket", + "server": "zendesk", + "name": "zendesk_update_ticket", + "description": "Update a Zendesk ticket.", + "params": [ + "id", + "status", + "comment" + ], + "source": "public", + "aliases": [ + "zendesk" + ] + }, + { + "id": "zendesk.zendesk_search", + "server": "zendesk", + "name": "zendesk_search", + "description": "Search Zendesk tickets and users.", + "params": [ + "query" + ], + "source": "public", + "aliases": [ + "zendesk" + ] + }, + { + "id": "discord.discord_send_message", + "server": "discord", + "name": "discord_send_message", + "description": "Send a message to a Discord channel.", + "params": [ + "channel_id", + "content" + ], + "source": "public", + "aliases": [ + "discord" + ] + }, + { + "id": "discord.discord_list_channels", + "server": "discord", + "name": "discord_list_channels", + "description": "List Discord guild channels.", + "params": [ + "guild_id" + ], + "source": "public", + "aliases": [ + "discord" + ] + }, + { + "id": "discord.discord_get_channel", + "server": "discord", + "name": "discord_get_channel", + "description": "Get a Discord channel.", + "params": [ + "channel_id" + ], + "source": "public", + "aliases": [ + "discord" + ] + }, + { + "id": "discord.discord_add_reaction", + "server": "discord", + "name": "discord_add_reaction", + "description": "Add a reaction to a Discord message.", + "params": [ + "channel_id", + "message_id", + "emoji" + ], + "source": "public", + "aliases": [ + "discord" + ] + }, + { + "id": "discord.discord_list_members", + "server": "discord", + "name": "discord_list_members", + "description": "List Discord guild members.", + "params": [ + "guild_id" + ], + "source": "public", + "aliases": [ + "discord" + ] + }, + { + "id": "github.create_milestone", + "server": "github", + "name": "create_milestone", + "description": "Create a milestone in github.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.get_milestone", + "server": "github", + "name": "get_milestone", + "description": "Get a milestone in github.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.list_milestones", + "server": "github", + "name": "list_milestones", + "description": "List milestones in github.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.update_milestone", + "server": "github", + "name": "update_milestone", + "description": "Update a milestone in github.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.delete_milestone", + "server": "github", + "name": "delete_milestone", + "description": "Delete a milestone in github.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.create_hook", + "server": "github", + "name": "create_hook", + "description": "Create a hook in github.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.get_hook", + "server": "github", + "name": "get_hook", + "description": "Get a hook in github.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.list_hooks", + "server": "github", + "name": "list_hooks", + "description": "List hooks in github.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.update_hook", + "server": "github", + "name": "update_hook", + "description": "Update a hook in github.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.delete_hook", + "server": "github", + "name": "delete_hook", + "description": "Delete a hook in github.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.create_environment", + "server": "github", + "name": "create_environment", + "description": "Create a environment in github.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.get_environment", + "server": "github", + "name": "get_environment", + "description": "Get a environment in github.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.list_environments", + "server": "github", + "name": "list_environments", + "description": "List environments in github.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.update_environment", + "server": "github", + "name": "update_environment", + "description": "Update a environment in github.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.delete_environment", + "server": "github", + "name": "delete_environment", + "description": "Delete a environment in github.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.create_package", + "server": "github", + "name": "create_package", + "description": "Create a package in github.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.get_package", + "server": "github", + "name": "get_package", + "description": "Get a package in github.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.list_packages", + "server": "github", + "name": "list_packages", + "description": "List packages in github.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.update_package", + "server": "github", + "name": "update_package", + "description": "Update a package in github.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.delete_package", + "server": "github", + "name": "delete_package", + "description": "Delete a package in github.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.create_check_run", + "server": "github", + "name": "create_check_run", + "description": "Create a check run in github.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.get_check_run", + "server": "github", + "name": "get_check_run", + "description": "Get a check run in github.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.list_check_runs", + "server": "github", + "name": "list_check_runs", + "description": "List check runs in github.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.update_check_run", + "server": "github", + "name": "update_check_run", + "description": "Update a check run in github.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.delete_check_run", + "server": "github", + "name": "delete_check_run", + "description": "Delete a check run in github.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.create_artifact", + "server": "github", + "name": "create_artifact", + "description": "Create a artifact in github.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.get_artifact", + "server": "github", + "name": "get_artifact", + "description": "Get a artifact in github.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.list_artifacts", + "server": "github", + "name": "list_artifacts", + "description": "List artifacts in github.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.update_artifact", + "server": "github", + "name": "update_artifact", + "description": "Update a artifact in github.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.delete_artifact", + "server": "github", + "name": "delete_artifact", + "description": "Delete a artifact in github.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.create_runner", + "server": "github", + "name": "create_runner", + "description": "Create a runner in github.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.get_runner", + "server": "github", + "name": "get_runner", + "description": "Get a runner in github.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.list_runners", + "server": "github", + "name": "list_runners", + "description": "List runners in github.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.update_runner", + "server": "github", + "name": "update_runner", + "description": "Update a runner in github.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "github.delete_runner", + "server": "github", + "name": "delete_runner", + "description": "Delete a runner in github.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "github", + "gh", + "git hub" + ] + }, + { + "id": "gitlab.create_pipeline", + "server": "gitlab", + "name": "create_pipeline", + "description": "Create a pipeline in gitlab.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.get_pipeline", + "server": "gitlab", + "name": "get_pipeline", + "description": "Get a pipeline in gitlab.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.list_pipelines", + "server": "gitlab", + "name": "list_pipelines", + "description": "List pipelines in gitlab.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.update_pipeline", + "server": "gitlab", + "name": "update_pipeline", + "description": "Update a pipeline in gitlab.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.delete_pipeline", + "server": "gitlab", + "name": "delete_pipeline", + "description": "Delete a pipeline in gitlab.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.create_job", + "server": "gitlab", + "name": "create_job", + "description": "Create a job in gitlab.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.get_job", + "server": "gitlab", + "name": "get_job", + "description": "Get a job in gitlab.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.list_jobs", + "server": "gitlab", + "name": "list_jobs", + "description": "List jobs in gitlab.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.update_job", + "server": "gitlab", + "name": "update_job", + "description": "Update a job in gitlab.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.delete_job", + "server": "gitlab", + "name": "delete_job", + "description": "Delete a job in gitlab.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.create_wiki", + "server": "gitlab", + "name": "create_wiki", + "description": "Create a wiki in gitlab.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.get_wiki", + "server": "gitlab", + "name": "get_wiki", + "description": "Get a wiki in gitlab.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.list_wikis", + "server": "gitlab", + "name": "list_wikis", + "description": "List wikis in gitlab.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.update_wiki", + "server": "gitlab", + "name": "update_wiki", + "description": "Update a wiki in gitlab.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.delete_wiki", + "server": "gitlab", + "name": "delete_wiki", + "description": "Delete a wiki in gitlab.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.create_snippet", + "server": "gitlab", + "name": "create_snippet", + "description": "Create a snippet in gitlab.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.get_snippet", + "server": "gitlab", + "name": "get_snippet", + "description": "Get a snippet in gitlab.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.list_snippets", + "server": "gitlab", + "name": "list_snippets", + "description": "List snippets in gitlab.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.update_snippet", + "server": "gitlab", + "name": "update_snippet", + "description": "Update a snippet in gitlab.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.delete_snippet", + "server": "gitlab", + "name": "delete_snippet", + "description": "Delete a snippet in gitlab.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.create_label", + "server": "gitlab", + "name": "create_label", + "description": "Create a label in gitlab.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.get_label", + "server": "gitlab", + "name": "get_label", + "description": "Get a label in gitlab.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.list_labels", + "server": "gitlab", + "name": "list_labels", + "description": "List labels in gitlab.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.update_label", + "server": "gitlab", + "name": "update_label", + "description": "Update a label in gitlab.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.delete_label", + "server": "gitlab", + "name": "delete_label", + "description": "Delete a label in gitlab.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.create_milestone", + "server": "gitlab", + "name": "create_milestone", + "description": "Create a milestone in gitlab.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.get_milestone", + "server": "gitlab", + "name": "get_milestone", + "description": "Get a milestone in gitlab.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.list_milestones", + "server": "gitlab", + "name": "list_milestones", + "description": "List milestones in gitlab.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.update_milestone", + "server": "gitlab", + "name": "update_milestone", + "description": "Update a milestone in gitlab.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.delete_milestone", + "server": "gitlab", + "name": "delete_milestone", + "description": "Delete a milestone in gitlab.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.create_release", + "server": "gitlab", + "name": "create_release", + "description": "Create a release in gitlab.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.get_release", + "server": "gitlab", + "name": "get_release", + "description": "Get a release in gitlab.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.list_releases", + "server": "gitlab", + "name": "list_releases", + "description": "List releases in gitlab.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.update_release", + "server": "gitlab", + "name": "update_release", + "description": "Update a release in gitlab.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "gitlab.delete_release", + "server": "gitlab", + "name": "delete_release", + "description": "Delete a release in gitlab.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "gitlab", + "gl" + ] + }, + { + "id": "jira.create_board", + "server": "jira", + "name": "create_board", + "description": "Create a board in jira.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.get_board", + "server": "jira", + "name": "get_board", + "description": "Get a board in jira.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.list_boards", + "server": "jira", + "name": "list_boards", + "description": "List boards in jira.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.update_board", + "server": "jira", + "name": "update_board", + "description": "Update a board in jira.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.delete_board", + "server": "jira", + "name": "delete_board", + "description": "Delete a board in jira.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.create_sprint", + "server": "jira", + "name": "create_sprint", + "description": "Create a sprint in jira.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.get_sprint", + "server": "jira", + "name": "get_sprint", + "description": "Get a sprint in jira.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.list_sprints", + "server": "jira", + "name": "list_sprints", + "description": "List sprints in jira.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.update_sprint", + "server": "jira", + "name": "update_sprint", + "description": "Update a sprint in jira.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.delete_sprint", + "server": "jira", + "name": "delete_sprint", + "description": "Delete a sprint in jira.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.create_filter", + "server": "jira", + "name": "create_filter", + "description": "Create a filter in jira.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.get_filter", + "server": "jira", + "name": "get_filter", + "description": "Get a filter in jira.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.list_filters", + "server": "jira", + "name": "list_filters", + "description": "List filters in jira.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.update_filter", + "server": "jira", + "name": "update_filter", + "description": "Update a filter in jira.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.delete_filter", + "server": "jira", + "name": "delete_filter", + "description": "Delete a filter in jira.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.create_webhook", + "server": "jira", + "name": "create_webhook", + "description": "Create a webhook in jira.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.get_webhook", + "server": "jira", + "name": "get_webhook", + "description": "Get a webhook in jira.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.list_webhooks", + "server": "jira", + "name": "list_webhooks", + "description": "List webhooks in jira.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.update_webhook", + "server": "jira", + "name": "update_webhook", + "description": "Update a webhook in jira.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.delete_webhook", + "server": "jira", + "name": "delete_webhook", + "description": "Delete a webhook in jira.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.create_attachment", + "server": "jira", + "name": "create_attachment", + "description": "Create a attachment in jira.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.get_attachment", + "server": "jira", + "name": "get_attachment", + "description": "Get a attachment in jira.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.list_attachments", + "server": "jira", + "name": "list_attachments", + "description": "List attachments in jira.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.update_attachment", + "server": "jira", + "name": "update_attachment", + "description": "Update a attachment in jira.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.delete_attachment", + "server": "jira", + "name": "delete_attachment", + "description": "Delete a attachment in jira.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.create_version", + "server": "jira", + "name": "create_version", + "description": "Create a version in jira.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.get_version", + "server": "jira", + "name": "get_version", + "description": "Get a version in jira.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.list_versions", + "server": "jira", + "name": "list_versions", + "description": "List versions in jira.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.update_version", + "server": "jira", + "name": "update_version", + "description": "Update a version in jira.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "jira.delete_version", + "server": "jira", + "name": "delete_version", + "description": "Delete a version in jira.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "jira", + "atlassian" + ] + }, + { + "id": "linear.create_cycle", + "server": "linear", + "name": "create_cycle", + "description": "Create a cycle in linear.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.get_cycle", + "server": "linear", + "name": "get_cycle", + "description": "Get a cycle in linear.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.list_cycles", + "server": "linear", + "name": "list_cycles", + "description": "List cycles in linear.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.update_cycle", + "server": "linear", + "name": "update_cycle", + "description": "Update a cycle in linear.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.delete_cycle", + "server": "linear", + "name": "delete_cycle", + "description": "Delete a cycle in linear.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.create_label", + "server": "linear", + "name": "create_label", + "description": "Create a label in linear.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.get_label", + "server": "linear", + "name": "get_label", + "description": "Get a label in linear.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.list_labels", + "server": "linear", + "name": "list_labels", + "description": "List labels in linear.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.update_label", + "server": "linear", + "name": "update_label", + "description": "Update a label in linear.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.delete_label", + "server": "linear", + "name": "delete_label", + "description": "Delete a label in linear.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.create_user", + "server": "linear", + "name": "create_user", + "description": "Create a user in linear.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.get_user", + "server": "linear", + "name": "get_user", + "description": "Get a user in linear.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.list_users", + "server": "linear", + "name": "list_users", + "description": "List users in linear.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.update_user", + "server": "linear", + "name": "update_user", + "description": "Update a user in linear.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.delete_user", + "server": "linear", + "name": "delete_user", + "description": "Delete a user in linear.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.create_comment", + "server": "linear", + "name": "create_comment", + "description": "Create a comment in linear.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.get_comment", + "server": "linear", + "name": "get_comment", + "description": "Get a comment in linear.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.list_comments", + "server": "linear", + "name": "list_comments", + "description": "List comments in linear.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.update_comment", + "server": "linear", + "name": "update_comment", + "description": "Update a comment in linear.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.delete_comment", + "server": "linear", + "name": "delete_comment", + "description": "Delete a comment in linear.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.create_attachment", + "server": "linear", + "name": "create_attachment", + "description": "Create a attachment in linear.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.get_attachment", + "server": "linear", + "name": "get_attachment", + "description": "Get a attachment in linear.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.list_attachments", + "server": "linear", + "name": "list_attachments", + "description": "List attachments in linear.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.update_attachment", + "server": "linear", + "name": "update_attachment", + "description": "Update a attachment in linear.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "linear" + ] + }, + { + "id": "linear.delete_attachment", + "server": "linear", + "name": "delete_attachment", + "description": "Delete a attachment in linear.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "linear" + ] + }, + { + "id": "slack.create_reminder", + "server": "slack", + "name": "create_reminder", + "description": "Create a reminder in slack.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.get_reminder", + "server": "slack", + "name": "get_reminder", + "description": "Get a reminder in slack.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.list_reminders", + "server": "slack", + "name": "list_reminders", + "description": "List reminders in slack.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.update_reminder", + "server": "slack", + "name": "update_reminder", + "description": "Update a reminder in slack.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.delete_reminder", + "server": "slack", + "name": "delete_reminder", + "description": "Delete a reminder in slack.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.create_canvase", + "server": "slack", + "name": "create_canvase", + "description": "Create a canvase in slack.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.get_canvase", + "server": "slack", + "name": "get_canvase", + "description": "Get a canvase in slack.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.list_canvases", + "server": "slack", + "name": "list_canvases", + "description": "List canvases in slack.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.update_canvase", + "server": "slack", + "name": "update_canvase", + "description": "Update a canvase in slack.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.delete_canvase", + "server": "slack", + "name": "delete_canvase", + "description": "Delete a canvase in slack.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.create_bookmark", + "server": "slack", + "name": "create_bookmark", + "description": "Create a bookmark in slack.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.get_bookmark", + "server": "slack", + "name": "get_bookmark", + "description": "Get a bookmark in slack.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.list_bookmarks", + "server": "slack", + "name": "list_bookmarks", + "description": "List bookmarks in slack.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.update_bookmark", + "server": "slack", + "name": "update_bookmark", + "description": "Update a bookmark in slack.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.delete_bookmark", + "server": "slack", + "name": "delete_bookmark", + "description": "Delete a bookmark in slack.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.create_usergroup", + "server": "slack", + "name": "create_usergroup", + "description": "Create a usergroup in slack.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.get_usergroup", + "server": "slack", + "name": "get_usergroup", + "description": "Get a usergroup in slack.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.list_usergroups", + "server": "slack", + "name": "list_usergroups", + "description": "List usergroups in slack.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.update_usergroup", + "server": "slack", + "name": "update_usergroup", + "description": "Update a usergroup in slack.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.delete_usergroup", + "server": "slack", + "name": "delete_usergroup", + "description": "Delete a usergroup in slack.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.create_file", + "server": "slack", + "name": "create_file", + "description": "Create a file in slack.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.get_file", + "server": "slack", + "name": "get_file", + "description": "Get a file in slack.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.list_files", + "server": "slack", + "name": "list_files", + "description": "List files in slack.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.update_file", + "server": "slack", + "name": "update_file", + "description": "Update a file in slack.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.delete_file", + "server": "slack", + "name": "delete_file", + "description": "Delete a file in slack.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.create_pin", + "server": "slack", + "name": "create_pin", + "description": "Create a pin in slack.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.get_pin", + "server": "slack", + "name": "get_pin", + "description": "Get a pin in slack.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.list_pins", + "server": "slack", + "name": "list_pins", + "description": "List pins in slack.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.update_pin", + "server": "slack", + "name": "update_pin", + "description": "Update a pin in slack.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "slack" + ] + }, + { + "id": "slack.delete_pin", + "server": "slack", + "name": "delete_pin", + "description": "Delete a pin in slack.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "slack" + ] + }, + { + "id": "notion.create_comment", + "server": "notion", + "name": "create_comment", + "description": "Create a comment in notion.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "notion" + ] + }, + { + "id": "notion.get_comment", + "server": "notion", + "name": "get_comment", + "description": "Get a comment in notion.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "notion" + ] + }, + { + "id": "notion.list_comments", + "server": "notion", + "name": "list_comments", + "description": "List comments in notion.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "notion" + ] + }, + { + "id": "notion.update_comment", + "server": "notion", + "name": "update_comment", + "description": "Update a comment in notion.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "notion" + ] + }, + { + "id": "notion.delete_comment", + "server": "notion", + "name": "delete_comment", + "description": "Delete a comment in notion.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "notion" + ] + }, + { + "id": "notion.create_users_token", + "server": "notion", + "name": "create_users_token", + "description": "Create a users token in notion.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "notion" + ] + }, + { + "id": "notion.get_users_token", + "server": "notion", + "name": "get_users_token", + "description": "Get a users token in notion.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "notion" + ] + }, + { + "id": "notion.list_users_tokens", + "server": "notion", + "name": "list_users_tokens", + "description": "List users tokens in notion.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "notion" + ] + }, + { + "id": "notion.update_users_token", + "server": "notion", + "name": "update_users_token", + "description": "Update a users token in notion.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "notion" + ] + }, + { + "id": "notion.delete_users_token", + "server": "notion", + "name": "delete_users_token", + "description": "Delete a users token in notion.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "notion" + ] + }, + { + "id": "notion.create_data_source", + "server": "notion", + "name": "create_data_source", + "description": "Create a data source in notion.", + "params": [ + "name", + "title", + "description", + "body" + ], + "source": "expanded", + "aliases": [ + "notion" + ] + }, + { + "id": "notion.get_data_source", + "server": "notion", + "name": "get_data_source", + "description": "Get a data source in notion.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "notion" + ] + }, + { + "id": "notion.list_data_sources", + "server": "notion", + "name": "list_data_sources", + "description": "List data sources in notion.", + "params": [ + "limit", + "cursor", + "status" + ], + "source": "expanded", + "aliases": [ + "notion" + ] + }, + { + "id": "notion.update_data_source", + "server": "notion", + "name": "update_data_source", + "description": "Update a data source in notion.", + "params": [ + "id", + "fields" + ], + "source": "expanded", + "aliases": [ + "notion" + ] + }, + { + "id": "notion.delete_data_source", + "server": "notion", + "name": "delete_data_source", + "description": "Delete a data source in notion.", + "params": [ + "id" + ], + "source": "expanded", + "aliases": [ + "notion" + ] + } +] diff --git a/experiments/mcp-tool-search/data/catalog_stats.json b/experiments/mcp-tool-search/data/catalog_stats.json new file mode 100644 index 00000000..14dba3c7 --- /dev/null +++ b/experiments/mcp-tool-search/data/catalog_stats.json @@ -0,0 +1,11 @@ +{ + "n_tools": 536, + "n_queries": 150, + "n_servers": 51, + "by_source": { + "public": 366, + "expanded": 170 + }, + "reproduction": true, + "zed_fixtures_public": false +} diff --git a/experiments/mcp-tool-search/data/queries.json b/experiments/mcp-tool-search/data/queries.json new file mode 100644 index 00000000..2c2d4b1f --- /dev/null +++ b/experiments/mcp-tool-search/data/queries.json @@ -0,0 +1,1386 @@ +[ + { + "id": "q001", + "query": "Create a GitHub issue", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "kind": "verb_product" + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q003", + "query": "Send a Slack message", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q004", + "query": "List open pull requests", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "kind": "resource" + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "gold": [ + "postgres.query" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q007", + "query": "Read a local file from disk", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "kind": "paraphrase" + }, + { + "id": "q008", + "query": "What files are in this folder", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "kind": "paraphrase" + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "kind": "paraphrase" + }, + { + "id": "q010", + "query": "Search the web with Brave", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q012", + "query": "Show git status", + "gold": [ + "git.git_status" + ], + "alts": [], + "kind": "identifier" + }, + { + "id": "q013", + "query": "Commit staged changes", + "gold": [ + "git.git_commit" + ], + "alts": [], + "kind": "paraphrase" + }, + { + "id": "q014", + "query": "Create a Linear issue", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "kind": "paraphrase" + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q017", + "query": "Merge the pull request", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "kind": "ambiguous" + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "kind": "resource" + }, + { + "id": "q020", + "query": "List Sentry issues", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "kind": "paraphrase" + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "kind": "paraphrase" + }, + { + "id": "q026", + "query": "Create a calendar event", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "kind": "resource" + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q029", + "query": "Run a docker container", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q030", + "query": "Show docker container logs", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "kind": "ambiguous" + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "kind": "ambiguous" + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q034", + "query": "Search Notion pages", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q035", + "query": "Create a Notion page", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q036", + "query": "Query a Notion database", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q040", + "query": "Run a SOQL query", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "kind": "identifier" + }, + { + "id": "q041", + "query": "List Shopify orders", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "kind": "paraphrase" + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q045", + "query": "Search Datadog logs", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q046", + "query": "List S3 buckets", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q047", + "query": "Upload an object to S3", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "kind": "paraphrase" + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q050", + "query": "List Vercel deployments", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q052", + "query": "Export images from Figma", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "kind": "paraphrase" + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q054", + "query": "Create an Okta user", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q058", + "query": "Run terraform plan", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "kind": "identifier" + }, + { + "id": "q059", + "query": "Create an Asana task", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q060", + "query": "Create a Trello card", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q062", + "query": "Create a Confluence page", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q063", + "query": "Send a Discord message", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q065", + "query": "Get a Redis key", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q067", + "query": "List SQLite tables", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q068", + "query": "Describe a postgres table", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q069", + "query": "github_create_issue", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "kind": "identifier" + }, + { + "id": "q070", + "query": "createIssue", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "kind": "identifier" + }, + { + "id": "q071", + "query": "list-directory", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "kind": "identifier" + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "kind": "paraphrase" + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "kind": "paraphrase" + }, + { + "id": "q074", + "query": "Convert time between timezones", + "gold": [ + "time.convert_time" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "gold": [ + "github.star_repository" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q076", + "query": "Fork a repository", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "kind": "resource" + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q078", + "query": "Create a gist", + "gold": [ + "github.create_gist" + ], + "alts": [], + "kind": "resource" + }, + { + "id": "q079", + "query": "Search GitHub code", + "gold": [ + "github.search_code" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q080", + "query": "List GitHub notifications", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "kind": "resource" + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "kind": "verb_product" + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "kind": "resource" + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "kind": "verb_product" + }, + { + "id": "q087", + "query": "Search Slack messages", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q088", + "query": "Create a Slack channel", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "kind": "identifier" + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q092", + "query": "List Linear teams", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q094", + "query": "Move a local file", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "kind": "paraphrase" + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "kind": "paraphrase" + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "kind": "paraphrase" + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "kind": "paraphrase" + }, + { + "id": "q098", + "query": "Search memory nodes", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q100", + "query": "Stage files with git add", + "gold": [ + "git.git_add" + ], + "alts": [], + "kind": "identifier" + }, + { + "id": "q101", + "query": "Create a Shopify product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q103", + "query": "Create a payment intent", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "kind": "resource" + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "kind": "paraphrase" + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q109", + "query": "Stop a docker container", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q110", + "query": "Build a docker image", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q112", + "query": "Type text into a browser input", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "kind": "paraphrase" + }, + { + "id": "q113", + "query": "List GitLab projects", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q117", + "query": "List Google Calendar events", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q118", + "query": "Delete a calendar event", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "kind": "resource" + }, + { + "id": "q119", + "query": "List Vercel projects", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q121", + "query": "Post a Figma comment", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q122", + "query": "List Okta users", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q123", + "query": "Range query Prometheus", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q125", + "query": "Apply terraform", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "kind": "identifier" + }, + { + "id": "q126", + "query": "List Asana tasks", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q129", + "query": "Search Confluence", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q132", + "query": "List Discord channels", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q133", + "query": "Set a Redis key", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q135", + "query": "Run a MySQL query", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q137", + "query": "Get an S3 object", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "gold": [ + "github.actions_list" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "gold": [ + "github.create_repository" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q143", + "query": "Checkout a git branch", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q144", + "query": "Show git log", + "gold": [ + "git.git_log" + ], + "alts": [], + "kind": "identifier" + }, + { + "id": "q145", + "query": "List postgres schemas", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "kind": "paraphrase" + }, + { + "id": "q148", + "query": "Write a new local file", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "kind": "paraphrase" + }, + { + "id": "q149", + "query": "Search GitHub issues", + "gold": [ + "github.search_issues" + ], + "alts": [], + "kind": "verb_product" + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "gold": [ + "github.get_me" + ], + "alts": [], + "kind": "paraphrase" + } +] diff --git a/experiments/mcp-tool-search/fixtures/zed-tweet-2026-10-20.png b/experiments/mcp-tool-search/fixtures/zed-tweet-2026-10-20.png new file mode 100644 index 00000000..2a0daeb5 Binary files /dev/null and b/experiments/mcp-tool-search/fixtures/zed-tweet-2026-10-20.png differ diff --git a/experiments/mcp-tool-search/harness.py b/experiments/mcp-tool-search/harness.py new file mode 100644 index 00000000..f38a6164 --- /dev/null +++ b/experiments/mcp-tool-search/harness.py @@ -0,0 +1,507 @@ +#!/usr/bin/env python3 +"""One-command MCP tool-search reproduction harness. + +This is a reproduction of Zed's unpublished 150×536 MCP tool-search +benchmark (tweet https://x.com/zeddotdev/status/2112588120558338261). +Zed's exact queries/tools/labels are not public. +""" + +from __future__ import annotations + +import argparse +import csv +import json +import os +import sys +import time +from collections import defaultdict +from pathlib import Path + +ROOT = Path(__file__).resolve().parent +sys.path.insert(0, str(ROOT)) + +from lib.catalog import ( # noqa: E402 + build_catalog, + build_distractors, + build_queries, + load_frozen, + materialize_codedb_corpus, + write_frozen, +) +from lib.codedb_client import CodedbSession # noqa: E402 +from lib.methods import CatalogSearch, MethodResult, MethodMeta, rrf, tokenize, VERBS # noqa: E402 +from lib.metrics import per_query_row, summarize # noqa: E402 + +RESULTS = ROOT / "results" +DATA = ROOT / "data" + + +def ensure_data() -> tuple[list[dict], list[dict]]: + if (DATA / "catalog.json").exists() and (DATA / "queries.json").exists(): + return load_frozen() + tools = build_catalog(536) + queries = build_queries(tools) + write_frozen(tools, queries) + return tools, queries + + +def write_method_outputs(name: str, queries: list[dict], mr: MethodResult, summary: dict) -> None: + RESULTS.mkdir(parents=True, exist_ok=True) + rows = [per_query_row(q, mr.rankings.get(q["id"], [])) for q in queries] + (RESULTS / f"{name}.json").write_text( + json.dumps({"summary": summary, "meta": mr.meta.__dict__, "rows": rows}, indent=2) + + "\n", + encoding="utf-8", + ) + with (RESULTS / f"{name}.csv").open("w", newline="", encoding="utf-8") as f: + w = csv.DictWriter( + f, + fieldnames=["id", "query", "kind", "hit1", "recall@1", "recall@3", "recall@5", "mrr@10", "ndcg@10", "ranking"], + ) + w.writeheader() + for r in rows: + w.writerow( + { + "id": r["id"], + "query": r["query"], + "kind": r["kind"], + "hit1": r["hit1"], + "recall@1": r["recall@1"], + "recall@3": r["recall@3"], + "recall@5": r["recall@5"], + "mrr@10": f"{r['mrr@10']:.4f}", + "ndcg@10": f"{r['ndcg@10']:.4f}", + "ranking": " ".join(r["ranking"]), + } + ) + + +def codedb_methods(session: CodedbSession, queries: list[dict], local: CatalogSearch) -> list[MethodResult]: + out: list[MethodResult] = [] + + def run(name: str, notes: str, fn) -> MethodResult: + rankings = {} + qms = [] + for q in queries: + t0 = time.perf_counter() + ranking = fn(q) + qms.append((time.perf_counter() - t0) * 1000.0) + rankings[q["id"]] = ranking[:10] + return MethodResult( + name, + rankings, + qms, + session.index_s, + None, + MethodMeta(name, False, False, notes), + ) + + out.append( + run( + "codedb_search", + "codedb_search via MCP (multi-word → searchContentRanked BM25+; single token → trigram substring)", + lambda q: session.search(q["query"], 20), + ) + ) + + def trigram_tokens(q: dict) -> list[str]: + toks = [t for t in tokenize(q["query"]) if t not in VERBS] + if not toks: + toks = tokenize(q["query"]) + lists = [] + for tok in toks[:4]: + lists.append(session.search(tok, 20)) + fused = rrf(lists) if lists else [] + return [r.tool_id for r in fused] + + out.append( + run( + "codedb_trigram_tokens", + "per-token codedb_search (single-token trigram path) fused with RRF", + trigram_tokens, + ) + ) + + def word_search(q: dict) -> list[str]: + toks = [t for t in tokenize(q["query"]) if t not in VERBS] or tokenize(q["query"]) + lists = [session.word(tok) for tok in toks[:3]] + fused = rrf(lists) if lists else [] + return [r.tool_id for r in fused] + + out.append(run("codedb_word", "codedb_word on distinctive query tokens, RRF", word_search)) + + def symbol_search(q: dict) -> list[str]: + toks = tokenize(q["query"]) + # Prefer the last content token; also try joined identifier. + candidates = [] + if toks: + candidates.append(toks[-1]) + if len(toks) >= 2: + candidates.append("_".join(toks[-2:])) + lists = [session.symbol(c) for c in candidates] + fused = rrf(lists) if lists else [] + return [r.tool_id for r in fused] + + out.append(run("codedb_symbol", "codedb_symbol fuzzy on guessed identifier tokens", symbol_search)) + + out.append( + run( + "codedb_find", + "codedb_find fuzzy filename search (server__name.py files)", + lambda q: session.find(q["query"], 20), + ) + ) + + # Hybrids with local weighted BM25 / trigram need those rankings. + # Recompute cheaply via CatalogSearch scorer already stored? We re-run weighted + # by calling local methods from earlier results if present. + return out + + +def fuse_hybrid(name: str, a: MethodResult, b: MethodResult, queries: list[dict], notes: str) -> MethodResult: + rankings = {} + qms = [] + for q in queries: + t0 = time.perf_counter() + fused = rrf([a.rankings.get(q["id"], []), b.rankings.get(q["id"], [])]) + qms.append((time.perf_counter() - t0) * 1000.0) + rankings[q["id"]] = [r.tool_id for r in fused][:10] + return MethodResult( + name, + rankings, + qms, + (a.index_s or 0) + (b.index_s or 0), + None, + MethodMeta(name, a.meta.needs_model or b.meta.needs_model, False, notes), + ) + + +def fmt_pct(x: float) -> str: + return f"{100.0 * x:.1f}%" + + +def fmt_ci(summary: dict, key: str) -> str: + mu = summary[key] + lo = summary[f"{key}_lo"] + hi = summary[f"{key}_hi"] + return f"{fmt_pct(mu)} [{fmt_pct(lo)}, {fmt_pct(hi)}]" + + +def write_results_md( + summaries: list[tuple[MethodResult, dict]], + scaling: list[dict], + gaps_preview: str, +) -> None: + lines = [ + "# MCP tool search — reproduction results", + "", + "**This is a reproduction, not Zed's unpublished benchmark.**", + "", + "Zed posted on 20 Oct 2026 that they tested 25 approaches on 150 queries", + "against 536 tools and shipped weighted BM25 with a few normalization", + "rules ([tweet](https://x.com/zeddotdev/status/2112588120558338261)).", + "The tweet image is a meme, not a write-up. The Zed blog has no post.", + "The `zed-industries/zed` repo has no public tool-search fixtures", + "(only an unrelated edit-prediction BM25 retriever).", + "", + "Catalog: 536 tools from real public MCP servers (GitHub, filesystem,", + "memory, postgres, Slack, Linear, Jira, Notion, Playwright, Docker,", + "Kubernetes, Stripe, …) plus expanded tools from the same families.", + "Queries: 150 hand-labelled natural-language / identifier queries.", + "See `GAPS.md` and `data/catalog_stats.json`.", + "", + "Metrics are means over 150 queries with 2000-resample bootstrap 95% CIs.", + "Differences of a few points are usually noise.", + "", + "", + "Cost tables are produced by `cost_eval.py` after this file is written.", + "", + "", + "## Headline table (n=536 tools, 150 queries)", + "", + "| Approach | R@1 | R@3 | R@5 | MRR@10 | nDCG@10 | p50 ms | p95 ms | index s | model/API |", + "|---|---:|---:|---:|---:|---:|---:|---:|---:|---|", + ] + for mr, s in summaries: + if mr.skipped: + lines.append( + f"| {mr.name} | — | — | — | — | — | — | — | — | skipped: {mr.skipped} |" + ) + continue + model = "model" if mr.meta.needs_model else ("API" if mr.meta.needs_api else "local") + lines.append( + "| {name} | {r1} | {r3} | {r5} | {mrr:.3f} | {nd:.3f} | {p50:.2f} | {p95:.2f} | {ix:.3f} | {model} |".format( + name=mr.name, + r1=fmt_pct(s["recall@1"]), + r3=fmt_pct(s["recall@3"]), + r5=fmt_pct(s["recall@5"]), + mrr=s["mrr@10"], + nd=s["ndcg@10"], + p50=s["p50_ms"], + p95=s["p95_ms"], + ix=s["index_s"], + model=model, + ) + ) + lines += [ + "", + "### Bootstrap 95% CIs for Recall@1 / MRR@10", + "", + "| Approach | Recall@1 | MRR@10 |", + "|---|---|---|", + ] + for mr, s in summaries: + if mr.skipped: + continue + lines.append(f"| {mr.name} | {fmt_ci(s, 'recall@1')} | {s['mrr@10']:.3f} [{s['mrr@10_lo']:.3f}, {s['mrr@10_hi']:.3f}] |") + + # Embedding vs trigram + by_name = {mr.name: (mr, s) for mr, s in summaries} + lines += [ + "", + "## Embedding vs trigram (same queries)", + "", + "Head-to-head on the 150-query / 536-tool set. `char_trigram` is an", + "in-process character-trigram cosine (codedb-like). `codedb_trigram_tokens`", + "is codedb's real trigram path (one search per token). Dense embedding is", + "`embed_bge_small` when fastembed is available, else hashed n-grams.", + "", + ] + pair_names = [ + "char_trigram", + "codedb_trigram_tokens", + "embed_bge_small", + "hashed_ngram_embed", + "hybrid_rrf_trigram_embed", + "hybrid_rrf_bm25_trigram", + "hybrid_rrf_weighted_embed", + ] + lines += [ + "| Approach | R@1 | R@3 | MRR@10 | nDCG@10 | p50 ms |", + "|---|---:|---:|---:|---:|---:|", + ] + for n in pair_names: + if n not in by_name: + continue + mr, s = by_name[n] + if mr.skipped: + lines.append(f"| {n} | skipped | — | — | — | — |") + continue + lines.append( + f"| {n} | {fmt_pct(s['recall@1'])} | {fmt_pct(s['recall@3'])} | {s['mrr@10']:.3f} | {s['ndcg@10']:.3f} | {s['p50_ms']:.2f} |" + ) + + lines += [ + "", + "## Where codedb ranks", + "", + "codedb is a **code-intelligence** engine. Here it is asked to retrieve", + "MCP tools from a synthetic catalog (one Python file per tool). That is", + "off-label: no field weights, no verb down-weighting, ranking is for source", + "files (centrality / call-graph) rather than tool metadata.", + "", + ] + codedb_names = [n for n, _ in by_name.items() if n.startswith("codedb") or n.startswith("hybrid_codedb")] + scored = [] + for n in codedb_names: + mr, s = by_name[n] + if not mr.skipped: + scored.append((s["mrr@10"], n, s)) + scored.sort(reverse=True) + if scored: + lines.append("Best codedb-family method by MRR@10: **{}** ({:.3f}).".format(scored[0][1], scored[0][0])) + lines.append("") + + if scaling: + lines += [ + "## Scaling (536 → ~2k → ~5k tools)", + "", + "Gold tools stay fixed; distractors reuse the same create/get/list verbs.", + "", + "| n tools | Approach | R@1 | R@5 | MRR@10 | p50 ms |", + "|---:|---|---:|---:|---:|---:|", + ] + for row in scaling: + lines.append( + f"| {row['n_tools']} | {row['name']} | {fmt_pct(row['recall@1'])} | {fmt_pct(row['recall@5'])} | {row['mrr@10']:.3f} | {row['p50_ms']:.2f} |" + ) + + lines += [ + "", + "## Approaches we could not run", + "", + gaps_preview, + "", + "Full gap list: [`GAPS.md`](GAPS.md).", + "", + "## Raw outputs", + "", + "Per-approach JSON/CSV lives in `results/`. Combined table:", + "`results/summary.json`. Cost: `results/cost.json`. Live: `results/live_agent.json`.", + "", + ] + (ROOT / "results.md").write_text("\n".join(lines) + "\n", encoding="utf-8") + (RESULTS / "results.md").write_text("\n".join(lines) + "\n", encoding="utf-8") + + +def run_scaling(base_tools: list[dict], queries: list[dict]) -> list[dict]: + rows = [] + sizes = [536, 2000, 5000] + methods = ("bm25_plain", "bm25_weighted", "char_trigram", "tfidf", "hybrid_rrf_weighted_trigram") + for n in sizes: + if n == 536: + tools = base_tools + else: + extra = build_distractors(n - len(base_tools), seed=20261020 + n) + tools = base_tools + extra + search = CatalogSearch(tools) + results = search.run_all(queries, want_dense=False) + for mr in results: + if mr.name not in methods or mr.skipped: + continue + s = summarize(queries, mr.rankings, mr.query_ms, mr.index_s, mr.rss_mb) + rows.append({"n_tools": n, "name": mr.name, **{k: s[k] for k in ("recall@1", "recall@3", "recall@5", "mrr@10", "ndcg@10", "p50_ms", "p95_ms", "index_s")}}) + return rows + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--skip-codedb", action="store_true") + ap.add_argument("--skip-dense", action="store_true") + ap.add_argument("--skip-scaling", action="store_true") + ap.add_argument("--codedb-bin", default=str(ROOT / "bin" / "codedb")) + args = ap.parse_args() + + tools, queries = ensure_data() + print(f"catalog={len(tools)} queries={len(queries)} servers={len({t['server'] for t in tools})}") + + search = CatalogSearch(tools) + local = search.run_all(queries, want_dense=not args.skip_dense) + all_results = list(local) + + codedb_ran = False + if not args.skip_codedb and Path(args.codedb_bin).exists(): + corpus = ROOT / "catalog_index" + print("materializing codedb corpus…") + materialize_codedb_corpus(tools, corpus) + sess = CodedbSession(str(Path(args.codedb_bin).resolve()), corpus) + print("starting codedb mcp…") + sess.start() + if not sess.ready: + print("codedb skipped:", sess.error) + all_results.append( + MethodResult( + "codedb_search", + {}, + [], + 0.0, + None, + MethodMeta("codedb_search", False, False, "codedb MCP"), + skipped=sess.error or "not ready", + ) + ) + else: + print(f"codedb ready in {sess.index_s:.2f}s") + codedb_ran = True + try: + cd = codedb_methods(sess, queries, search) + all_results.extend(cd) + by = {m.name: m for m in all_results} + if "codedb_search" in by and "bm25_weighted" in by: + all_results.append( + fuse_hybrid( + "hybrid_codedb_weighted_bm25", + by["codedb_search"], + by["bm25_weighted"], + queries, + "RRF codedb_search + weighted BM25", + ) + ) + if "codedb_trigram_tokens" in by and "bm25_weighted" in by: + all_results.append( + fuse_hybrid( + "hybrid_codedb_trigram_bm25", + by["codedb_trigram_tokens"], + by["bm25_weighted"], + queries, + "RRF codedb trigram tokens + weighted BM25", + ) + ) + embed = by.get("embed_bge_small") or by.get("hashed_ngram_embed") + if embed and not embed.skipped and "codedb_search" in by: + all_results.append( + fuse_hybrid( + "hybrid_codedb_embed", + by["codedb_search"], + embed, + queries, + "RRF codedb_search + embedding", + ) + ) + finally: + sess.close() + elif not args.skip_codedb: + all_results.append( + MethodResult( + "codedb_search", + {}, + [], + 0.0, + None, + MethodMeta("codedb_search", False, False, "codedb MCP"), + skipped=f"binary not found: {args.codedb_bin}", + ) + ) + + summaries: list[tuple[MethodResult, dict]] = [] + for mr in all_results: + if mr.skipped: + summaries.append((mr, {})) + continue + s = summarize(queries, mr.rankings, mr.query_ms, mr.index_s, mr.rss_mb) + write_method_outputs(mr.name, queries, mr, s) + summaries.append((mr, s)) + print(f"{mr.name:32s} R@1={s['recall@1']:.3f} R@5={s['recall@5']:.3f} MRR={s['mrr@10']:.3f} p50={s['p50_ms']:.2f}ms") + + scaling: list[dict] = [] + if not args.skip_scaling: + print("scaling…") + scaling = run_scaling(tools, queries) + (RESULTS / "scaling.json").write_text(json.dumps(scaling, indent=2) + "\n", encoding="utf-8") + + combined = [] + for mr, s in summaries: + item = {"name": mr.name, "skipped": mr.skipped, "notes": mr.meta.notes, "needs_model": mr.meta.needs_model} + item.update(s) + combined.append(item) + RESULTS.mkdir(parents=True, exist_ok=True) + (RESULTS / "summary.json").write_text( + json.dumps( + { + "reproduction": True, + "zed_fixtures_public": False, + "n_tools": len(tools), + "n_queries": len(queries), + "codedb_ran": codedb_ran, + "methods": combined, + "scaling": scaling, + }, + indent=2, + ) + + "\n", + encoding="utf-8", + ) + + gaps_preview = ( + "- Zed's exact 25-approach list, queries, tools, labels, field weights, and BM25 parameters are not public.\n" + "- LLM routing and trained decision models (LightGBM etc.) were not run: no API key / no labelled train split from Zed.\n" + "- Cross-encoder rerankers and API embedding models were not run unless those packages/keys are present.\n" + "- A second local dense model (gte/nomic) was not downloaded to keep the one-command path small.\n" + ) + write_results_md(summaries, scaling, gaps_preview) + print("wrote results.md") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/experiments/mcp-tool-search/lib/__init__.py b/experiments/mcp-tool-search/lib/__init__.py new file mode 100644 index 00000000..cd635170 --- /dev/null +++ b/experiments/mcp-tool-search/lib/__init__.py @@ -0,0 +1 @@ +"""MCP tool-search reproduction (Zed Oct 20 2026 tweet).""" diff --git a/experiments/mcp-tool-search/lib/catalog.py b/experiments/mcp-tool-search/lib/catalog.py new file mode 100644 index 00000000..09330f89 --- /dev/null +++ b/experiments/mcp-tool-search/lib/catalog.py @@ -0,0 +1,895 @@ +"""Faithful reproduction catalog: ~536 MCP tools + 150 labelled queries. + +Zed's exact queries, tools, and labels are not public (the Oct 20 2026 post +linked a meme, not a write-up or fixture repo). This catalog is built from +real public MCP server tool names/descriptions (GitHub, official +modelcontextprotocol servers, Linear, Slack, Postgres, filesystem, etc.) +and padded to 536 with additional realistic tools from the same families. + +THIS IS A REPRODUCTION, not Zed's unpublished benchmark. +""" + +from __future__ import annotations + +import json +import random +import re +from pathlib import Path +from typing import Any + +ROOT = Path(__file__).resolve().parents[1] +DATA = ROOT / "data" + +# Common MCP action verbs — the ranking failure mode Zed/StackOne describe. +VERBS = ( + "create", + "get", + "list", + "update", + "delete", + "search", + "send", + "add", + "remove", + "set", + "fetch", + "post", + "put", + "run", + "query", +) + +SERVER_ALIASES: dict[str, list[str]] = { + "github": ["github", "gh", "git hub"], + "gitlab": ["gitlab", "gl"], + "bitbucket": ["bitbucket", "bb"], + "slack": ["slack"], + "discord": ["discord"], + "linear": ["linear"], + "jira": ["jira", "atlassian"], + "asana": ["asana"], + "trello": ["trello"], + "clickup": ["clickup", "click up"], + "notion": ["notion"], + "confluence": ["confluence"], + "filesystem": ["filesystem", "files", "disk", "folder", "directory"], + "memory": ["memory", "knowledge graph", "remember"], + "postgres": ["postgres", "postgresql", "sql", "database"], + "sqlite": ["sqlite"], + "mysql": ["mysql"], + "mongodb": ["mongodb", "mongo"], + "redis": ["redis"], + "elasticsearch": ["elasticsearch", "elastic"], + "fetch": ["fetch", "http", "url", "webpage"], + "git": ["git"], + "playwright": ["playwright", "browser"], + "puppeteer": ["puppeteer"], + "docker": ["docker", "container"], + "kubernetes": ["kubernetes", "k8s"], + "sentry": ["sentry"], + "datadog": ["datadog"], + "stripe": ["stripe"], + "shopify": ["shopify"], + "salesforce": ["salesforce", "sfdc"], + "hubspot": ["hubspot"], + "zendesk": ["zendesk"], + "pagerduty": ["pagerduty", "pager duty"], + "gmail": ["gmail", "email", "mail"], + "google_drive": ["google drive", "gdrive", "drive"], + "google_calendar": ["google calendar", "gcal", "calendar"], + "brave_search": ["brave", "web search"], + "sequential_thinking": ["sequential thinking", "think step"], + "time": ["time", "timezone", "clock"], + "aws_s3": ["s3", "aws s3", "bucket"], + "cloudflare": ["cloudflare"], + "vercel": ["vercel"], + "supabase": ["supabase"], + "figma": ["figma"], + "twilio": ["twilio", "sms"], + "okta": ["okta"], + "prometheus": ["prometheus"], + "grafana": ["grafana"], + "circleci": ["circleci", "ci"], + "terraform": ["terraform"], +} + + +def _tool( + server: str, + name: str, + description: str, + params: list[str], + *, + source: str, +) -> dict[str, Any]: + tid = f"{server}.{name}" + return { + "id": tid, + "server": server, + "name": name, + "description": description, + "params": params, + "source": source, + "aliases": SERVER_ALIASES.get(server, [server.replace("_", " ")]), + } + + +def _real_core() -> list[dict[str, Any]]: + """Hand-authored tools from public MCP servers / published schemas.""" + rows: list[tuple[str, str, str, str, str]] = [ + # GitHub official MCP (classic + current names) + ("github", "get_me", "Get details of the authenticated GitHub user and their context.", "include_auth", "public"), + ("github", "get_teams", "Get teams for the authenticated GitHub user.", "", "public"), + ("github", "get_team_members", "List members of a GitHub team.", "org team_slug", "public"), + ("github", "search_repositories", "Search GitHub repositories by name, description, or topic.", "query perPage sort", "public"), + ("github", "get_file_contents", "Get file or directory contents from a GitHub repository.", "owner repo path ref", "public"), + ("github", "list_commits", "List commits on a GitHub repository branch.", "owner repo sha page", "public"), + ("github", "search_code", "Search code across GitHub repositories.", "query perPage", "public"), + ("github", "search_commits", "Search commits across GitHub.", "query", "public"), + ("github", "get_commit", "Get a single GitHub commit by SHA.", "owner repo sha", "public"), + ("github", "get_file_blame", "Get git blame for a file in a GitHub repository.", "owner repo path", "public"), + ("github", "list_branches", "List branches in a GitHub repository.", "owner repo", "public"), + ("github", "list_tags", "List git tags in a GitHub repository.", "owner repo", "public"), + ("github", "get_tag", "Get a git tag from a GitHub repository.", "owner repo tag", "public"), + ("github", "list_releases", "List GitHub releases for a repository.", "owner repo", "public"), + ("github", "get_latest_release", "Get the latest GitHub release.", "owner repo", "public"), + ("github", "get_release_by_tag", "Get a GitHub release by tag name.", "owner repo tag", "public"), + ("github", "create_or_update_file", "Create or update a single file in a GitHub repository.", "owner repo path content message branch", "public"), + ("github", "create_repository", "Create a new GitHub repository.", "name description private", "public"), + ("github", "delete_repository", "Delete a GitHub repository.", "owner repo", "public"), + ("github", "fork_repository", "Fork a GitHub repository.", "owner repo organization", "public"), + ("github", "create_branch", "Create a branch in a GitHub repository.", "owner repo branch from_branch", "public"), + ("github", "push_files", "Push multiple files to a GitHub repository in one commit.", "owner repo branch files message", "public"), + ("github", "delete_file", "Delete a file from a GitHub repository.", "owner repo path message branch", "public"), + ("github", "list_starred_repositories", "List repositories starred by a GitHub user.", "username", "public"), + ("github", "star_repository", "Star a GitHub repository.", "owner repo", "public"), + ("github", "unstar_repository", "Unstar a GitHub repository.", "owner repo", "public"), + ("github", "list_repository_collaborators", "List collaborators on a GitHub repository.", "owner repo", "public"), + ("github", "get_repository_tree", "Get the git tree of a GitHub repository.", "owner repo tree_sha recursive", "public"), + ("github", "create_issue", "Create an issue in a GitHub repository.", "owner repo title body labels assignees", "public"), + ("github", "issue_write", "Create or update a GitHub issue (consolidated write tool).", "method owner repo title body issue_number", "public"), + ("github", "issue_read", "Read a GitHub issue by number.", "method owner repo issue_number", "public"), + ("github", "search_issues", "Search GitHub issues.", "query owner repo", "public"), + ("github", "list_issues", "List issues in a GitHub repository.", "owner repo state labels", "public"), + ("github", "add_issue_comment", "Add a comment to a GitHub issue.", "owner repo issue_number body", "public"), + ("github", "update_issue_comment", "Update a comment on a GitHub issue.", "owner repo comment_id body", "public"), + ("github", "create_pull_request", "Open a new pull request on GitHub.", "owner repo title head base body draft", "public"), + ("github", "list_pull_requests", "List pull requests in a GitHub repository.", "owner repo state", "public"), + ("github", "search_pull_requests", "Search GitHub pull requests.", "query", "public"), + ("github", "pull_request_read", "Read a GitHub pull request by number.", "owner repo pull_number", "public"), + ("github", "update_pull_request", "Update title, body, or state of a GitHub pull request.", "owner repo pull_number title body state", "public"), + ("github", "merge_pull_request", "Merge a GitHub pull request.", "owner repo pull_number merge_method", "public"), + ("github", "update_pull_request_branch", "Update a pull request branch with the base branch.", "owner repo pull_number", "public"), + ("github", "pull_request_review_write", "Create or submit a GitHub pull request review.", "method owner repo pull_number body event", "public"), + ("github", "add_comment_to_pending_review", "Add a comment to a pending GitHub pull request review.", "owner repo pull_number path body line", "public"), + ("github", "assign_copilot_to_issue", "Assign GitHub Copilot to work on an issue.", "owner repo issue_number", "public"), + ("github", "request_copilot_review", "Request a Copilot review on a GitHub pull request.", "owner repo pull_number", "public"), + ("github", "get_code_scanning_alert", "Get a GitHub code scanning alert.", "owner repo alert_number", "public"), + ("github", "list_code_scanning_alerts", "List GitHub code scanning alerts.", "owner repo state", "public"), + ("github", "get_secret_scanning_alert", "Get a GitHub secret scanning alert.", "owner repo alert_number", "public"), + ("github", "list_secret_scanning_alerts", "List GitHub secret scanning alerts.", "owner repo", "public"), + ("github", "get_dependabot_alert", "Get a Dependabot alert.", "owner repo alert_number", "public"), + ("github", "list_dependabot_alerts", "List Dependabot alerts for a repository.", "owner repo", "public"), + ("github", "list_notifications", "List GitHub notifications for the authenticated user.", "all participating", "public"), + ("github", "get_notification_details", "Get details of a GitHub notification thread.", "thread_id", "public"), + ("github", "dismiss_notification", "Dismiss or mark a GitHub notification as done.", "thread_id", "public"), + ("github", "mark_all_notifications_read", "Mark all GitHub notifications as read.", "", "public"), + ("github", "list_discussions", "List GitHub discussions in a repository.", "owner repo", "public"), + ("github", "get_discussion", "Get a GitHub discussion.", "owner repo discussion_number", "public"), + ("github", "get_discussion_comments", "Get comments on a GitHub discussion.", "owner repo discussion_number", "public"), + ("github", "actions_list", "List GitHub Actions workflows.", "owner repo", "public"), + ("github", "actions_get", "Get a GitHub Actions workflow or run.", "owner repo workflow_id", "public"), + ("github", "actions_run_trigger", "Trigger a GitHub Actions workflow dispatch.", "owner repo workflow_id ref inputs", "public"), + ("github", "actions_get_job_logs", "Get logs for a GitHub Actions job.", "owner repo job_id", "public"), + ("github", "list_gists", "List GitHub gists for a user.", "username", "public"), + ("github", "get_gist", "Get a GitHub gist by id.", "gist_id", "public"), + ("github", "create_gist", "Create a GitHub gist.", "description files public", "public"), + ("github", "update_gist", "Update a GitHub gist.", "gist_id description files", "public"), + ("github", "projects_list", "List GitHub Projects.", "owner owner_type", "public"), + ("github", "projects_get", "Get a GitHub Project.", "project_number owner", "public"), + ("github", "projects_write", "Create or update a GitHub Project item.", "method project_id title", "public"), + ("github", "get_label", "Get a GitHub label.", "owner repo name", "public"), + ("github", "list_labels", "List labels in a GitHub repository.", "owner repo", "public"), + ("github", "label_write", "Create, update, or delete a GitHub label.", "method owner repo name color", "public"), + ("github", "search_users", "Search GitHub users.", "query", "public"), + ("github", "search_orgs", "Search GitHub organizations.", "query", "public"), + # Official filesystem / memory / fetch / git / postgres / sequential-thinking + ("filesystem", "read_file", "Read the complete contents of a file from the local filesystem.", "path", "public"), + ("filesystem", "read_text_file", "Read a text file from the local filesystem.", "path head tail", "public"), + ("filesystem", "read_media_file", "Read an image or audio file from the local filesystem.", "path", "public"), + ("filesystem", "write_file", "Create a new file or completely overwrite an existing file.", "path content", "public"), + ("filesystem", "edit_file", "Make a line-based edit to a text file using old and new strings.", "path old_string new_string", "public"), + ("filesystem", "create_directory", "Create a new directory or ensure a directory exists.", "path", "public"), + ("filesystem", "list_directory", "Get a detailed listing of files and directories.", "path", "public"), + ("filesystem", "list_directory_with_sizes", "Get a listing of files and directories with sizes.", "path", "public"), + ("filesystem", "directory_tree", "Get a recursive tree view of files and directories.", "path", "public"), + ("filesystem", "move_file", "Move or rename files and directories.", "source destination", "public"), + ("filesystem", "search_files", "Recursively search for files matching a pattern.", "path pattern", "public"), + ("filesystem", "get_file_info", "Retrieve metadata about a file or directory.", "path", "public"), + ("filesystem", "list_allowed_directories", "List directories the filesystem server is allowed to access.", "", "public"), + ("memory", "create_entities", "Create multiple entities in the knowledge graph.", "entities", "public"), + ("memory", "create_relations", "Create relations between entities in the knowledge graph.", "relations", "public"), + ("memory", "add_observations", "Add observations to existing entities in the knowledge graph.", "observations", "public"), + ("memory", "delete_entities", "Delete entities and their relations from the knowledge graph.", "entityNames", "public"), + ("memory", "delete_observations", "Delete specific observations from entities.", "deletions", "public"), + ("memory", "delete_relations", "Delete relations from the knowledge graph.", "relations", "public"), + ("memory", "read_graph", "Read the entire knowledge graph.", "", "public"), + ("memory", "search_nodes", "Search for nodes in the knowledge graph based on a query.", "query", "public"), + ("memory", "open_nodes", "Open specific nodes in the knowledge graph by name.", "names", "public"), + ("postgres", "query", "Run a read-only SQL query against PostgreSQL.", "sql", "public"), + ("postgres", "list_tables", "List tables in the connected PostgreSQL database.", "schema", "public"), + ("postgres", "describe_table", "Describe columns and types of a PostgreSQL table.", "schema table", "public"), + ("postgres", "list_schemas", "List schemas in the PostgreSQL database.", "", "public"), + ("postgres", "explain_query", "EXPLAIN a PostgreSQL query plan.", "sql", "public"), + ("fetch", "fetch", "Fetch a URL and return the contents as markdown or HTML.", "url max_length raw", "public"), + ("git", "git_status", "Show the working tree status of a git repository.", "repo_path", "public"), + ("git", "git_diff_unstaged", "Show unstaged git differences.", "repo_path", "public"), + ("git", "git_diff_staged", "Show staged git differences.", "repo_path", "public"), + ("git", "git_diff", "Show git diff between branches or commits.", "repo_path target", "public"), + ("git", "git_commit", "Record staged changes to the git repository.", "repo_path message", "public"), + ("git", "git_add", "Add file contents to the git staging area.", "repo_path files", "public"), + ("git", "git_reset", "Unstage all staged changes.", "repo_path", "public"), + ("git", "git_log", "Show commit logs.", "repo_path max_count", "public"), + ("git", "git_create_branch", "Create a new git branch.", "repo_path branch_name base_branch", "public"), + ("git", "git_checkout", "Switch git branches.", "repo_path branch_name", "public"), + ("git", "git_show", "Show a git commit.", "repo_path revision", "public"), + ("git", "git_branch", "List git branches.", "repo_path", "public"), + ("sequential_thinking", "sequentialthinking", "A detailed tool for dynamic and reflective problem-solving through thoughts.", "thought nextThoughtNeeded thoughtNumber totalThoughts", "public"), + ("time", "get_current_time", "Get the current time in a timezone.", "timezone", "public"), + ("time", "convert_time", "Convert time between timezones.", "source_timezone time target_timezone", "public"), + ("brave_search", "brave_web_search", "Search the web using Brave Search.", "query count offset", "public"), + ("brave_search", "brave_local_search", "Search for local businesses and places using Brave.", "query count", "public"), + # Slack official-style + ("slack", "slack_post_message", "Post a message to a Slack channel.", "channel_id text", "public"), + ("slack", "slack_reply_to_thread", "Reply to a Slack thread.", "channel_id thread_ts text", "public"), + ("slack", "slack_add_reaction", "Add an emoji reaction to a Slack message.", "channel_id timestamp emoji", "public"), + ("slack", "slack_get_channel_history", "Get recent messages from a Slack channel.", "channel_id limit", "public"), + ("slack", "slack_get_thread_replies", "Get replies in a Slack thread.", "channel_id thread_ts", "public"), + ("slack", "slack_list_channels", "List Slack channels in the workspace.", "limit cursor", "public"), + ("slack", "slack_get_users", "Get a list of Slack users.", "cursor limit", "public"), + ("slack", "slack_get_user_profile", "Get a Slack user's profile.", "user_id", "public"), + ("slack", "slack_join_channel", "Join a Slack channel.", "channel_id", "public"), + ("slack", "slack_leave_channel", "Leave a Slack channel.", "channel_id", "public"), + ("slack", "slack_create_channel", "Create a Slack channel.", "name is_private", "public"), + ("slack", "slack_set_channel_topic", "Set the topic of a Slack channel.", "channel_id topic", "public"), + ("slack", "slack_upload_file", "Upload a file to Slack.", "channel_id filename content", "public"), + ("slack", "slack_search_messages", "Search Slack messages.", "query count", "public"), + ("slack", "slack_list_users", "List Slack workspace users.", "limit cursor", "public"), + # Linear + ("linear", "linear_create_issue", "Create a Linear issue in a team.", "title description team teamId priority labels", "public"), + ("linear", "linear_update_issue", "Update a Linear issue.", "id title description state priority", "public"), + ("linear", "linear_list_issues", "List Linear issues, optionally filtered.", "team state assignee", "public"), + ("linear", "linear_get_issue", "Get a Linear issue by id or identifier.", "id", "public"), + ("linear", "linear_search_issues", "Search Linear issues by text.", "query", "public"), + ("linear", "linear_create_comment", "Add a comment to a Linear issue.", "issueId body", "public"), + ("linear", "linear_list_teams", "List Linear teams.", "", "public"), + ("linear", "linear_list_projects", "List Linear projects.", "team", "public"), + ("linear", "linear_create_project", "Create a Linear project.", "name team description", "public"), + ("linear", "linear_assign_issue", "Assign a Linear issue to a user.", "id assigneeId", "public"), + # Jira + ("jira", "jira_create_issue", "Create a Jira issue or ticket in a project.", "project key summary description issuetype", "public"), + ("jira", "jira_update_issue", "Update an existing Jira issue.", "issue_key fields", "public"), + ("jira", "jira_get_issue", "Get a Jira issue by key.", "issue_key", "public"), + ("jira", "jira_search_issues", "Search Jira issues with JQL.", "jql maxResults", "public"), + ("jira", "jira_add_comment", "Add a comment to a Jira issue.", "issue_key body", "public"), + ("jira", "jira_transition_issue", "Transition a Jira issue to a new status.", "issue_key transition", "public"), + ("jira", "jira_assign_issue", "Assign a Jira issue.", "issue_key account_id", "public"), + ("jira", "jira_list_projects", "List Jira projects.", "", "public"), + ("jira", "jira_create_sprint", "Create a Jira Software sprint.", "board_id name startDate endDate", "public"), + ("jira", "jira_add_worklog", "Add a worklog to a Jira issue.", "issue_key timeSpent comment", "public"), + # Notion + ("notion", "notion_search", "Search Notion pages and databases.", "query filter", "public"), + ("notion", "notion_get_page", "Retrieve a Notion page.", "page_id", "public"), + ("notion", "notion_create_page", "Create a Notion page in a parent page or database.", "parent properties children", "public"), + ("notion", "notion_update_page", "Update a Notion page properties.", "page_id properties", "public"), + ("notion", "notion_get_block", "Get a Notion block.", "block_id", "public"), + ("notion", "notion_append_block", "Append children blocks to a Notion block.", "block_id children", "public"), + ("notion", "notion_query_database", "Query a Notion database.", "database_id filter sorts", "public"), + ("notion", "notion_create_database", "Create a Notion database.", "parent title properties", "public"), + ("notion", "notion_list_users", "List Notion workspace users.", "", "public"), + ("notion", "notion_get_comments", "Get comments on a Notion page.", "block_id", "public"), + # Playwright / puppeteer + ("playwright", "browser_navigate", "Navigate the browser to a URL.", "url", "public"), + ("playwright", "browser_snapshot", "Capture an accessibility snapshot of the current page.", "", "public"), + ("playwright", "browser_click", "Click an element on the page.", "element ref", "public"), + ("playwright", "browser_type", "Type text into an editable element.", "element ref text", "public"), + ("playwright", "browser_fill_form", "Fill multiple form fields.", "fields", "public"), + ("playwright", "browser_take_screenshot", "Take a screenshot of the page or an element.", "filename fullPage", "public"), + ("playwright", "browser_evaluate", "Evaluate JavaScript on the page.", "function", "public"), + ("playwright", "browser_tab_list", "List open browser tabs.", "", "public"), + ("playwright", "browser_tab_new", "Open a new browser tab.", "url", "public"), + ("playwright", "browser_tab_close", "Close a browser tab.", "index", "public"), + ("playwright", "browser_wait_for", "Wait for text or time to pass.", "text time", "public"), + ("playwright", "browser_press_key", "Press a key on the page.", "key", "public"), + ("puppeteer", "puppeteer_navigate", "Navigate to a URL with Puppeteer.", "url", "public"), + ("puppeteer", "puppeteer_screenshot", "Take a screenshot with Puppeteer.", "name selector", "public"), + ("puppeteer", "puppeteer_click", "Click an element with Puppeteer.", "selector", "public"), + ("puppeteer", "puppeteer_fill", "Fill an input with Puppeteer.", "selector value", "public"), + ("puppeteer", "puppeteer_evaluate", "Execute JavaScript with Puppeteer.", "script", "public"), + # Docker / k8s + ("docker", "docker_ps", "List running Docker containers.", "all", "public"), + ("docker", "docker_logs", "Fetch logs from a Docker container.", "container tail", "public"), + ("docker", "docker_run", "Run a Docker container.", "image name ports env", "public"), + ("docker", "docker_stop", "Stop a Docker container.", "container", "public"), + ("docker", "docker_rm", "Remove a Docker container.", "container force", "public"), + ("docker", "docker_images", "List Docker images.", "", "public"), + ("docker", "docker_build", "Build a Docker image.", "path tag", "public"), + ("docker", "docker_exec", "Run a command in a running Docker container.", "container command", "public"), + ("kubernetes", "k8s_list_pods", "List Kubernetes pods in a namespace.", "namespace", "public"), + ("kubernetes", "k8s_get_pod", "Get a Kubernetes pod.", "namespace name", "public"), + ("kubernetes", "k8s_pod_logs", "Get logs from a Kubernetes pod.", "namespace name container", "public"), + ("kubernetes", "k8s_apply", "Apply a Kubernetes manifest.", "manifest namespace", "public"), + ("kubernetes", "k8s_delete", "Delete a Kubernetes resource.", "kind name namespace", "public"), + ("kubernetes", "k8s_list_deployments", "List Kubernetes deployments.", "namespace", "public"), + ("kubernetes", "k8s_scale_deployment", "Scale a Kubernetes deployment.", "namespace name replicas", "public"), + ("kubernetes", "k8s_list_services", "List Kubernetes services.", "namespace", "public"), + ("kubernetes", "k8s_get_events", "List Kubernetes events.", "namespace", "public"), + # Stripe / Shopify / Salesforce + ("stripe", "stripe_create_customer", "Create a Stripe customer.", "email name", "public"), + ("stripe", "stripe_get_customer", "Retrieve a Stripe customer.", "customer_id", "public"), + ("stripe", "stripe_list_customers", "List Stripe customers.", "email limit", "public"), + ("stripe", "stripe_create_payment_intent", "Create a Stripe PaymentIntent.", "amount currency customer", "public"), + ("stripe", "stripe_get_payment_intent", "Retrieve a Stripe PaymentIntent.", "id", "public"), + ("stripe", "stripe_create_invoice", "Create a Stripe invoice.", "customer", "public"), + ("stripe", "stripe_list_invoices", "List Stripe invoices.", "customer", "public"), + ("stripe", "stripe_create_product", "Create a Stripe product.", "name description", "public"), + ("stripe", "stripe_create_price", "Create a Stripe price.", "product unit_amount currency", "public"), + ("stripe", "stripe_list_subscriptions", "List Stripe subscriptions.", "customer status", "public"), + ("stripe", "stripe_cancel_subscription", "Cancel a Stripe subscription.", "subscription_id", "public"), + ("stripe", "stripe_create_refund", "Create a Stripe refund.", "payment_intent amount", "public"), + ("shopify", "shopify_list_products", "List Shopify products.", "limit query", "public"), + ("shopify", "shopify_get_product", "Get a Shopify product.", "id", "public"), + ("shopify", "shopify_create_product", "Create a Shopify product.", "title body_html vendor", "public"), + ("shopify", "shopify_update_product", "Update a Shopify product.", "id title", "public"), + ("shopify", "shopify_list_orders", "List Shopify orders.", "status limit", "public"), + ("shopify", "shopify_get_order", "Get a Shopify order.", "id", "public"), + ("shopify", "shopify_create_order", "Create a Shopify order.", "line_items email", "public"), + ("shopify", "shopify_list_customers", "List Shopify customers.", "limit", "public"), + ("salesforce", "salesforce_query", "Run a SOQL query against Salesforce.", "soql", "public"), + ("salesforce", "salesforce_create_record", "Create a Salesforce record.", "sobject fields", "public"), + ("salesforce", "salesforce_update_record", "Update a Salesforce record.", "sobject id fields", "public"), + ("salesforce", "salesforce_get_record", "Get a Salesforce record by id.", "sobject id", "public"), + ("salesforce", "salesforce_delete_record", "Delete a Salesforce record.", "sobject id", "public"), + ("salesforce", "salesforce_list_opportunities", "List Salesforce opportunities.", "stage", "public"), + ("salesforce", "salesforce_create_lead", "Create a Salesforce lead.", "firstName lastName company email", "public"), + ("salesforce", "salesforce_convert_lead", "Convert a Salesforce lead.", "leadId", "public"), + # Observability / incident + ("sentry", "sentry_list_issues", "List Sentry issues for a project.", "organization project query", "public"), + ("sentry", "sentry_get_issue", "Get a Sentry issue.", "organization issue_id", "public"), + ("sentry", "sentry_list_events", "List events for a Sentry issue.", "organization issue_id", "public"), + ("sentry", "sentry_update_issue", "Update status of a Sentry issue.", "organization issue_id status", "public"), + ("sentry", "sentry_list_projects", "List Sentry projects.", "organization", "public"), + ("datadog", "datadog_query_metrics", "Query Datadog timeseries metrics.", "query from_ts to_ts", "public"), + ("datadog", "datadog_list_monitors", "List Datadog monitors.", "query", "public"), + ("datadog", "datadog_get_monitor", "Get a Datadog monitor.", "monitor_id", "public"), + ("datadog", "datadog_create_monitor", "Create a Datadog monitor.", "name type query message", "public"), + ("datadog", "datadog_list_logs", "Search Datadog logs.", "query from_ts to_ts", "public"), + ("pagerduty", "pagerduty_list_incidents", "List PagerDuty incidents.", "status urgency", "public"), + ("pagerduty", "pagerduty_get_incident", "Get a PagerDuty incident.", "id", "public"), + ("pagerduty", "pagerduty_create_incident", "Create a PagerDuty incident.", "title service_id urgency", "public"), + ("pagerduty", "pagerduty_acknowledge_incident", "Acknowledge a PagerDuty incident.", "id", "public"), + ("pagerduty", "pagerduty_resolve_incident", "Resolve a PagerDuty incident.", "id", "public"), + ("pagerduty", "pagerduty_list_oncalls", "List PagerDuty on-call users.", "schedule_id", "public"), + # Gmail / Drive / Calendar + ("gmail", "gmail_send_email", "Send an email via Gmail.", "to subject body cc", "public"), + ("gmail", "gmail_search_emails", "Search Gmail messages.", "query maxResults", "public"), + ("gmail", "gmail_get_email", "Get a Gmail message by id.", "id", "public"), + ("gmail", "gmail_create_draft", "Create a Gmail draft.", "to subject body", "public"), + ("gmail", "gmail_list_labels", "List Gmail labels.", "", "public"), + ("gmail", "gmail_modify_email", "Add or remove Gmail labels on a message.", "id addLabelIds removeLabelIds", "public"), + ("gmail", "gmail_trash_email", "Move a Gmail message to trash.", "id", "public"), + ("google_drive", "gdrive_search", "Search files in Google Drive.", "query", "public"), + ("google_drive", "gdrive_get_file", "Get Google Drive file metadata or content.", "file_id", "public"), + ("google_drive", "gdrive_create_file", "Create a file in Google Drive.", "name mimeType content parent", "public"), + ("google_drive", "gdrive_update_file", "Update a Google Drive file.", "file_id content name", "public"), + ("google_drive", "gdrive_list_folder", "List files in a Google Drive folder.", "folder_id", "public"), + ("google_drive", "gdrive_share_file", "Share a Google Drive file.", "file_id email role", "public"), + ("google_calendar", "gcal_list_events", "List Google Calendar events.", "calendar_id timeMin timeMax", "public"), + ("google_calendar", "gcal_create_event", "Create a Google Calendar event.", "calendar_id summary start end attendees", "public"), + ("google_calendar", "gcal_update_event", "Update a Google Calendar event.", "calendar_id event_id summary", "public"), + ("google_calendar", "gcal_delete_event", "Delete a Google Calendar event.", "calendar_id event_id", "public"), + ("google_calendar", "gcal_list_calendars", "List Google calendars.", "", "public"), + # GitLab / Bitbucket + ("gitlab", "gitlab_create_issue", "Create a GitLab issue.", "project_id title description", "public"), + ("gitlab", "gitlab_list_issues", "List GitLab issues.", "project_id state", "public"), + ("gitlab", "gitlab_get_issue", "Get a GitLab issue.", "project_id issue_iid", "public"), + ("gitlab", "gitlab_create_merge_request", "Create a GitLab merge request.", "project_id source_branch target_branch title", "public"), + ("gitlab", "gitlab_list_merge_requests", "List GitLab merge requests.", "project_id state", "public"), + ("gitlab", "gitlab_get_merge_request", "Get a GitLab merge request.", "project_id merge_request_iid", "public"), + ("gitlab", "gitlab_merge_merge_request", "Merge a GitLab merge request.", "project_id merge_request_iid", "public"), + ("gitlab", "gitlab_list_projects", "List GitLab projects.", "search", "public"), + ("gitlab", "gitlab_get_file", "Get a file from a GitLab project.", "project_id file_path ref", "public"), + ("gitlab", "gitlab_create_pipeline", "Create a GitLab CI pipeline.", "project_id ref", "public"), + ("bitbucket", "bitbucket_create_pr", "Create a Bitbucket pull request.", "workspace repo title source dest", "public"), + ("bitbucket", "bitbucket_list_prs", "List Bitbucket pull requests.", "workspace repo", "public"), + ("bitbucket", "bitbucket_create_issue", "Create a Bitbucket issue.", "workspace repo title", "public"), + ("bitbucket", "bitbucket_list_repos", "List Bitbucket repositories.", "workspace", "public"), + # Databases + ("sqlite", "sqlite_query", "Run a SQL query against SQLite.", "sql", "public"), + ("sqlite", "sqlite_list_tables", "List tables in the SQLite database.", "", "public"), + ("sqlite", "sqlite_describe_table", "Describe a SQLite table.", "table", "public"), + ("mysql", "mysql_query", "Run a SQL query against MySQL.", "sql", "public"), + ("mysql", "mysql_list_tables", "List MySQL tables.", "schema", "public"), + ("mongodb", "mongo_find", "Find documents in a MongoDB collection.", "database collection filter", "public"), + ("mongodb", "mongo_insert", "Insert documents into MongoDB.", "database collection documents", "public"), + ("mongodb", "mongo_update", "Update documents in MongoDB.", "database collection filter update", "public"), + ("mongodb", "mongo_delete", "Delete documents from MongoDB.", "database collection filter", "public"), + ("mongodb", "mongo_list_collections", "List MongoDB collections.", "database", "public"), + ("redis", "redis_get", "Get a Redis key.", "key", "public"), + ("redis", "redis_set", "Set a Redis key.", "key value ttl", "public"), + ("redis", "redis_delete", "Delete Redis keys.", "keys", "public"), + ("redis", "redis_keys", "List Redis keys matching a pattern.", "pattern", "public"), + ("elasticsearch", "es_search", "Search an Elasticsearch index.", "index query", "public"), + ("elasticsearch", "es_index_doc", "Index a document in Elasticsearch.", "index document id", "public"), + ("elasticsearch", "es_get_doc", "Get a document from Elasticsearch.", "index id", "public"), + # Cloud + ("aws_s3", "s3_list_buckets", "List AWS S3 buckets.", "", "public"), + ("aws_s3", "s3_list_objects", "List objects in an S3 bucket.", "bucket prefix", "public"), + ("aws_s3", "s3_get_object", "Get an object from S3.", "bucket key", "public"), + ("aws_s3", "s3_put_object", "Upload an object to S3.", "bucket key body", "public"), + ("aws_s3", "s3_delete_object", "Delete an S3 object.", "bucket key", "public"), + ("cloudflare", "cf_list_zones", "List Cloudflare zones.", "", "public"), + ("cloudflare", "cf_list_dns", "List DNS records for a Cloudflare zone.", "zone_id", "public"), + ("cloudflare", "cf_create_dns", "Create a Cloudflare DNS record.", "zone_id type name content", "public"), + ("cloudflare", "cf_purge_cache", "Purge Cloudflare cache.", "zone_id files", "public"), + ("vercel", "vercel_list_projects", "List Vercel projects.", "", "public"), + ("vercel", "vercel_list_deployments", "List Vercel deployments.", "projectId", "public"), + ("vercel", "vercel_get_deployment", "Get a Vercel deployment.", "id", "public"), + ("vercel", "vercel_create_project", "Create a Vercel project.", "name", "public"), + ("supabase", "supabase_list_tables", "List tables in a Supabase project.", "", "public"), + ("supabase", "supabase_select", "Select rows from a Supabase table.", "table filter", "public"), + ("supabase", "supabase_insert", "Insert rows into a Supabase table.", "table rows", "public"), + ("supabase", "supabase_update", "Update rows in a Supabase table.", "table filter values", "public"), + ("supabase", "supabase_invoke_fn", "Invoke a Supabase edge function.", "name body", "public"), + # Misc public MCP-style + ("figma", "figma_get_file", "Get a Figma file.", "file_key", "public"), + ("figma", "figma_get_file_nodes", "Get specific nodes from a Figma file.", "file_key ids", "public"), + ("figma", "figma_get_comments", "Get comments on a Figma file.", "file_key", "public"), + ("figma", "figma_post_comment", "Post a comment on a Figma file.", "file_key message", "public"), + ("figma", "figma_get_images", "Export images from a Figma file.", "file_key ids format", "public"), + ("twilio", "twilio_send_sms", "Send an SMS via Twilio.", "to from_ body", "public"), + ("twilio", "twilio_list_messages", "List Twilio SMS messages.", "to", "public"), + ("twilio", "twilio_make_call", "Place a phone call via Twilio.", "to from_ url", "public"), + ("okta", "okta_list_users", "List Okta users.", "q", "public"), + ("okta", "okta_get_user", "Get an Okta user.", "user_id", "public"), + ("okta", "okta_create_user", "Create an Okta user.", "profile", "public"), + ("okta", "okta_deactivate_user", "Deactivate an Okta user.", "user_id", "public"), + ("prometheus", "prom_query", "Instant query Prometheus.", "query", "public"), + ("prometheus", "prom_query_range", "Range query Prometheus.", "query start end step", "public"), + ("prometheus", "prom_list_metrics", "List Prometheus metric names.", "match", "public"), + ("grafana", "grafana_search_dashboards", "Search Grafana dashboards.", "query", "public"), + ("grafana", "grafana_get_dashboard", "Get a Grafana dashboard.", "uid", "public"), + ("grafana", "grafana_create_annotation", "Create a Grafana annotation.", "text time tags", "public"), + ("circleci", "circleci_list_pipelines", "List CircleCI pipelines.", "project_slug", "public"), + ("circleci", "circleci_get_pipeline", "Get a CircleCI pipeline.", "pipeline_id", "public"), + ("circleci", "circleci_trigger_pipeline", "Trigger a CircleCI pipeline.", "project_slug branch", "public"), + ("terraform", "tf_plan", "Run terraform plan.", "dir", "public"), + ("terraform", "tf_apply", "Run terraform apply.", "dir auto_approve", "public"), + ("terraform", "tf_state_list", "List resources in terraform state.", "dir", "public"), + ("asana", "asana_create_task", "Create an Asana task.", "name notes projects assignee", "public"), + ("asana", "asana_list_tasks", "List Asana tasks.", "project assignee", "public"), + ("asana", "asana_get_task", "Get an Asana task.", "task_gid", "public"), + ("asana", "asana_update_task", "Update an Asana task.", "task_gid name completed", "public"), + ("asana", "asana_create_project", "Create an Asana project.", "name workspace", "public"), + ("trello", "trello_create_card", "Create a Trello card.", "idList name desc", "public"), + ("trello", "trello_list_cards", "List cards on a Trello board or list.", "idBoard idList", "public"), + ("trello", "trello_create_board", "Create a Trello board.", "name", "public"), + ("trello", "trello_add_comment", "Comment on a Trello card.", "idCard text", "public"), + ("clickup", "clickup_create_task", "Create a ClickUp task.", "list_id name description", "public"), + ("clickup", "clickup_list_tasks", "List ClickUp tasks.", "list_id", "public"), + ("clickup", "clickup_get_task", "Get a ClickUp task.", "task_id", "public"), + ("clickup", "clickup_update_task", "Update a ClickUp task.", "task_id name status", "public"), + ("confluence", "confluence_get_page", "Get a Confluence page.", "id", "public"), + ("confluence", "confluence_create_page", "Create a Confluence page.", "spaceId title body", "public"), + ("confluence", "confluence_update_page", "Update a Confluence page.", "id title body", "public"), + ("confluence", "confluence_search", "Search Confluence content.", "query", "public"), + ("hubspot", "hubspot_create_contact", "Create a HubSpot contact.", "email firstname lastname", "public"), + ("hubspot", "hubspot_get_contact", "Get a HubSpot contact.", "id", "public"), + ("hubspot", "hubspot_list_contacts", "List HubSpot contacts.", "limit", "public"), + ("hubspot", "hubspot_create_deal", "Create a HubSpot deal.", "dealname amount pipeline", "public"), + ("hubspot", "hubspot_search_contacts", "Search HubSpot contacts.", "query", "public"), + ("zendesk", "zendesk_create_ticket", "Create a Zendesk support ticket.", "subject comment requester", "public"), + ("zendesk", "zendesk_get_ticket", "Get a Zendesk ticket.", "id", "public"), + ("zendesk", "zendesk_list_tickets", "List Zendesk tickets.", "status", "public"), + ("zendesk", "zendesk_update_ticket", "Update a Zendesk ticket.", "id status comment", "public"), + ("zendesk", "zendesk_search", "Search Zendesk tickets and users.", "query", "public"), + ("discord", "discord_send_message", "Send a message to a Discord channel.", "channel_id content", "public"), + ("discord", "discord_list_channels", "List Discord guild channels.", "guild_id", "public"), + ("discord", "discord_get_channel", "Get a Discord channel.", "channel_id", "public"), + ("discord", "discord_add_reaction", "Add a reaction to a Discord message.", "channel_id message_id emoji", "public"), + ("discord", "discord_list_members", "List Discord guild members.", "guild_id", "public"), + ] + tools = [] + for server, name, desc, params, source in rows: + tools.append(_tool(server, name, desc, [p for p in params.split() if p], source=source)) + return tools + + +def _expand_family(server: str, resources: list[str], actions: list[str], source: str) -> list[dict[str, Any]]: + out = [] + for resource in resources: + singular = resource.rstrip("s") if resource.endswith("s") and not resource.endswith("ss") else resource + for action in actions: + if action == "list": + name = f"{action}_{resource}" + desc = f"{action.capitalize()} {resource.replace('_', ' ')} in {server.replace('_', ' ')}." + elif action == "search": + name = f"{action}_{resource}" + desc = f"Search {resource.replace('_', ' ')} in {server.replace('_', ' ')}." + else: + name = f"{action}_{singular}" + desc = f"{action.capitalize()} a {singular.replace('_', ' ')} in {server.replace('_', ' ')}." + params = { + "create": ["name", "title", "description", "body"], + "get": ["id"], + "list": ["limit", "cursor", "status"], + "update": ["id", "fields"], + "delete": ["id"], + "search": ["query", "limit"], + }.get(action, ["id"]) + out.append(_tool(server, name, desc, params, source=source)) + return out + + +def _pad_to_target(core: list[dict[str, Any]], target: int = 536) -> list[dict[str, Any]]: + seen = {t["id"] for t in core} + extras: list[dict[str, Any]] = [] + families = [ + ("github", ["milestones", "hooks", "environments", "packages", "check_runs", "artifacts", "runners"], ["create", "get", "list", "update", "delete"]), + ("gitlab", ["pipelines", "jobs", "wikis", "snippets", "labels", "milestones", "releases"], ["create", "get", "list", "update", "delete"]), + ("jira", ["boards", "sprints", "filters", "webhooks", "attachments", "versions"], ["create", "get", "list", "update", "delete"]), + ("linear", ["cycles", "labels", "users", "comments", "attachments"], ["create", "get", "list", "update", "delete"]), + ("slack", ["reminders", "canvases", "bookmarks", "usergroups", "files", "pins"], ["create", "get", "list", "update", "delete"]), + ("notion", ["comments", "users_tokens", "data_sources"], ["create", "get", "list", "update", "delete"]), + ("postgres", ["indexes", "views", "roles", "extensions", "publications"], ["list", "get", "create", "delete"]), + ("filesystem", ["symlinks", "permissions"], ["get", "list", "update"]), + ("docker", ["volumes", "networks", "compose_services"], ["create", "get", "list", "delete"]), + ("kubernetes", ["configmaps", "secrets", "ingresses", "namespaces", "nodes", "jobs"], ["create", "get", "list", "update", "delete"]), + ("stripe", ["coupons", "disputes", "payouts", "webhooks", "checkout_sessions"], ["create", "get", "list", "update", "delete"]), + ("shopify", ["collections", "inventory_items", "discounts", "fulfillments"], ["create", "get", "list", "update", "delete"]), + ("salesforce", ["accounts", "contacts", "cases", "campaigns", "tasks"], ["create", "get", "list", "update", "delete"]), + ("sentry", ["releases", "alerts", "projects_keys"], ["create", "get", "list", "update", "delete"]), + ("datadog", ["dashboards", "slos", "synthetics", "incidents"], ["create", "get", "list", "update", "delete"]), + ("gmail", ["filters", "threads", "attachments", "vacation"], ["create", "get", "list", "update", "delete"]), + ("google_drive", ["permissions", "revisions", "comments", "drives"], ["create", "get", "list", "update", "delete"]), + ("google_calendar", ["acl_rules", "freebusy"], ["get", "list", "create", "delete"]), + ("aws_s3", ["bucket_policies", "multipart_uploads"], ["create", "get", "list", "delete"]), + ("cloudflare", ["workers", "waf_rules", "certificates"], ["create", "get", "list", "update", "delete"]), + ("vercel", ["env_vars", "domains", "aliases", "checks"], ["create", "get", "list", "update", "delete"]), + ("supabase", ["buckets", "policies", "functions"], ["create", "get", "list", "update", "delete"]), + ("hubspot", ["companies", "tickets", "engagements"], ["create", "get", "list", "update", "delete"]), + ("zendesk", ["users", "organizations", "macros"], ["create", "get", "list", "update", "delete"]), + ("discord", ["roles", "webhooks", "threads", "emojis"], ["create", "get", "list", "update", "delete"]), + ("asana", ["sections", "portfolios", "tags", "stories"], ["create", "get", "list", "update", "delete"]), + ("trello", ["lists", "checklists", "labels", "members"], ["create", "get", "list", "update", "delete"]), + ("clickup", ["lists", "folders", "spaces", "comments"], ["create", "get", "list", "update", "delete"]), + ("confluence", ["spaces", "attachments", "labels"], ["create", "get", "list", "update", "delete"]), + ("okta", ["groups", "apps", "factors"], ["create", "get", "list", "update", "delete"]), + ("twilio", ["phone_numbers", "verify_services"], ["create", "get", "list", "delete"]), + ("figma", ["projects", "components", "styles"], ["get", "list"]), + ("prometheus", ["alerts", "targets", "rules"], ["get", "list"]), + ("grafana", ["folders", "alert_rules", "datasources"], ["create", "get", "list", "update", "delete"]), + ("circleci", ["workflows", "jobs", "contexts"], ["get", "list", "create"]), + ("terraform", ["workspaces", "variables"], ["create", "get", "list", "update", "delete"]), + ("pagerduty", ["services", "schedules", "escalation_policies"], ["create", "get", "list", "update", "delete"]), + ("elasticsearch", ["indices", "aliases", "templates"], ["create", "get", "list", "delete"]), + ("redis", ["hashes", "lists", "streams"], ["get", "list", "create", "delete"]), + ("mongodb", ["indexes", "aggregations"], ["create", "get", "list", "delete"]), + ("mysql", ["users", "indexes", "views"], ["create", "get", "list", "delete"]), + ("sqlite", ["indexes", "views"], ["create", "get", "list", "delete"]), + ("git", ["stashes", "remotes", "tags"], ["create", "get", "list", "delete"]), + ("playwright", ["console_messages", "network_requests"], ["get", "list"]), + ("memory", ["observations_batch"], ["create", "get", "list", "delete"]), + ("fetch", ["headers"], ["get"]), + ("brave_search", ["news", "images"], ["search"]), + ("time", ["timezones"], ["list", "get"]), + ("sequential_thinking", ["revisions"], ["list"]), + ("bitbucket", ["pipelines", "downloads", "snippets"], ["create", "get", "list", "delete"]), + ] + for server, resources, actions in families: + for t in _expand_family(server, resources, actions, source="expanded"): + if t["id"] not in seen: + extras.append(t) + seen.add(t["id"]) + if len(core) + len(extras) >= target: + return core + extras[: target - len(core)] + if len(core) + len(extras) < target: + raise RuntimeError(f"only produced {len(core)+len(extras)} tools, need {target}") + return core + extras[: target - len(core)] + + +def build_catalog(n: int = 536) -> list[dict[str, Any]]: + core = _real_core() + # Dedup core by id, keep first. + uniq = [] + seen = set() + for t in core: + if t["id"] in seen: + continue + seen.add(t["id"]) + uniq.append(t) + return _pad_to_target(uniq, n) + + +DISTRACTOR_VENDORS = [ + "acme", "northstar", "globex", "initech", "umbrella", "wayne", "stark", + "hooli", "piedpiper", "wonka", "oscorp", "cyberdyne", "tyrell", "weyland", + "massive", "apex", "lumen", "nimbus", "cobalt", "aurora", "harbor", + "ridge", "cinder", "quartz", "helium", "ion", "kestrel", "lynx", "otter", + "puffin", "quail", "raven", "sable", "teal", "umbra", "velvet", "willow", + "xenon", "yarrow", "zephyr", +] + +DISTRACTOR_RESOURCES = [ + "issues", "tickets", "tasks", "messages", "comments", "users", "projects", + "orders", "invoices", "customers", "files", "documents", "events", + "alerts", "incidents", "deployments", "pipelines", "records", "items", + "contacts", "leads", "candidates", "employees", "channels", "boards", +] + + +def build_distractors(count: int, seed: int = 20261020) -> list[dict[str, Any]]: + rng = random.Random(seed) + out = [] + i = 0 + while len(out) < count: + vendor = DISTRACTOR_VENDORS[i % len(DISTRACTOR_VENDORS)] + resource = DISTRACTOR_RESOURCES[(i // len(DISTRACTOR_VENDORS)) % len(DISTRACTOR_RESOURCES)] + action = VERBS[i % len(VERBS)] + suffix = i // (len(DISTRACTOR_VENDORS) * len(DISTRACTOR_RESOURCES) * len(VERBS)) + server = f"{vendor}_{suffix}" if suffix else vendor + singular = resource[:-1] if resource.endswith("s") else resource + name = f"{action}_{resource}" if action in ("list", "search") else f"{action}_{singular}" + if suffix: + name = f"{name}_{suffix}" + desc = ( + f"{action.capitalize()} {resource.replace('_', ' ')} in the {vendor} " + f"{singular} workspace. Supports the usual {action} workflow." + ) + out.append(_tool(server, name, desc, ["id", "name", "query", "limit"], source="distractor")) + i += 1 + return out + + +def _hand_queries() -> list[dict[str, Any]]: + """Natural-language queries labelled against the reproduction catalog.""" + return [ + {"id": "q001", "query": "Create a GitHub issue", "gold": ["github.create_issue"], "alts": ["github.issue_write"], "kind": "verb_product"}, + {"id": "q002", "query": "Create a Jira ticket", "gold": ["jira.jira_create_issue"], "alts": [], "kind": "verb_product"}, + {"id": "q003", "query": "Send a Slack message", "gold": ["slack.slack_post_message"], "alts": [], "kind": "verb_product"}, + {"id": "q004", "query": "List open pull requests", "gold": ["github.list_pull_requests"], "alts": ["github.search_pull_requests"], "kind": "resource"}, + {"id": "q005", "query": "Open a pull request on GitHub", "gold": ["github.create_pull_request"], "alts": [], "kind": "verb_product"}, + {"id": "q006", "query": "Run a SQL query against postgres", "gold": ["postgres.query"], "alts": [], "kind": "verb_product"}, + {"id": "q007", "query": "Read a local file from disk", "gold": ["filesystem.read_file"], "alts": ["filesystem.read_text_file"], "kind": "paraphrase"}, + {"id": "q008", "query": "What files are in this folder", "gold": ["filesystem.list_directory"], "alts": ["filesystem.list_directory_with_sizes", "filesystem.directory_tree"], "kind": "paraphrase"}, + {"id": "q009", "query": "Remember this fact in the knowledge graph", "gold": ["memory.add_observations"], "alts": ["memory.create_entities"], "kind": "paraphrase"}, + {"id": "q010", "query": "Search the web with Brave", "gold": ["brave_search.brave_web_search"], "alts": [], "kind": "verb_product"}, + {"id": "q011", "query": "Fetch the contents of a URL", "gold": ["fetch.fetch"], "alts": [], "kind": "verb_product"}, + {"id": "q012", "query": "Show git status", "gold": ["git.git_status"], "alts": [], "kind": "identifier"}, + {"id": "q013", "query": "Commit staged changes", "gold": ["git.git_commit"], "alts": [], "kind": "paraphrase"}, + {"id": "q014", "query": "Create a Linear issue", "gold": ["linear.linear_create_issue"], "alts": [], "kind": "verb_product"}, + {"id": "q015", "query": "Make a task in my project tracker Linear", "gold": ["linear.linear_create_issue"], "alts": [], "kind": "paraphrase"}, + {"id": "q016", "query": "Post a comment on a GitHub issue", "gold": ["github.add_issue_comment"], "alts": [], "kind": "verb_product"}, + {"id": "q017", "query": "Merge the pull request", "gold": ["github.merge_pull_request"], "alts": ["gitlab.gitlab_merge_merge_request"], "kind": "ambiguous"}, + {"id": "q018", "query": "Trigger a GitHub Actions workflow", "gold": ["github.actions_run_trigger"], "alts": [], "kind": "verb_product"}, + {"id": "q019", "query": "Get Dependabot alerts", "gold": ["github.list_dependabot_alerts"], "alts": ["github.get_dependabot_alert"], "kind": "resource"}, + {"id": "q020", "query": "List Sentry issues", "gold": ["sentry.sentry_list_issues"], "alts": [], "kind": "verb_product"}, + {"id": "q021", "query": "Create a Stripe customer", "gold": ["stripe.stripe_create_customer"], "alts": [], "kind": "verb_product"}, + {"id": "q022", "query": "Refund a Stripe payment", "gold": ["stripe.stripe_create_refund"], "alts": [], "kind": "paraphrase"}, + {"id": "q023", "query": "Send an email with Gmail", "gold": ["gmail.gmail_send_email"], "alts": [], "kind": "verb_product"}, + {"id": "q024", "query": "Search my Gmail inbox", "gold": ["gmail.gmail_search_emails"], "alts": [], "kind": "verb_product"}, + {"id": "q025", "query": "Upload a file to Google Drive", "gold": ["google_drive.gdrive_create_file"], "alts": [], "kind": "paraphrase"}, + {"id": "q026", "query": "Create a calendar event", "gold": ["google_calendar.gcal_create_event"], "alts": [], "kind": "resource"}, + {"id": "q027", "query": "List Kubernetes pods", "gold": ["kubernetes.k8s_list_pods"], "alts": [], "kind": "verb_product"}, + {"id": "q028", "query": "Get logs from a k8s pod", "gold": ["kubernetes.k8s_pod_logs"], "alts": [], "kind": "verb_product"}, + {"id": "q029", "query": "Run a docker container", "gold": ["docker.docker_run"], "alts": [], "kind": "verb_product"}, + {"id": "q030", "query": "Show docker container logs", "gold": ["docker.docker_logs"], "alts": [], "kind": "verb_product"}, + {"id": "q031", "query": "Navigate the browser to a URL", "gold": ["playwright.browser_navigate"], "alts": ["puppeteer.puppeteer_navigate"], "kind": "ambiguous"}, + {"id": "q032", "query": "Take a screenshot of the page", "gold": ["playwright.browser_take_screenshot"], "alts": ["puppeteer.puppeteer_screenshot"], "kind": "ambiguous"}, + {"id": "q033", "query": "Create a GitLab merge request", "gold": ["gitlab.gitlab_create_merge_request"], "alts": [], "kind": "verb_product"}, + {"id": "q034", "query": "Search Notion pages", "gold": ["notion.notion_search"], "alts": [], "kind": "verb_product"}, + {"id": "q035", "query": "Create a Notion page", "gold": ["notion.notion_create_page"], "alts": [], "kind": "verb_product"}, + {"id": "q036", "query": "Query a Notion database", "gold": ["notion.notion_query_database"], "alts": [], "kind": "verb_product"}, + {"id": "q037", "query": "Create a Zendesk ticket", "gold": ["zendesk.zendesk_create_ticket"], "alts": [], "kind": "verb_product"}, + {"id": "q038", "query": "Create a HubSpot contact", "gold": ["hubspot.hubspot_create_contact"], "alts": [], "kind": "verb_product"}, + {"id": "q039", "query": "Create a Salesforce lead", "gold": ["salesforce.salesforce_create_lead"], "alts": [], "kind": "verb_product"}, + {"id": "q040", "query": "Run a SOQL query", "gold": ["salesforce.salesforce_query"], "alts": [], "kind": "identifier"}, + {"id": "q041", "query": "List Shopify orders", "gold": ["shopify.shopify_list_orders"], "alts": [], "kind": "verb_product"}, + {"id": "q042", "query": "Create a PagerDuty incident", "gold": ["pagerduty.pagerduty_create_incident"], "alts": [], "kind": "verb_product"}, + {"id": "q043", "query": "Who is on call in PagerDuty", "gold": ["pagerduty.pagerduty_list_oncalls"], "alts": [], "kind": "paraphrase"}, + {"id": "q044", "query": "Query Datadog metrics", "gold": ["datadog.datadog_query_metrics"], "alts": [], "kind": "verb_product"}, + {"id": "q045", "query": "Search Datadog logs", "gold": ["datadog.datadog_list_logs"], "alts": [], "kind": "verb_product"}, + {"id": "q046", "query": "List S3 buckets", "gold": ["aws_s3.s3_list_buckets"], "alts": [], "kind": "verb_product"}, + {"id": "q047", "query": "Upload an object to S3", "gold": ["aws_s3.s3_put_object"], "alts": [], "kind": "paraphrase"}, + {"id": "q048", "query": "Purge Cloudflare cache", "gold": ["cloudflare.cf_purge_cache"], "alts": [], "kind": "verb_product"}, + {"id": "q049", "query": "Create a Cloudflare DNS record", "gold": ["cloudflare.cf_create_dns"], "alts": [], "kind": "verb_product"}, + {"id": "q050", "query": "List Vercel deployments", "gold": ["vercel.vercel_list_deployments"], "alts": [], "kind": "verb_product"}, + {"id": "q051", "query": "Insert a row into Supabase", "gold": ["supabase.supabase_insert"], "alts": [], "kind": "verb_product"}, + {"id": "q052", "query": "Export images from Figma", "gold": ["figma.figma_get_images"], "alts": [], "kind": "paraphrase"}, + {"id": "q053", "query": "Send an SMS with Twilio", "gold": ["twilio.twilio_send_sms"], "alts": [], "kind": "verb_product"}, + {"id": "q054", "query": "Create an Okta user", "gold": ["okta.okta_create_user"], "alts": [], "kind": "verb_product"}, + {"id": "q055", "query": "Instant query Prometheus", "gold": ["prometheus.prom_query"], "alts": [], "kind": "verb_product"}, + {"id": "q056", "query": "Search Grafana dashboards", "gold": ["grafana.grafana_search_dashboards"], "alts": [], "kind": "verb_product"}, + {"id": "q057", "query": "Trigger a CircleCI pipeline", "gold": ["circleci.circleci_trigger_pipeline"], "alts": [], "kind": "verb_product"}, + {"id": "q058", "query": "Run terraform plan", "gold": ["terraform.tf_plan"], "alts": [], "kind": "identifier"}, + {"id": "q059", "query": "Create an Asana task", "gold": ["asana.asana_create_task"], "alts": [], "kind": "verb_product"}, + {"id": "q060", "query": "Create a Trello card", "gold": ["trello.trello_create_card"], "alts": [], "kind": "verb_product"}, + {"id": "q061", "query": "Create a ClickUp task", "gold": ["clickup.clickup_create_task"], "alts": [], "kind": "verb_product"}, + {"id": "q062", "query": "Create a Confluence page", "gold": ["confluence.confluence_create_page"], "alts": [], "kind": "verb_product"}, + {"id": "q063", "query": "Send a Discord message", "gold": ["discord.discord_send_message"], "alts": [], "kind": "verb_product"}, + {"id": "q064", "query": "Search Elasticsearch", "gold": ["elasticsearch.es_search"], "alts": [], "kind": "verb_product"}, + {"id": "q065", "query": "Get a Redis key", "gold": ["redis.redis_get"], "alts": [], "kind": "verb_product"}, + {"id": "q066", "query": "Find MongoDB documents", "gold": ["mongodb.mongo_find"], "alts": [], "kind": "verb_product"}, + {"id": "q067", "query": "List SQLite tables", "gold": ["sqlite.sqlite_list_tables"], "alts": [], "kind": "verb_product"}, + {"id": "q068", "query": "Describe a postgres table", "gold": ["postgres.describe_table"], "alts": [], "kind": "verb_product"}, + {"id": "q069", "query": "github_create_issue", "gold": ["github.create_issue"], "alts": ["github.issue_write"], "kind": "identifier"}, + {"id": "q070", "query": "createIssue", "gold": ["github.create_issue"], "alts": ["jira.jira_create_issue", "linear.linear_create_issue"], "kind": "identifier"}, + {"id": "q071", "query": "list-directory", "gold": ["filesystem.list_directory"], "alts": [], "kind": "identifier"}, + {"id": "q072", "query": "sequential thinking next thought", "gold": ["sequential_thinking.sequentialthinking"], "alts": [], "kind": "paraphrase"}, + {"id": "q073", "query": "What time is it in Tokyo", "gold": ["time.get_current_time"], "alts": ["time.convert_time"], "kind": "paraphrase"}, + {"id": "q074", "query": "Convert time between timezones", "gold": ["time.convert_time"], "alts": [], "kind": "verb_product"}, + {"id": "q075", "query": "Star a GitHub repo", "gold": ["github.star_repository"], "alts": [], "kind": "verb_product"}, + {"id": "q076", "query": "Fork a repository", "gold": ["github.fork_repository"], "alts": [], "kind": "resource"}, + {"id": "q077", "query": "Get the latest GitHub release", "gold": ["github.get_latest_release"], "alts": [], "kind": "verb_product"}, + {"id": "q078", "query": "Create a gist", "gold": ["github.create_gist"], "alts": [], "kind": "resource"}, + {"id": "q079", "query": "Search GitHub code", "gold": ["github.search_code"], "alts": [], "kind": "verb_product"}, + {"id": "q080", "query": "List GitHub notifications", "gold": ["github.list_notifications"], "alts": [], "kind": "verb_product"}, + {"id": "q081", "query": "Assign Copilot to an issue", "gold": ["github.assign_copilot_to_issue"], "alts": [], "kind": "verb_product"}, + {"id": "q082", "query": "Get secret scanning alerts", "gold": ["github.list_secret_scanning_alerts"], "alts": ["github.get_secret_scanning_alert"], "kind": "resource"}, + {"id": "q083", "query": "Create or update a file on GitHub", "gold": ["github.create_or_update_file"], "alts": ["github.push_files"], "kind": "verb_product"}, + {"id": "q084", "query": "List collaborators on a repo", "gold": ["github.list_repository_collaborators"], "alts": [], "kind": "resource"}, + {"id": "q085", "query": "Reply in a Slack thread", "gold": ["slack.slack_reply_to_thread"], "alts": [], "kind": "verb_product"}, + {"id": "q086", "query": "Add an emoji reaction on Slack", "gold": ["slack.slack_add_reaction"], "alts": ["discord.discord_add_reaction"], "kind": "verb_product"}, + {"id": "q087", "query": "Search Slack messages", "gold": ["slack.slack_search_messages"], "alts": [], "kind": "verb_product"}, + {"id": "q088", "query": "Create a Slack channel", "gold": ["slack.slack_create_channel"], "alts": [], "kind": "verb_product"}, + {"id": "q089", "query": "Transition a Jira issue", "gold": ["jira.jira_transition_issue"], "alts": [], "kind": "verb_product"}, + {"id": "q090", "query": "Search Jira with JQL", "gold": ["jira.jira_search_issues"], "alts": [], "kind": "identifier"}, + {"id": "q091", "query": "Add a worklog to Jira", "gold": ["jira.jira_add_worklog"], "alts": [], "kind": "verb_product"}, + {"id": "q092", "query": "List Linear teams", "gold": ["linear.linear_list_teams"], "alts": [], "kind": "verb_product"}, + {"id": "q093", "query": "Comment on a Linear issue", "gold": ["linear.linear_create_comment"], "alts": [], "kind": "verb_product"}, + {"id": "q094", "query": "Move a local file", "gold": ["filesystem.move_file"], "alts": [], "kind": "paraphrase"}, + {"id": "q095", "query": "Recursively search files by pattern", "gold": ["filesystem.search_files"], "alts": [], "kind": "paraphrase"}, + {"id": "q096", "query": "Edit a file in place with old and new string", "gold": ["filesystem.edit_file"], "alts": [], "kind": "paraphrase"}, + {"id": "q097", "query": "Read the whole knowledge graph", "gold": ["memory.read_graph"], "alts": [], "kind": "paraphrase"}, + {"id": "q098", "query": "Search memory nodes", "gold": ["memory.search_nodes"], "alts": [], "kind": "verb_product"}, + {"id": "q099", "query": "Show unstaged git diff", "gold": ["git.git_diff_unstaged"], "alts": [], "kind": "verb_product"}, + {"id": "q100", "query": "Stage files with git add", "gold": ["git.git_add"], "alts": [], "kind": "identifier"}, + {"id": "q101", "query": "Create a Shopify product", "gold": ["shopify.shopify_create_product"], "alts": [], "kind": "verb_product"}, + {"id": "q102", "query": "Cancel a Stripe subscription", "gold": ["stripe.stripe_cancel_subscription"], "alts": [], "kind": "verb_product"}, + {"id": "q103", "query": "Create a payment intent", "gold": ["stripe.stripe_create_payment_intent"], "alts": [], "kind": "resource"}, + {"id": "q104", "query": "Acknowledge a PagerDuty incident", "gold": ["pagerduty.pagerduty_acknowledge_incident"], "alts": [], "kind": "verb_product"}, + {"id": "q105", "query": "Create a Datadog monitor", "gold": ["datadog.datadog_create_monitor"], "alts": [], "kind": "verb_product"}, + {"id": "q106", "query": "Resolve a Sentry issue", "gold": ["sentry.sentry_update_issue"], "alts": [], "kind": "paraphrase"}, + {"id": "q107", "query": "Scale a Kubernetes deployment", "gold": ["kubernetes.k8s_scale_deployment"], "alts": [], "kind": "verb_product"}, + {"id": "q108", "query": "Apply a Kubernetes manifest", "gold": ["kubernetes.k8s_apply"], "alts": [], "kind": "verb_product"}, + {"id": "q109", "query": "Stop a docker container", "gold": ["docker.docker_stop"], "alts": [], "kind": "verb_product"}, + {"id": "q110", "query": "Build a docker image", "gold": ["docker.docker_build"], "alts": [], "kind": "verb_product"}, + {"id": "q111", "query": "Click an element in Playwright", "gold": ["playwright.browser_click"], "alts": [], "kind": "verb_product"}, + {"id": "q112", "query": "Type text into a browser input", "gold": ["playwright.browser_type"], "alts": ["puppeteer.puppeteer_fill"], "kind": "paraphrase"}, + {"id": "q113", "query": "List GitLab projects", "gold": ["gitlab.gitlab_list_projects"], "alts": [], "kind": "verb_product"}, + {"id": "q114", "query": "Create a Bitbucket pull request", "gold": ["bitbucket.bitbucket_create_pr"], "alts": [], "kind": "verb_product"}, + {"id": "q115", "query": "Create a Gmail draft", "gold": ["gmail.gmail_create_draft"], "alts": [], "kind": "verb_product"}, + {"id": "q116", "query": "Share a Google Drive file", "gold": ["google_drive.gdrive_share_file"], "alts": [], "kind": "verb_product"}, + {"id": "q117", "query": "List Google Calendar events", "gold": ["google_calendar.gcal_list_events"], "alts": [], "kind": "verb_product"}, + {"id": "q118", "query": "Delete a calendar event", "gold": ["google_calendar.gcal_delete_event"], "alts": [], "kind": "resource"}, + {"id": "q119", "query": "List Vercel projects", "gold": ["vercel.vercel_list_projects"], "alts": [], "kind": "verb_product"}, + {"id": "q120", "query": "Select rows from Supabase", "gold": ["supabase.supabase_select"], "alts": [], "kind": "verb_product"}, + {"id": "q121", "query": "Post a Figma comment", "gold": ["figma.figma_post_comment"], "alts": [], "kind": "verb_product"}, + {"id": "q122", "query": "List Okta users", "gold": ["okta.okta_list_users"], "alts": [], "kind": "verb_product"}, + {"id": "q123", "query": "Range query Prometheus", "gold": ["prometheus.prom_query_range"], "alts": [], "kind": "verb_product"}, + {"id": "q124", "query": "Create a Grafana annotation", "gold": ["grafana.grafana_create_annotation"], "alts": [], "kind": "verb_product"}, + {"id": "q125", "query": "Apply terraform", "gold": ["terraform.tf_apply"], "alts": [], "kind": "identifier"}, + {"id": "q126", "query": "List Asana tasks", "gold": ["asana.asana_list_tasks"], "alts": [], "kind": "verb_product"}, + {"id": "q127", "query": "Comment on a Trello card", "gold": ["trello.trello_add_comment"], "alts": [], "kind": "verb_product"}, + {"id": "q128", "query": "Update a ClickUp task", "gold": ["clickup.clickup_update_task"], "alts": [], "kind": "verb_product"}, + {"id": "q129", "query": "Search Confluence", "gold": ["confluence.confluence_search"], "alts": [], "kind": "verb_product"}, + {"id": "q130", "query": "Create a HubSpot deal", "gold": ["hubspot.hubspot_create_deal"], "alts": [], "kind": "verb_product"}, + {"id": "q131", "query": "List Zendesk tickets", "gold": ["zendesk.zendesk_list_tickets"], "alts": [], "kind": "verb_product"}, + {"id": "q132", "query": "List Discord channels", "gold": ["discord.discord_list_channels"], "alts": [], "kind": "verb_product"}, + {"id": "q133", "query": "Set a Redis key", "gold": ["redis.redis_set"], "alts": [], "kind": "verb_product"}, + {"id": "q134", "query": "Insert MongoDB documents", "gold": ["mongodb.mongo_insert"], "alts": [], "kind": "verb_product"}, + {"id": "q135", "query": "Run a MySQL query", "gold": ["mysql.mysql_query"], "alts": [], "kind": "verb_product"}, + {"id": "q136", "query": "Index a document in Elasticsearch", "gold": ["elasticsearch.es_index_doc"], "alts": [], "kind": "verb_product"}, + {"id": "q137", "query": "Get an S3 object", "gold": ["aws_s3.s3_get_object"], "alts": [], "kind": "verb_product"}, + {"id": "q138", "query": "List Cloudflare zones", "gold": ["cloudflare.cf_list_zones"], "alts": [], "kind": "verb_product"}, + {"id": "q139", "query": "Create a Stripe invoice", "gold": ["stripe.stripe_create_invoice"], "alts": [], "kind": "verb_product"}, + {"id": "q140", "query": "List GitHub Actions workflows", "gold": ["github.actions_list"], "alts": [], "kind": "verb_product"}, + {"id": "q141", "query": "Get GitHub Actions job logs", "gold": ["github.actions_get_job_logs"], "alts": [], "kind": "verb_product"}, + {"id": "q142", "query": "Create a GitHub repository", "gold": ["github.create_repository"], "alts": [], "kind": "verb_product"}, + {"id": "q143", "query": "Checkout a git branch", "gold": ["git.git_checkout"], "alts": [], "kind": "verb_product"}, + {"id": "q144", "query": "Show git log", "gold": ["git.git_log"], "alts": [], "kind": "identifier"}, + {"id": "q145", "query": "List postgres schemas", "gold": ["postgres.list_schemas"], "alts": [], "kind": "verb_product"}, + {"id": "q146", "query": "Open nodes in the knowledge graph", "gold": ["memory.open_nodes"], "alts": [], "kind": "verb_product"}, + {"id": "q147", "query": "Get file metadata from the filesystem", "gold": ["filesystem.get_file_info"], "alts": [], "kind": "paraphrase"}, + {"id": "q148", "query": "Write a new local file", "gold": ["filesystem.write_file"], "alts": [], "kind": "paraphrase"}, + {"id": "q149", "query": "Search GitHub issues", "gold": ["github.search_issues"], "alts": [], "kind": "verb_product"}, + {"id": "q150", "query": "Get the authenticated GitHub user", "gold": ["github.get_me"], "alts": [], "kind": "paraphrase"}, + ] + + +def build_queries(tools: list[dict[str, Any]]) -> list[dict[str, Any]]: + ids = {t["id"] for t in tools} + queries = [] + missing = [] + for q in _hand_queries(): + gold = [g for g in q["gold"] if g in ids] + alts = [a for a in q.get("alts", []) if a in ids] + if not gold: + missing.append(q["id"] + ":" + ",".join(q["gold"])) + continue + queries.append({**q, "gold": gold, "alts": alts}) + if missing: + raise RuntimeError("gold tools missing from catalog: " + "; ".join(missing)) + if len(queries) != 150: + raise RuntimeError(f"expected 150 queries, got {len(queries)}") + return queries + + +def tool_document(t: dict[str, Any]) -> str: + params = " ".join(t.get("params") or []) + return ( + f"server {t['server']}\n" + f"name {t['name']}\n" + f"tool {t['id']}\n" + f"{t['description']}\n" + f"params {params}\n" + ) + + +def materialize_codedb_corpus(tools: list[dict[str, Any]], dest: Path) -> Path: + """One Python file per tool so codedb can index name/desc/params/server.""" + dest.mkdir(parents=True, exist_ok=True) + for existing in dest.glob("*.py"): + existing.unlink() + id_map: dict[str, str] = {} + for t in tools: + safe = f"{t['server']}__{t['name']}" + safe = re.sub(r"[^a-zA-Z0-9_]+", "_", safe) + path = dest / f"{safe}.py" + id_map[f"{safe}.py"] = t["id"] + params = t.get("params") or [] + sig = ", ".join(f"{p}: str = ''" for p in params) or "" + path.write_text( + f'"""{t["description"]}\n\n' + f'Server: {t["server"]}\n' + f'Tool: {t["name"]}\n' + f'Id: {t["id"]}\n' + f'Params: {" ".join(params)}\n' + f'"""\n\n' + f"def {t['name']}({sig}):\n" + f' """{t["description"]} ({t["server"]})"""\n' + f" return {t['id']!r}\n", + encoding="utf-8", + ) + (dest / "id_map.json").write_text(json.dumps(id_map, indent=2) + "\n", encoding="utf-8") + return dest + + +def write_frozen(tools: list[dict[str, Any]], queries: list[dict[str, Any]]) -> None: + DATA.mkdir(parents=True, exist_ok=True) + (DATA / "catalog.json").write_text(json.dumps(tools, indent=2) + "\n", encoding="utf-8") + (DATA / "queries.json").write_text(json.dumps(queries, indent=2) + "\n", encoding="utf-8") + stats = { + "n_tools": len(tools), + "n_queries": len(queries), + "n_servers": len({t["server"] for t in tools}), + "by_source": {}, + "reproduction": True, + "zed_fixtures_public": False, + } + for t in tools: + stats["by_source"][t["source"]] = stats["by_source"].get(t["source"], 0) + 1 + (DATA / "catalog_stats.json").write_text(json.dumps(stats, indent=2) + "\n", encoding="utf-8") + + +def load_frozen() -> tuple[list[dict[str, Any]], list[dict[str, Any]]]: + tools = json.loads((DATA / "catalog.json").read_text(encoding="utf-8")) + queries = json.loads((DATA / "queries.json").read_text(encoding="utf-8")) + return tools, queries + + +if __name__ == "__main__": + tools = build_catalog(536) + queries = build_queries(tools) + write_frozen(tools, queries) + print(f"wrote {len(tools)} tools, {len(queries)} queries") diff --git a/experiments/mcp-tool-search/lib/codedb_client.py b/experiments/mcp-tool-search/lib/codedb_client.py new file mode 100644 index 00000000..7db36403 --- /dev/null +++ b/experiments/mcp-tool-search/lib/codedb_client.py @@ -0,0 +1,268 @@ +"""Talk to a long-lived codedb MCP process for tool-catalog search.""" + +from __future__ import annotations + +import json +import os +import re +import subprocess +import threading +import time +from pathlib import Path +from typing import Any + + +class MCPProcess: + def __init__(self, binary: str, root: str) -> None: + env = os.environ.copy() + env["CODEDB_QUIET"] = "1" + self.proc = subprocess.Popen( + [binary, root, "mcp"], + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + cwd=root, + env=env, + text=True, + bufsize=1, + ) + self._id = 1 + self._lock = threading.Lock() + self._lines: list[str] = [] + self._stderr: list[str] = [] + threading.Thread(target=self._read, daemon=True).start() + threading.Thread(target=self._read_err, daemon=True).start() + + def _read(self) -> None: + assert self.proc.stdout + for line in self.proc.stdout: + line = line.strip() + if line: + with self._lock: + self._lines.append(line) + + def _read_err(self) -> None: + assert self.proc.stderr + for line in self.proc.stderr: + with self._lock: + self._stderr.append(line.rstrip()) + + def send(self, msg: dict[str, Any]) -> None: + assert self.proc.stdin + self.proc.stdin.write(json.dumps(msg) + "\n") + self.proc.stdin.flush() + + def recv(self, timeout: float = 10.0) -> dict[str, Any] | None: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + with self._lock: + if self._lines: + return json.loads(self._lines.pop(0)) + time.sleep(0.01) + return None + + def next_id(self) -> int: + self._id += 1 + return self._id + + def call_tool(self, name: str, args: dict[str, Any], timeout: float = 30.0) -> dict[str, Any] | None: + req_id = self.next_id() + self.send( + { + "jsonrpc": "2.0", + "id": req_id, + "method": "tools/call", + "params": {"name": name, "arguments": args}, + } + ) + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + msg = self.recv(timeout=1.0) + if msg is None: + continue + if msg.get("id") == req_id: + return msg + return None + + def close(self) -> None: + try: + if self.proc.stdin: + self.proc.stdin.close() + except Exception: + pass + try: + self.proc.wait(timeout=5) + except subprocess.TimeoutExpired: + self.proc.kill() + + def stderr(self) -> str: + with self._lock: + return "\n".join(self._stderr[-40:]) + + +def tool_text(resp: dict[str, Any] | None) -> str: + if not resp: + return "" + content = resp.get("result", {}).get("content", []) + return "\n".join(c.get("text", "") for c in content if isinstance(c, dict)) + + +def initialize(p: MCPProcess) -> bool: + p.send( + { + "jsonrpc": "2.0", + "id": 1, + "method": "initialize", + "params": { + "protocolVersion": "2024-11-05", + "capabilities": {}, + "clientInfo": {"name": "mcp-tool-search-harness", "version": "1"}, + }, + } + ) + resp = p.recv(timeout=15) + if resp is None or "result" not in resp: + return False + p.send({"jsonrpc": "2.0", "method": "notifications/initialized", "params": {}}) + return True + + +def wait_ready(p: MCPProcess, timeout: float = 90.0) -> bool: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + resp = p.call_tool("codedb_status", {}, timeout=8.0) + text = tool_text(resp) + m = re.search(r"outlines:\s*(\d+)", text) + if m and int(m.group(1)) > 0 and "scan: ready" in text: + return True + time.sleep(0.4) + return False + + +def parse_search_json(text: str) -> list[str]: + try: + data = json.loads(text) + except json.JSONDecodeError: + paths = [] + for line in text.splitlines(): + m = re.search(r"([A-Za-z0-9_./-]+\.py):\d+", line) + if m: + paths.append(m.group(1)) + return paths + out = [] + for row in data.get("results") or []: + path = row.get("path") + if path: + out.append(path) + return out + + +_ID_MAP: dict[str, str] = {} + + +def load_id_map(corpus: Path) -> None: + global _ID_MAP + p = corpus / "id_map.json" + if p.exists(): + _ID_MAP = json.loads(p.read_text(encoding="utf-8")) + + +def path_to_id(path: str) -> str: + base = Path(path).name + if base in _ID_MAP: + return _ID_MAP[base] + if base.endswith(".py"): + stem = base[:-3] + if "__" in stem: + server, name = stem.split("__", 1) + return f"{server}.{name}" + return stem + return base + + +class CodedbSession: + def __init__(self, binary: str, corpus: Path) -> None: + self.binary = binary + self.corpus = corpus + load_id_map(corpus) + self.proc: MCPProcess | None = None + self.index_s = 0.0 + self.ready = False + self.error: str | None = None + + def start(self) -> None: + t0 = time.perf_counter() + self.proc = MCPProcess(self.binary, str(self.corpus)) + if not initialize(self.proc): + self.error = "initialize failed: " + self.proc.stderr() + return + if not wait_ready(self.proc): + self.error = "scan not ready: " + self.proc.stderr() + return + self.index_s = time.perf_counter() - t0 + self.ready = True + + def close(self) -> None: + if self.proc: + self.proc.close() + + def search(self, query: str, max_results: int = 20) -> list[str]: + assert self.proc + resp = self.proc.call_tool( + "codedb_search", + {"query": query, "max_results": max_results, "format": "json"}, + timeout=20.0, + ) + return [path_to_id(p) for p in parse_search_json(tool_text(resp))] + + def word(self, word: str) -> list[str]: + assert self.proc + resp = self.proc.call_tool("codedb_word", {"word": word}, timeout=15.0) + text = tool_text(resp) + ids = [] + for line in text.splitlines(): + m = re.search(r"([A-Za-z0-9_./-]+\.py)", line) + if m: + tid = path_to_id(m.group(1)) + if tid not in ids: + ids.append(tid) + return ids + + def symbol(self, name: str, max_results: int = 20) -> list[str]: + assert self.proc + resp = self.proc.call_tool( + "codedb_symbol", + {"name": name, "max_results": max_results, "format": "json", "fuzzy": True}, + timeout=15.0, + ) + text = tool_text(resp) + ids = [] + try: + data = json.loads(text) + rows = data.get("results") or data.get("symbols") or [] + if isinstance(rows, list): + for row in rows: + if isinstance(row, dict) and row.get("path"): + tid = path_to_id(row["path"]) + if tid not in ids: + ids.append(tid) + except json.JSONDecodeError: + for m in re.finditer(r"([A-Za-z0-9_./-]+\.py)", text): + tid = path_to_id(m.group(1)) + if tid not in ids: + ids.append(tid) + return ids + + def find(self, query: str, max_results: int = 20) -> list[str]: + assert self.proc + resp = self.proc.call_tool( + "codedb_find", + {"query": query, "max_results": max_results}, + timeout=15.0, + ) + ids = [] + for m in re.finditer(r"([A-Za-z0-9_./-]+\.py)", tool_text(resp)): + tid = path_to_id(m.group(1)) + if tid not in ids: + ids.append(tid) + return ids diff --git a/experiments/mcp-tool-search/lib/cost.py b/experiments/mcp-tool-search/lib/cost.py new file mode 100644 index 00000000..c4dc1bbe --- /dev/null +++ b/experiments/mcp-tool-search/lib/cost.py @@ -0,0 +1,121 @@ +"""Offline agent cost model: search-call counts and tokens burned.""" + +from __future__ import annotations + +import math +from typing import Any + +from lib.metrics import mean, percentile, relevant +from lib.schema_payload import all_schemas_payload, count_tokens, search_payload + +PAGE_KS = (3, 5, 10) +MAX_CALLS_CAP = 32 # safety; 536/3 ≈ 179 would be the true worst case + + +def complete_ranking(partial: list[str], catalog_ids: list[str]) -> list[str]: + seen = set(partial) + return list(partial) + [tid for tid in catalog_ids if tid not in seen] + + +def first_hit_rank(ranking: list[str], rel: set[str]) -> int | None: + for i, tid in enumerate(ranking, start=1): + if tid in rel: + return i + return None + + +def page_cost( + ranking: list[str], + rel: set[str], + by_id: dict[str, dict[str, Any]], + query: str, + k: int, +) -> dict[str, Any]: + """Page through `ranking` in slices of k until a relevant tool appears.""" + calls = 0 + tokens = 0 + pages: list[dict[str, Any]] = [] + hit_rank = first_hit_rank(ranking, rel) + if hit_rank is None: + return { + "calls": MAX_CALLS_CAP, + "tokens": 0, + "found": False, + "hit_rank": None, + "success_within": {1: 0.0, 2: 0.0, 3: 0.0}, + } + n_pages = max(1, math.ceil(hit_rank / k)) + for p in range(n_pages): + chunk_ids = ranking[p * k : (p + 1) * k] + chunk = [by_id[tid] for tid in chunk_ids if tid in by_id] + payload = search_payload(chunk, query, offset=p * k) + tok = count_tokens(payload) + calls += 1 + tokens += tok + pages.append({"offset": p * k, "ids": chunk_ids, "tokens": tok}) + return { + "calls": calls, + "tokens": tokens, + "found": True, + "hit_rank": hit_rank, + "success_within": {1: 1.0 if calls <= 1 else 0.0, 2: 1.0 if calls <= 2 else 0.0, 3: 1.0 if calls <= 3 else 0.0}, + "pages": pages, + } + + +def query_cost( + q: dict[str, Any], + ranking: list[str], + catalog_ids: list[str], + by_id: dict[str, dict[str, Any]], +) -> dict[str, Any]: + rel = relevant(q) + full = complete_ranking(ranking, catalog_ids) + row: dict[str, Any] = { + "id": q["id"], + "query": q["query"], + "hit_rank": first_hit_rank(full, rel), + } + for k in PAGE_KS: + c = page_cost(full, rel, by_id, q["query"], k) + row[f"k{k}_calls"] = c["calls"] + row[f"k{k}_tokens"] = c["tokens"] + row[f"k{k}_found"] = c["found"] + row[f"k{k}_success1"] = c["success_within"][1] + row[f"k{k}_success2"] = c["success_within"][2] + row[f"k{k}_success3"] = c["success_within"][3] + return row + + +def summarize_cost(rows: list[dict[str, Any]], k: int) -> dict[str, Any]: + calls = [float(r[f"k{k}_calls"]) for r in rows] + toks = [float(r[f"k{k}_tokens"]) for r in rows] + found = [r[f"k{k}_found"] for r in rows] + n_ok = sum(1 for f in found if f) + tok_ok = [t for t, f in zip(toks, found) if f] + return { + "k": k, + "n": len(rows), + "n_found": n_ok, + "mean_calls": mean(calls), + "p95_calls": percentile(calls, 95), + "success_within_1": mean([float(r[f"k{k}_success1"]) for r in rows]), + "success_within_2": mean([float(r[f"k{k}_success2"]) for r in rows]), + "success_within_3": mean([float(r[f"k{k}_success3"]) for r in rows]), + "mean_tokens": mean(toks), + "p95_tokens": percentile(toks, 95), + "total_tokens": int(sum(toks)), + "tokens_per_successful_query": mean(tok_ok) if tok_ok else float("inf"), + "total_tokens_successful": int(sum(tok_ok)), + } + + +def all_schemas_baseline(tools: list[dict[str, Any]]) -> dict[str, Any]: + payload = all_schemas_payload(tools) + n = count_tokens(payload) + return { + "n_tools": len(tools), + "tokens_per_turn": n, + "encoding": "o200k_base", + "payload_bytes": len(payload.encode("utf-8")), + } diff --git a/experiments/mcp-tool-search/lib/methods.py b/experiments/mcp-tool-search/lib/methods.py new file mode 100644 index 00000000..de6f7ab6 --- /dev/null +++ b/experiments/mcp-tool-search/lib/methods.py @@ -0,0 +1,775 @@ +"""Local reproductions of Zed-style MCP tool-search approaches.""" + +from __future__ import annotations + +import math +import re +import time +from collections import Counter, defaultdict +from dataclasses import dataclass, field +from typing import Any, Callable, Iterable + +VERBS = { + "create", + "get", + "list", + "update", + "delete", + "search", + "send", + "add", + "remove", + "set", + "fetch", + "post", + "put", + "run", + "query", + "make", + "open", + "show", + "read", + "write", + "find", +} + +IDENT_SPLIT = re.compile( + r"[^a-zA-Z0-9]+|(?<=[a-z])(?=[A-Z])|(?<=[A-Z])(?=[A-Z][a-z])|(?<=[a-zA-Z])(?=[0-9])|(?<=[0-9])(?=[a-zA-Z])" +) + + +def split_ident(text: str) -> list[str]: + parts: list[str] = [] + for raw in IDENT_SPLIT.split(text): + tok = raw.strip().lower() + if len(tok) >= 2: + parts.append(tok) + return parts + + +def tokenize(text: str, *, split_idents: bool = True) -> list[str]: + if not text: + return [] + if split_idents: + return split_ident(text) + return [t for t in re.split(r"[^a-z0-9]+", text.lower()) if len(t) >= 2] + + +@dataclass +class Ranked: + tool_id: str + score: float + + +class Timer: + def __init__(self) -> None: + self.samples_ms: list[float] = [] + + def add(self, seconds: float) -> None: + self.samples_ms.append(seconds * 1000.0) + + def percentile(self, p: float) -> float: + if not self.samples_ms: + return 0.0 + xs = sorted(self.samples_ms) + if len(xs) == 1: + return xs[0] + k = (len(xs) - 1) * (p / 100.0) + f = int(math.floor(k)) + c = int(math.ceil(k)) + if f == c: + return xs[f] + return xs[f] * (c - k) + xs[c] * (k - f) + + +def rrf(rank_lists: list[list[str]], k: int = 60) -> list[Ranked]: + scores: dict[str, float] = defaultdict(float) + for ranks in rank_lists: + for i, tid in enumerate(ranks): + scores[tid] += 1.0 / (k + i + 1) + return [Ranked(tid, s) for tid, s in sorted(scores.items(), key=lambda x: -x[1])] + + +class BM25Index: + def __init__(self, k1: float = 1.5, b: float = 0.75, plus_delta: float = 0.0) -> None: + self.k1 = k1 + self.b = b + self.plus_delta = plus_delta + self.doc_ids: list[str] = [] + self.tf: list[Counter[str]] = [] + self.dl: list[int] = [] + self.df: Counter[str] = Counter() + self.avgdl = 0.0 + self.n = 0 + + def build(self, docs: list[tuple[str, list[str]]]) -> float: + t0 = time.perf_counter() + self.doc_ids = [] + self.tf = [] + self.dl = [] + self.df = Counter() + for did, tokens in docs: + self.doc_ids.append(did) + c = Counter(tokens) + self.tf.append(c) + self.dl.append(len(tokens)) + self.df.update(c.keys()) + self.n = len(self.doc_ids) + self.avgdl = (sum(self.dl) / self.n) if self.n else 0.0 + return time.perf_counter() - t0 + + def idf(self, term: str) -> float: + df = self.df.get(term, 0) + return math.log(1.0 + (self.n - df + 0.5) / (df + 0.5)) + + def score_query(self, q_tokens: list[str], term_weights: dict[str, float] | None = None) -> list[Ranked]: + scores = [0.0] * self.n + seen: set[str] = set() + avgdl = self.avgdl or 1.0 + for term in q_tokens: + if term in seen: + continue + seen.add(term) + if term not in self.df: + continue + idf = self.idf(term) + w = 1.0 if term_weights is None else term_weights.get(term, 1.0) + for i, tfmap in enumerate(self.tf): + tf = tfmap.get(term, 0) + if tf == 0: + continue + denom = tf + self.k1 * (1.0 - self.b + self.b * self.dl[i] / avgdl) + scores[i] += w * idf * (tf * (self.k1 + 1.0) / denom + self.plus_delta) + ranked = [Ranked(self.doc_ids[i], s) for i, s in enumerate(scores) if s > 0] + ranked.sort(key=lambda r: -r.score) + return ranked + + +class TfidfIndex: + def __init__(self) -> None: + self.doc_ids: list[str] = [] + self.tf: list[Counter[str]] = [] + self.df: Counter[str] = Counter() + self.idf: dict[str, float] = {} + self.norms: list[float] = [] + self.n = 0 + + def build(self, docs: list[tuple[str, list[str]]]) -> float: + t0 = time.perf_counter() + self.doc_ids = [] + self.tf = [] + self.df = Counter() + for did, tokens in docs: + self.doc_ids.append(did) + c = Counter(tokens) + self.tf.append(c) + self.df.update(c.keys()) + self.n = len(self.doc_ids) + self.idf = { + term: math.log((self.n - df + 0.5) / (df + 0.5) + 1.0) for term, df in self.df.items() + } + self.norms = [] + for c in self.tf: + acc = 0.0 + for term, tf in c.items(): + w = (1.0 + math.log(tf)) * self.idf[term] + acc += w * w + self.norms.append(math.sqrt(acc) if acc else 1.0) + return time.perf_counter() - t0 + + def score_query(self, q_tokens: list[str]) -> list[Ranked]: + qtf = Counter(q_tokens) + qvec: dict[str, float] = {} + qn = 0.0 + for term, tf in qtf.items(): + if term not in self.idf: + continue + w = (1.0 + math.log(tf)) * self.idf[term] + qvec[term] = w + qn += w * w + qn = math.sqrt(qn) if qn else 1.0 + ranked: list[Ranked] = [] + for i, c in enumerate(self.tf): + dot = 0.0 + for term, qw in qvec.items(): + tf = c.get(term, 0) + if tf: + dot += qw * (1.0 + math.log(tf)) * self.idf[term] + if dot > 0: + ranked.append(Ranked(self.doc_ids[i], dot / (self.norms[i] * qn))) + ranked.sort(key=lambda r: -r.score) + return ranked + + +class TrigramIndex: + def __init__(self) -> None: + self.doc_ids: list[str] = [] + self.sets: list[set[str]] = [] + + def _grams(self, text: str) -> set[str]: + s = f" {text.lower()} " + return {s[i : i + 3] for i in range(len(s) - 2)} + + def build(self, docs: list[tuple[str, str]]) -> float: + t0 = time.perf_counter() + self.doc_ids = [] + self.sets = [] + for did, text in docs: + self.doc_ids.append(did) + self.sets.append(self._grams(text)) + return time.perf_counter() - t0 + + def score_query(self, query: str) -> list[Ranked]: + q = self._grams(query) + if not q: + return [] + ranked: list[Ranked] = [] + for did, grams in zip(self.doc_ids, self.sets): + inter = len(q & grams) + if inter: + ranked.append(Ranked(did, inter / math.sqrt(len(q) * len(grams)))) + ranked.sort(key=lambda r: -r.score) + return ranked + + +class HashedEmbedIndex: + """Fallback local embedding: hashed character n-grams, not a neural model.""" + + def __init__(self, dim: int = 256) -> None: + self.dim = dim + self.doc_ids: list[str] = [] + self.vecs: list[list[float]] = [] + + def _embed(self, text: str) -> list[float]: + v = [0.0] * self.dim + s = text.lower() + for n in (3, 4): + for i in range(max(0, len(s) - n + 1)): + h = hash(s[i : i + n]) + v[h % self.dim] += 1.0 + v[(h >> 8) % self.dim] -= 0.35 + nrm = math.sqrt(sum(x * x for x in v)) or 1.0 + return [x / nrm for x in v] + + def build(self, docs: list[tuple[str, str]]) -> float: + t0 = time.perf_counter() + self.doc_ids = [d for d, _ in docs] + self.vecs = [self._embed(text) for _, text in docs] + return time.perf_counter() - t0 + + def score_query(self, query: str) -> list[Ranked]: + qv = self._embed(query) + ranked = [] + for did, dv in zip(self.doc_ids, self.vecs): + score = sum(a * b for a, b in zip(qv, dv)) + if score > 0: + ranked.append(Ranked(did, score)) + ranked.sort(key=lambda r: -r.score) + return ranked + + +class DenseEmbedIndex: + def __init__(self, model_name: str = "BAAI/bge-small-en-v1.5") -> None: + self.model_name = model_name + self.doc_ids: list[str] = [] + self.vecs = None + self.model = None + + def available(self) -> bool: + try: + from fastembed import TextEmbedding # noqa: F401 + + return True + except Exception: + return False + + def build(self, docs: list[tuple[str, str]]) -> float: + from fastembed import TextEmbedding + import numpy as np + + t0 = time.perf_counter() + self.model = TextEmbedding(model_name=self.model_name) + self.doc_ids = [d for d, _ in docs] + texts = [text for _, text in docs] + mat = np.vstack(list(self.model.embed(texts))) + norms = np.linalg.norm(mat, axis=1, keepdims=True) + norms[norms == 0] = 1.0 + self.vecs = mat / norms + return time.perf_counter() - t0 + + def score_query(self, query: str) -> list[Ranked]: + import numpy as np + + assert self.model is not None and self.vecs is not None + q = np.vstack(list(self.model.embed([query])))[0] + n = float(np.linalg.norm(q)) or 1.0 + q = q / n + sims = self.vecs @ q + order = np.argsort(-sims) + ranked = [] + for i in order: + if sims[i] <= 0: + break + ranked.append(Ranked(self.doc_ids[int(i)], float(sims[i]))) + return ranked + + +@dataclass +class MethodMeta: + name: str + needs_model: bool + needs_api: bool + notes: str = "" + + +@dataclass +class MethodResult: + name: str + rankings: dict[str, list[str]] + query_ms: list[float] + index_s: float + rss_mb: float | None + meta: MethodMeta + skipped: str | None = None + + +def rss_mb() -> float | None: + try: + with open("/proc/self/status", encoding="utf-8") as f: + for line in f: + if line.startswith("VmRSS:"): + return int(line.split()[1]) / 1024.0 + except OSError: + return None + return None + + +def top_ids(ranked: Iterable[Ranked], k: int = 10) -> list[str]: + out = [] + for r in ranked: + if r.tool_id not in out: + out.append(r.tool_id) + if len(out) >= k: + break + return out + + +class CatalogSearch: + def __init__(self, tools: list[dict[str, Any]]) -> None: + self.tools = tools + self.by_id = {t["id"]: t for t in tools} + self.aliases: dict[str, str] = {} + for t in tools: + for a in t.get("aliases") or [t["server"]]: + self.aliases[a.lower()] = t["server"] + + def fields(self, t: dict[str, Any]) -> dict[str, str]: + return { + "name": t["name"], + "server": t["server"] + " " + " ".join(t.get("aliases") or []), + "description": t["description"], + "params": " ".join(t.get("params") or []), + "full": f"{t['server']} {t['name']} {t['description']} {' '.join(t.get('params') or [])}", + } + + def query_term_weights(self, tokens: list[str]) -> dict[str, float]: + w = {} + for tok in tokens: + w[tok] = 0.25 if tok in VERBS else 1.0 + return w + + def namespace_boost(self, query: str, tool_id: str, score: float) -> float: + t = self.by_id[tool_id] + q = query.lower() + for alias in t.get("aliases") or [t["server"]]: + if alias.lower() in q: + return score * 1.35 + return score + + def run_all(self, queries: list[dict[str, Any]], *, want_dense: bool = True) -> list[MethodResult]: + results: list[MethodResult] = [] + full_docs = [(t["id"], tokenize(self.fields(t)["full"])) for t in self.tools] + name_docs = [(t["id"], tokenize(t["name"])) for t in self.tools] + desc_docs = [(t["id"], tokenize(t["description"])) for t in self.tools] + field_docs = { + "name": [(t["id"], tokenize(self.fields(t)["name"])) for t in self.tools], + "server": [(t["id"], tokenize(self.fields(t)["server"])) for t in self.tools], + "description": [(t["id"], tokenize(self.fields(t)["description"])) for t in self.tools], + "params": [(t["id"], tokenize(self.fields(t)["params"])) for t in self.tools], + } + texts = [(t["id"], self.fields(t)["full"]) for t in self.tools] + + def eval_method( + name: str, + index_s: float, + scorer: Callable[[dict[str, Any]], list[Ranked]], + meta: MethodMeta, + ) -> MethodResult: + rankings: dict[str, list[str]] = {} + qms: list[float] = [] + before = rss_mb() + for q in queries: + t0 = time.perf_counter() + ranked = scorer(q) + qms.append((time.perf_counter() - t0) * 1000.0) + rankings[q["id"]] = top_ids(ranked, 10) + after = rss_mb() + rss = None + if before is not None and after is not None: + rss = max(0.0, after - before) if after >= before else after + return MethodResult(name, rankings, qms, index_s, rss, meta) + + # 1. name exact / substring + def name_exact(q: dict[str, Any]) -> list[Ranked]: + needle = re.sub(r"[^a-z0-9_]+", "", q["query"].lower()) + out = [] + for t in self.tools: + compact = re.sub(r"[^a-z0-9_]+", "", t["name"].lower()) + if compact == needle or t["name"].lower() == q["query"].lower(): + out.append(Ranked(t["id"], 2.0)) + elif needle and needle in compact: + out.append(Ranked(t["id"], 1.0)) + out.sort(key=lambda r: -r.score) + return out + + results.append( + eval_method( + "name_exact_substr", + 0.0, + name_exact, + MethodMeta("name_exact_substr", False, False, "exact/substring on tool name"), + ) + ) + + def boolean_and(q: dict[str, Any]) -> list[Ranked]: + toks = tokenize(q["query"]) + if not toks: + return [] + out = [] + for tid, tokens in full_docs: + bag = set(tokens) + if all(t in bag for t in toks): + out.append(Ranked(tid, float(len(toks)))) + return out + + results.append( + eval_method("boolean_and", 0.0, boolean_and, MethodMeta("boolean_and", False, False, "all query tokens must match")) + ) + + def boolean_or(q: dict[str, Any]) -> list[Ranked]: + toks = tokenize(q["query"]) + out = [] + for tid, tokens in full_docs: + bag = set(tokens) + hit = sum(1 for t in toks if t in bag) + if hit: + out.append(Ranked(tid, float(hit))) + out.sort(key=lambda r: -r.score) + return out + + results.append( + eval_method("boolean_or", 0.0, boolean_or, MethodMeta("boolean_or", False, False, "token overlap count")) + ) + + bm25 = BM25Index() + t_bm25 = bm25.build(full_docs) + + def plain_bm25(q: dict[str, Any]) -> list[Ranked]: + return bm25.score_query(tokenize(q["query"], split_idents=False)) + + results.append( + eval_method( + "bm25_plain", + t_bm25, + plain_bm25, + MethodMeta("bm25_plain", False, False, "Okapi BM25, split on non-alnum only"), + ) + ) + + def bm25_ident(q: dict[str, Any]) -> list[Ranked]: + return bm25.score_query(tokenize(q["query"], split_idents=True)) + + results.append( + eval_method( + "bm25_ident_split", + t_bm25, + bm25_ident, + MethodMeta("bm25_ident_split", False, False, "BM25 + identifier splitting"), + ) + ) + + bm25_name = BM25Index() + t_name = bm25_name.build(name_docs) + results.append( + eval_method( + "bm25_name_only", + t_name, + lambda q: bm25_name.score_query(tokenize(q["query"])), + MethodMeta("bm25_name_only", False, False, "BM25 over tool name field"), + ) + ) + + bm25_desc = BM25Index() + t_desc = bm25_desc.build(desc_docs) + results.append( + eval_method( + "bm25_desc_only", + t_desc, + lambda q: bm25_desc.score_query(tokenize(q["query"])), + MethodMeta("bm25_desc_only", False, False, "BM25 over description field"), + ) + ) + + field_indexes = {} + t_fields = 0.0 + for fname, docs in field_docs.items(): + ix = BM25Index() + t_fields += ix.build(docs) + field_indexes[fname] = ix + field_w = {"name": 3.0, "server": 2.2, "description": 1.0, "params": 0.45} + + def weighted_bm25(q: dict[str, Any]) -> list[Ranked]: + toks = tokenize(q["query"]) + tw = self.query_term_weights(toks) + scores: dict[str, float] = defaultdict(float) + for fname, ix in field_indexes.items(): + for r in ix.score_query(toks, tw): + scores[r.tool_id] += field_w[fname] * r.score + ranked = [Ranked(tid, self.namespace_boost(q["query"], tid, s)) for tid, s in scores.items()] + ranked.sort(key=lambda r: -r.score) + return ranked + + results.append( + eval_method( + "bm25_weighted", + t_fields, + weighted_bm25, + MethodMeta( + "bm25_weighted", + False, + False, + "Zed-like: field BM25 + ident split + verb downweight + name/namespace boosts", + ), + ) + ) + + tfidf = TfidfIndex() + t_tfidf = tfidf.build(full_docs) + results.append( + eval_method( + "tfidf", + t_tfidf, + lambda q: tfidf.score_query(tokenize(q["query"])), + MethodMeta("tfidf", False, False, "cosine TF-IDF"), + ) + ) + + def bm25_tfidf_hybrid(q: dict[str, Any]) -> list[Ranked]: + toks = tokenize(q["query"]) + b = {r.tool_id: r.score for r in bm25.score_query(toks)} + t = {r.tool_id: r.score for r in tfidf.score_query(toks)} + ids = set(b) | set(t) + def norm(m: dict[str, float]) -> dict[str, float]: + mx = max(m.values()) if m else 1.0 + return {k: v / mx for k, v in m.items()} if mx else m + bn, tn = norm(b), norm(t) + ranked = [Ranked(i, 0.2 * bn.get(i, 0.0) + 0.8 * tn.get(i, 0.0)) for i in ids] + ranked.sort(key=lambda r: -r.score) + return ranked + + results.append( + eval_method( + "bm25_tfidf_hybrid", + t_bm25 + t_tfidf, + bm25_tfidf_hybrid, + MethodMeta("bm25_tfidf_hybrid", False, False, "StackOne-style 20/80 BM25+TF-IDF"), + ) + ) + + tri = TrigramIndex() + t_tri = tri.build(texts) + results.append( + eval_method( + "char_trigram", + t_tri, + lambda q: tri.score_query(q["query"]), + MethodMeta("char_trigram", False, False, "character trigram cosine — local analog of codedb trigrams"), + ) + ) + + hashed = HashedEmbedIndex() + t_hash = hashed.build(texts) + results.append( + eval_method( + "hashed_ngram_embed", + t_hash, + lambda q: hashed.score_query(q["query"]), + MethodMeta("hashed_ngram_embed", False, False, "fallback sparse hashed n-gram embedding, not neural"), + ) + ) + + def hybrid_rrf(a: Callable[[dict[str, Any]], list[Ranked]], b: Callable[[dict[str, Any]], list[Ranked]]): + def inner(q: dict[str, Any]) -> list[Ranked]: + return rrf([top_ids(a(q), 50), top_ids(b(q), 50)]) + return inner + + results.append( + eval_method( + "hybrid_rrf_bm25_trigram", + t_bm25 + t_tri, + hybrid_rrf(lambda q: bm25.score_query(tokenize(q["query"])), lambda q: tri.score_query(q["query"])), + MethodMeta("hybrid_rrf_bm25_trigram", False, False, "RRF of BM25 + char trigram"), + ) + ) + results.append( + eval_method( + "hybrid_rrf_weighted_trigram", + t_fields + t_tri, + hybrid_rrf(weighted_bm25, lambda q: tri.score_query(q["query"])), + MethodMeta("hybrid_rrf_weighted_trigram", False, False, "RRF of weighted BM25 + char trigram"), + ) + ) + results.append( + eval_method( + "hybrid_rrf_weighted_hashed", + t_fields + t_hash, + hybrid_rrf(weighted_bm25, lambda q: hashed.score_query(q["query"])), + MethodMeta("hybrid_rrf_weighted_hashed", False, False, "RRF of weighted BM25 + hashed embedding"), + ) + ) + + def decision_router(q: dict[str, Any]) -> list[Ranked]: + toks = tokenize(q["query"]) + compact = re.sub(r"[^a-z0-9]+", "", q["query"].lower()) + looks_ident = " " not in q["query"].strip() and ( "_" in q["query"] or any(c.isupper() for c in q["query"])) + if looks_ident or compact in {re.sub(r"[^a-z0-9]+", "", t["name"].lower()) for t in self.tools}: + return name_exact(q) or weighted_bm25(q) + return weighted_bm25(q) + + results.append( + eval_method( + "decision_heuristic", + t_fields, + decision_router, + MethodMeta( + "decision_heuristic", + False, + False, + "rule-based router (identifier → name match, else weighted BM25); not a trained decision model", + ), + ) + ) + + def rewrite_bm25(q: dict[str, Any]) -> list[Ranked]: + text = q["query"] + repl = { + "ticket": "issue", + "pr": "pull request", + "mr": "merge request", + "repo": "repository", + "sms": "sms message", + "on call": "oncall", + "k8s": "kubernetes", + } + low = text.lower() + for a, b in repl.items(): + low = re.sub(rf"\b{re.escape(a)}\b", b, low) + return weighted_bm25({"query": low, "id": q["id"]}) + + results.append( + eval_method( + "query_rewrite_weighted_bm25", + t_fields, + rewrite_bm25, + MethodMeta("query_rewrite_weighted_bm25", False, False, "tiny synonym rewrite then weighted BM25 (not an LLM)"), + ) + ) + + def rerank_overlap(q: dict[str, Any]) -> list[Ranked]: + base = weighted_bm25(q)[:20] + qtok = set(tokenize(q["query"])) + reranked = [] + for r in base: + t = self.by_id[r.tool_id] + bag = set(tokenize(self.fields(t)["full"])) + overlap = len(qtok & bag) / (len(qtok) or 1) + rare = sum(1 for tok in qtok if tok not in VERBS and tok in bag) + reranked.append(Ranked(r.tool_id, r.score + 2.0 * overlap + 1.5 * rare)) + reranked.sort(key=lambda x: -x.score) + return reranked + + results.append( + eval_method( + "lexical_rerank_weighted_top20", + t_fields, + rerank_overlap, + MethodMeta("lexical_rerank_weighted_top20", False, False, "lexical overlap rerank of weighted BM25 top-20; not a cross-encoder"), + ) + ) + + dense = DenseEmbedIndex() + if want_dense and dense.available(): + try: + t_dense = dense.build(texts) + + def dense_score(q: dict[str, Any]) -> list[Ranked]: + return dense.score_query(q["query"]) + + results.append( + eval_method( + "embed_bge_small", + t_dense, + dense_score, + MethodMeta("embed_bge_small", True, False, "fastembed BAAI/bge-small-en-v1.5"), + ) + ) + results.append( + eval_method( + "hybrid_rrf_bm25_embed", + t_bm25 + t_dense, + hybrid_rrf(lambda q: bm25.score_query(tokenize(q["query"])), dense_score), + MethodMeta("hybrid_rrf_bm25_embed", True, False, "RRF BM25 + bge-small"), + ) + ) + results.append( + eval_method( + "hybrid_rrf_weighted_embed", + t_fields + t_dense, + hybrid_rrf(weighted_bm25, dense_score), + MethodMeta("hybrid_rrf_weighted_embed", True, False, "RRF weighted BM25 + bge-small"), + ) + ) + results.append( + eval_method( + "hybrid_rrf_trigram_embed", + t_tri + t_dense, + hybrid_rrf(lambda q: tri.score_query(q["query"]), dense_score), + MethodMeta("hybrid_rrf_trigram_embed", True, False, "RRF char-trigram + bge-small"), + ) + ) + except Exception as exc: + results.append( + MethodResult( + "embed_bge_small", + {}, + [], + 0.0, + None, + MethodMeta("embed_bge_small", True, False, "fastembed failed"), + skipped=str(exc), + ) + ) + else: + results.append( + MethodResult( + "embed_bge_small", + {}, + [], + 0.0, + None, + MethodMeta("embed_bge_small", True, False, "fastembed not installed"), + skipped="fastembed not installed", + ) + ) + + return results diff --git a/experiments/mcp-tool-search/lib/metrics.py b/experiments/mcp-tool-search/lib/metrics.py new file mode 100644 index 00000000..333e6c0d --- /dev/null +++ b/experiments/mcp-tool-search/lib/metrics.py @@ -0,0 +1,126 @@ +"""IR metrics + bootstrap CIs for the MCP tool-search harness.""" + +from __future__ import annotations + +import math +import random +from typing import Any + + +def relevant(q: dict[str, Any]) -> set[str]: + return set(q.get("gold") or []) | set(q.get("alts") or []) + + +def gold_gain(q: dict[str, Any], tid: str) -> float: + if tid in (q.get("gold") or []): + return 2.0 + if tid in (q.get("alts") or []): + return 1.0 + return 0.0 + + +def recall_at(ranking: list[str], rel: set[str], k: int) -> float: + if not rel: + return 0.0 + return 1.0 if any(t in rel for t in ranking[:k]) else 0.0 + + +def mrr_at(ranking: list[str], rel: set[str], k: int = 10) -> float: + for i, tid in enumerate(ranking[:k], start=1): + if tid in rel: + return 1.0 / i + return 0.0 + + +def dcg(gains: list[float]) -> float: + return sum(g / math.log2(i + 2) for i, g in enumerate(gains)) + + +def ndcg_at(ranking: list[str], q: dict[str, Any], k: int = 10) -> float: + gains = [gold_gain(q, tid) for tid in ranking[:k]] + ideal = sorted( + [gold_gain(q, tid) for tid in relevant(q)], + reverse=True, + )[:k] + if not ideal or dcg(ideal) == 0: + return 0.0 + return dcg(gains) / dcg(ideal) + + +def per_query_row(q: dict[str, Any], ranking: list[str]) -> dict[str, Any]: + rel = relevant(q) + return { + "id": q["id"], + "query": q["query"], + "kind": q.get("kind"), + "gold": q.get("gold"), + "alts": q.get("alts"), + "ranking": ranking, + "hit1": ranking[0] if ranking else None, + "recall@1": recall_at(ranking, rel, 1), + "recall@3": recall_at(ranking, rel, 3), + "recall@5": recall_at(ranking, rel, 5), + "mrr@10": mrr_at(ranking, rel, 10), + "ndcg@10": ndcg_at(ranking, q, 10), + } + + +def mean(xs: list[float]) -> float: + return sum(xs) / len(xs) if xs else 0.0 + + +def percentile(xs: list[float], p: float) -> float: + if not xs: + return 0.0 + ys = sorted(xs) + if len(ys) == 1: + return ys[0] + k = (len(ys) - 1) * (p / 100.0) + f = int(math.floor(k)) + c = int(math.ceil(k)) + if f == c: + return ys[f] + return ys[f] * (c - k) + ys[c] * (k - f) + + +def bootstrap_ci( + values: list[float], + n: int = 2000, + seed: int = 20261020, +) -> tuple[float, float, float]: + rng = random.Random(seed) + if not values: + return 0.0, 0.0, 0.0 + means = [] + m = len(values) + for _ in range(n): + sample = [values[rng.randrange(m)] for _ in range(m)] + means.append(sum(sample) / m) + means.sort() + lo = means[int(0.025 * n)] + hi = means[min(n - 1, int(0.975 * n))] + return mean(values), lo, hi + + +def summarize( + queries: list[dict[str, Any]], + rankings: dict[str, list[str]], + query_ms: list[float], + index_s: float, + rss_mb: float | None, +) -> dict[str, Any]: + rows = [per_query_row(q, rankings.get(q["id"], [])) for q in queries] + out: dict[str, Any] = { + "n": len(rows), + "index_s": index_s, + "rss_mb": rss_mb, + "p50_ms": percentile(query_ms, 50), + "p95_ms": percentile(query_ms, 95), + } + for key in ("recall@1", "recall@3", "recall@5", "mrr@10", "ndcg@10"): + vals = [float(r[key]) for r in rows] + mu, lo, hi = bootstrap_ci(vals) + out[key] = mu + out[f"{key}_lo"] = lo + out[f"{key}_hi"] = hi + return out diff --git a/experiments/mcp-tool-search/lib/schema_payload.py b/experiments/mcp-tool-search/lib/schema_payload.py new file mode 100644 index 00000000..08460fde --- /dev/null +++ b/experiments/mcp-tool-search/lib/schema_payload.py @@ -0,0 +1,62 @@ +"""MCP-style tool schemas and o200k_base token counts.""" + +from __future__ import annotations + +import json +from functools import lru_cache +from typing import Any + +ENCODING_NAME = "o200k_base" + + +@lru_cache(maxsize=1) +def _enc(): + import tiktoken + + return tiktoken.get_encoding(ENCODING_NAME) + + +def count_tokens(text: str) -> int: + return len(_enc().encode(text)) + + +def input_schema(tool: dict[str, Any]) -> dict[str, Any]: + props: dict[str, Any] = {} + required: list[str] = [] + for p in tool.get("params") or []: + props[p] = { + "type": "string", + "description": f"{p.replace('_', ' ')} for {tool['name']}", + } + if p in ("id", "query", "path", "owner", "repo", "title", "name", "sql", "soql"): + required.append(p) + schema: dict[str, Any] = {"type": "object", "properties": props} + if required: + schema["required"] = required + return schema + + +def tool_schema(tool: dict[str, Any]) -> dict[str, Any]: + """One tool as an MCP tools/list / search_tools hit.""" + return { + "name": tool["id"], + "description": tool["description"], + "inputSchema": input_schema(tool), + "_meta": {"server": tool["server"]}, + } + + +def search_payload(tools: list[dict[str, Any]], query: str, offset: int = 0) -> str: + """Compact JSON a search_tools MCP call would return to the model.""" + body = { + "query": query, + "offset": offset, + "count": len(tools), + "tools": [tool_schema(t) for t in tools], + } + return json.dumps(body, ensure_ascii=False, separators=(",", ":"), sort_keys=True) + + +def all_schemas_payload(tools: list[dict[str, Any]]) -> str: + body = {"tools": [tool_schema(t) for t in tools]} + return json.dumps(body, ensure_ascii=False, separators=(",", ":"), sort_keys=True) diff --git a/experiments/mcp-tool-search/live_agent.py b/experiments/mcp-tool-search/live_agent.py new file mode 100644 index 00000000..5b6034da --- /dev/null +++ b/experiments/mcp-tool-search/live_agent.py @@ -0,0 +1,317 @@ +#!/usr/bin/env python3 +"""Live agent loop: a small model + search_tools backed by one approach. + +Skipped automatically when no LLM API key is present. Ready to run: + + OPENAI_API_KEY=... python live_agent.py + ANTHROPIC_API_KEY=... python live_agent.py + +Uses gpt-4o-mini or claude-haiku-4-5. Each of the 150 queries is a fresh +conversation with one tool, `search_tools`, then the model must name the +tool it would call. We record API usage tokens, search-call count, whether +the named tool is gold/alt, and wall time. +""" + +from __future__ import annotations + +import argparse +import json +import os +import sys +import time +from pathlib import Path + +ROOT = Path(__file__).resolve().parent +sys.path.insert(0, str(ROOT)) + +from lib.catalog import load_frozen # noqa: E402 +from lib.cost import complete_ranking, first_hit_rank # noqa: E402 +from lib.metrics import relevant # noqa: E402 +from lib.schema_payload import search_payload # noqa: E402 + +RESULTS = ROOT / "results" +DEFAULT_APPROACHES = ( + "bm25_plain", + "bm25_weighted", + "embed_bge_small", + "hybrid_rrf_bm25_embed", + "codedb_search", + "char_trigram", +) + + +CHECKED_KEYS = ( + "OPENAI_API_KEY", + "ANTHROPIC_API_KEY", + "AZURE_OPENAI_API_KEY", + "GOOGLE_API_KEY", + "GEMINI_API_KEY", + "COHERE_API_KEY", + "TOGETHER_API_KEY", + "GROQ_API_KEY", + "MISTRAL_API_KEY", + "OPENROUTER_API_KEY", +) + + +def detect_provider() -> tuple[str, str] | None: + if os.environ.get("OPENAI_API_KEY"): + return "openai", os.environ.get("LIVE_AGENT_MODEL") or "gpt-4o-mini" + if os.environ.get("ANTHROPIC_API_KEY"): + return "anthropic", os.environ.get("LIVE_AGENT_MODEL") or "claude-haiku-4-5" + return None + + +def skip_reason() -> str: + present = [k for k in CHECKED_KEYS if os.environ.get(k)] + if present: + return ( + f"Keys present ({', '.join(present)}) but no supported live-loop " + "provider adapter (need OPENAI_API_KEY or ANTHROPIC_API_KEY)." + ) + return ( + "No LLM API key in the environment (checked " + + ", ".join(CHECKED_KEYS) + + "). Offline cost model still ran. To execute the live loop: " + "OPENAI_API_KEY=... python live_agent.py" + ) + + +def load_rankings(name: str) -> dict[str, list[str]]: + p = RESULTS / f"{name}.json" + if not p.exists(): + raise FileNotFoundError(p) + data = json.loads(p.read_text(encoding="utf-8")) + return {r["id"]: r.get("ranking") or [] for r in data["rows"]} + + +SYSTEM = """You pick MCP tools. You do not have the catalog in context. +Use search_tools to find a tool for the user request, then answer with ONLY +the tool id (server.name) you would call. Do not call a tool that was not +returned by search_tools. If the first page is wrong, search again with a +narrower query.""" + + +def openai_loop(model: str, query: str, page_fn, k: int, max_calls: int) -> dict: + from openai import OpenAI + + client = OpenAI() + tools = [ + { + "type": "function", + "function": { + "name": "search_tools", + "description": "Search the MCP tool catalog. Returns the top-k tool schemas.", + "parameters": { + "type": "object", + "properties": {"query": {"type": "string"}}, + "required": ["query"], + }, + }, + } + ] + messages = [ + {"role": "system", "content": SYSTEM}, + {"role": "user", "content": query["query"]}, + ] + search_calls = 0 + in_tok = 0 + out_tok = 0 + named = None + t0 = time.perf_counter() + for _ in range(max_calls + 2): + resp = client.chat.completions.create(model=model, messages=messages, tools=tools, temperature=0) + if resp.usage: + in_tok += resp.usage.prompt_tokens or 0 + out_tok += resp.usage.completion_tokens or 0 + msg = resp.choices[0].message + if msg.tool_calls: + messages.append(msg.model_dump(exclude_unset=True)) + for tc in msg.tool_calls: + if tc.function.name != "search_tools": + continue + search_calls += 1 + args = json.loads(tc.function.arguments or "{}") + payload = page_fn(args.get("query") or query["query"], search_calls) + messages.append( + {"role": "tool", "tool_call_id": tc.id, "content": payload} + ) + continue + named = (msg.content or "").strip().split()[0] if msg.content else None + break + return { + "search_calls": search_calls, + "input_tokens": in_tok, + "output_tokens": out_tok, + "named_tool": named, + "wall_s": time.perf_counter() - t0, + } + + +def anthropic_loop(model: str, query: str, page_fn, k: int, max_calls: int) -> dict: + import anthropic + + client = anthropic.Anthropic() + tools = [ + { + "name": "search_tools", + "description": "Search the MCP tool catalog. Returns the top-k tool schemas.", + "input_schema": { + "type": "object", + "properties": {"query": {"type": "string"}}, + "required": ["query"], + }, + } + ] + messages = [{"role": "user", "content": query["query"]}] + search_calls = 0 + in_tok = 0 + out_tok = 0 + named = None + t0 = time.perf_counter() + for _ in range(max_calls + 2): + resp = client.messages.create( + model=model, + max_tokens=256, + system=SYSTEM, + tools=tools, + messages=messages, + temperature=0, + ) + in_tok += resp.usage.input_tokens + out_tok += resp.usage.output_tokens + if resp.stop_reason == "tool_use": + messages.append({"role": "assistant", "content": resp.content}) + tool_results = [] + for block in resp.content: + if block.type != "tool_use": + continue + search_calls += 1 + q = (block.input or {}).get("query") or query["query"] + tool_results.append( + { + "type": "tool_result", + "tool_use_id": block.id, + "content": page_fn(q, search_calls), + } + ) + messages.append({"role": "user", "content": tool_results}) + continue + text = "".join(b.text for b in resp.content if getattr(b, "type", "") == "text") + named = text.strip().split()[0] if text.strip() else None + break + return { + "search_calls": search_calls, + "input_tokens": in_tok, + "output_tokens": out_tok, + "named_tool": named, + "wall_s": time.perf_counter() - t0, + } + + +def make_page_fn(ranking: list[str], by_id: dict, catalog_ids: list[str], k: int, original: str): + full = complete_ranking(ranking, catalog_ids) + + def page_fn(search_query: str, call_n: int) -> str: + # Offset paging on the original ranking. A rewritten query still + # pages the same list in this harness (documented in GAPS.md) unless + # the caller re-runs a real index; keep it deterministic + cheap. + start = max(0, (call_n - 1) * k) + ids = full[start : start + k] + tools = [by_id[i] for i in ids if i in by_id] + return search_payload(tools, search_query or original, offset=start) + + return page_fn + + +def run_live(approaches: list[str], k: int, max_calls: int, limit: int | None) -> dict: + provider = detect_provider() + if provider is None: + return { + "skipped": True, + "reason": skip_reason(), + "checked_keys": list(CHECKED_KEYS), + "command": "OPENAI_API_KEY=... python live_agent.py", + } + kind, model = provider + tools, queries = load_frozen() + if limit: + queries = queries[:limit] + by_id = {t["id"]: t for t in tools} + catalog_ids = [t["id"] for t in tools] + loop = openai_loop if kind == "openai" else anthropic_loop + out_approaches = [] + for name in approaches: + rankings = load_rankings(name) + rows = [] + for q in queries: + page_fn = make_page_fn(rankings.get(q["id"], []), by_id, catalog_ids, k, q["query"]) + rec = loop(model, q, page_fn, k, max_calls) + rel = relevant(q) + named = rec["named_tool"] + rec.update( + { + "id": q["id"], + "query": q["query"], + "gold": q["gold"], + "correct": bool(named and named in rel), + "gold_in_first_page": first_hit_rank( + complete_ranking(rankings.get(q["id"], []), catalog_ids), rel + ) + is not None + and first_hit_rank(complete_ranking(rankings.get(q["id"], []), catalog_ids), rel) + <= k, + } + ) + rows.append(rec) + print( + f"{name} {q['id']} calls={rec['search_calls']} " + f"in={rec['input_tokens']} out={rec['output_tokens']} " + f"named={named} ok={rec['correct']}" + ) + n = len(rows) or 1 + out_approaches.append( + { + "name": name, + "n": len(rows), + "correct_rate": sum(r["correct"] for r in rows) / n, + "mean_search_calls": sum(r["search_calls"] for r in rows) / n, + "total_input_tokens": sum(r["input_tokens"] for r in rows), + "total_output_tokens": sum(r["output_tokens"] for r in rows), + "mean_wall_s": sum(r["wall_s"] for r in rows) / n, + "rows": rows, + } + ) + return { + "skipped": False, + "provider": kind, + "model": model, + "k": k, + "approaches": out_approaches, + } + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--approaches", default=",".join(DEFAULT_APPROACHES)) + ap.add_argument("--k", type=int, default=5) + ap.add_argument("--max-calls", type=int, default=3) + ap.add_argument("--limit", type=int, default=0, help="cap queries (0 = all 150)") + args = ap.parse_args() + result = run_live( + [a.strip() for a in args.approaches.split(",") if a.strip()], + k=args.k, + max_calls=args.max_calls, + limit=args.limit or None, + ) + RESULTS.mkdir(parents=True, exist_ok=True) + (RESULTS / "live_agent.json").write_text(json.dumps(result, indent=2) + "\n", encoding="utf-8") + if result.get("skipped"): + print("live agent skipped:", result["reason"]) + return 0 + print("wrote", RESULTS / "live_agent.json") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/experiments/mcp-tool-search/requirements.txt b/experiments/mcp-tool-search/requirements.txt new file mode 100644 index 00000000..f48b76f6 --- /dev/null +++ b/experiments/mcp-tool-search/requirements.txt @@ -0,0 +1,7 @@ +numpy>=1.26 +tiktoken>=0.8 +# optional; harness degrades if missing +fastembed>=0.4.0 +# optional; live_agent.py uses these when a provider key is present +openai>=1.40 +anthropic>=0.34 diff --git a/experiments/mcp-tool-search/results.md b/experiments/mcp-tool-search/results.md new file mode 100644 index 00000000..04b29070 --- /dev/null +++ b/experiments/mcp-tool-search/results.md @@ -0,0 +1,377 @@ +# MCP tool search — reproduction results + +**This is a reproduction, not Zed's unpublished benchmark.** Zed's exact +queries, tools, labels, and 25-approach list are not public. See +[`GAPS.md`](GAPS.md). + +## Headline + +On this 150×536 public reproduction: + +| Result | Number | Note | +|---|---|---| +| Best no-model ranker | `boolean_or` / `hybrid_rrf_bm25_trigram` / `bm25_plain` | R@1 94.7% / 92.0% / 90.7%. CIs overlap. | +| Zed-like weighted BM25 | **85.3% R@1** | Worse than plain BM25 here (CI overlap). Field/verb weights are our guess, not Zed's. | +| Best overall | `hybrid_rrf_bm25_embed` and `hybrid_codedb_weighted_bm25` | **93.3% R@1**, MRR@10 0.96. Embedding needs a model; the codedb hybrid does not. | +| **Embedding vs trigram** | bge-small **92.7%** vs char-trigram **85.3%** vs codedb-trigram **60.7%** | Dense wins R@1; CIs: embed 88–97%, trigram 79–91%. Embed is ~6× slower than in-process trigram. | +| **Where codedb ranks** | `codedb_search` **91.3% R@1** (3rd among shipped-like locals) | Word/symbol/find are the wrong primitive (30–67%). | +| Scaling 536→5k | BM25 and trigram **do not fall off** | Distractors are vendor-distinct; they do not recreate StackOne's lookalike-tool failure. | +| **Agent cost (k=5)** | cheapest **373 tok/q**, 1.01 calls | `hybrid_codedb_weighted_bm25`. All-schemas dump is **38,003** tok/turn (102×). | +| Cost vs recall rank | **different** (29/30 move) | `boolean_or` is #1 recall, #19 cost. Weighted first-page hits beat rank-1. | +| Live agent | **skipped** | No LLM key. `OPENAI_API_KEY=... python live_agent.py` | + +Zed shipped weighted BM25 because it is fast, local, and good enough. On +*this* catalog that conclusion is supportable for **plain** BM25 (0.14 ms +p50, no model). Our guessed field weights did not beat plain BM25; that +is a gap in the reproduction, not a refutation of Zed. + +codedb's own multi-word search (`searchContentRanked` BM25+) already +matches plain BM25 on this task. To *win* tool search it would need +fielded BM25 / verb down-weight / namespace boosts — see GAPS.md. + +--- + +Zed posted on 20 Oct 2026 that they tested 25 approaches on 150 queries +against 536 tools and shipped weighted BM25 with a few normalization +rules ([tweet](https://x.com/zeddotdev/status/2112588120558338261)). +The tweet image is a meme (`fixtures/zed-tweet-2026-10-20.png`), not a +write-up. The Zed blog has no post. `zed-industries/zed` has no public +tool-search fixtures (only an unrelated edit-prediction BM25 retriever). + +Catalog: 536 tools (366 public MCP names + 170 expanded), 51 servers, +150 hand-labelled queries. Frozen in `data/`. + +Metrics are means over 150 queries with 2000-resample bootstrap 95% CIs. +Differences of a few points are usually noise. codedb latencies include +MCP RPC; `hybrid_codedb_*` p50 is RRF-fusion only. + + +## Agent cost (search calls + tokens, not recall@k) + +Recall@k is not the agent metric. An agent calls `search(query)`, gets +the top-k **full tool schemas** (name, description, inputSchema, as an +MCP `tools/list` / `search_tools` hit), and pages until the gold/alt +tool appears. Tokens are `tiktoken` **o200k_base** over the compact JSON +payload actually returned to the model. + +Paging model: offset slices of the saved ranking. +Call *n* returns ranks `[(n-1)k : n k]`. Tools missing from the saved +top-10 are appended in catalog order (pessimistic tail). + +**All-schemas baseline:** `38,003` o200k tokens +per turn to load all 536 tool schemas up front +(170,223 bytes). No search calls. + +### Ranking by tokens burned per successful query (k=5) + +Primary page size is 5 (FastMCP-style). Sorted by mean o200k tokens +returned until gold appears. **This ranking is not the recall@1 ranking.** +29 of 30 approaches change place. `boolean_or` wins recall@1 (94.7%) +and lands 19th on cost: its first page is fatter, and the rare miss +pages deep into the catalog tail. Methods that put gold in the first +page (even at rank 2–5) are cheaper even when they lose rank-1. + +| Cost # | Recall # | Δ | Approach | R@1 | tok/success | mean calls | success@1 | +|---:|---:|---:|---|---:|---:|---:|---:| +| 1 | 3 | +2 | hybrid_codedb_weighted_bm25 | 93.3% | 372.8 | 1.01 | 99.3% | +| 2 | 14 | +12 | hybrid_rrf_weighted_trigram | 86.0% | 376.5 | 1.01 | 99.3% | +| 3 | 13 | +10 | hybrid_rrf_weighted_hashed | 86.0% | 379.0 | 1.01 | 99.3% | +| 4 | 12 | +8 | lexical_rerank_weighted_top20 | 88.0% | 380.4 | 1.01 | 98.7% | +| 5 | 18 | +13 | hashed_ngram_embed | 84.0% | 380.9 | 1.01 | 99.3% | +| 6 | 15 | +9 | bm25_weighted | 85.3% | 382.8 | 1.02 | 98.7% | +| 7 | 17 | +10 | decision_heuristic | 85.3% | 382.8 | 1.02 | 98.7% | +| 8 | 25 | +17 | hybrid_codedb_trigram_bm25 | 70.7% | 384.9 | 1.05 | 95.3% | +| 9 | 10 | +1 | hybrid_rrf_trigram_embed | 90.0% | 388.1 | 1.05 | 99.3% | +| 10 | 4 | -6 | hybrid_rrf_bm25_embed | 93.3% | 389.6 | 1.05 | 99.3% | +| 11 | 2 | -9 | hybrid_codedb_embed | 93.3% | 393.0 | 1.05 | 99.3% | +| 12 | 5 | -7 | embed_bge_small | 92.7% | 393.7 | 1.05 | 99.3% | +| 13 | 11 | -2 | hybrid_rrf_weighted_embed | 88.7% | 397.3 | 1.05 | 99.3% | +| 14 | 19 | +5 | query_rewrite_weighted_bm25 | 83.3% | 398.8 | 1.05 | 97.3% | +| 15 | 6 | -9 | hybrid_rrf_bm25_trigram | 92.0% | 446.9 | 1.20 | 98.7% | +| 16 | 23 | +7 | tfidf | 80.0% | 449.4 | 1.19 | 98.7% | +| 17 | 22 | +5 | bm25_tfidf_hybrid | 82.0% | 450.0 | 1.19 | 98.7% | +| 18 | 8 | -10 | bm25_ident_split | 90.7% | 452.4 | 1.20 | 98.7% | +| 19 | 1 | -18 | boolean_or | 94.7% | 464.9 | 1.11 | 99.3% | +| 20 | 9 | -11 | bm25_plain | 90.7% | 465.3 | 1.23 | 98.0% | +| 21 | 16 | -5 | char_trigram | 85.3% | 503.1 | 1.32 | 98.0% | +| 22 | 7 | -15 | codedb_search | 91.3% | 529.3 | 1.37 | 96.7% | +| 23 | 20 | -3 | bm25_desc_only | 82.7% | 597.3 | 1.52 | 96.0% | +| 24 | 26 | +2 | codedb_word | 67.3% | 748.4 | 1.87 | 91.3% | +| 25 | 21 | -4 | bm25_name_only | 82.0% | 1036.0 | 2.63 | 92.7% | +| 26 | 27 | +1 | codedb_trigram_tokens | 60.7% | 1088.8 | 2.78 | 85.3% | +| 27 | 24 | -3 | boolean_and | 76.7% | 2707.6 | 6.79 | 77.3% | +| 28 | 29 | +1 | codedb_symbol | 30.7% | 3668.0 | 9.29 | 61.3% | +| 29 | 28 | -1 | codedb_find | 48.7% | 6801.5 | 17.26 | 50.7% | +| 30 | 30 | +0 | name_exact_substr | 0.0% | 14978.9 | 38.48 | 0.7% | + +Cheapest: **hybrid_codedb_weighted_bm25** at 373 tokens/query (37,630 less than the all-schemas dump, 99.0% saved). Mean search calls 1.01, first-page hit 99.3%. + +Mean tokens can exceed p95: a few deep-tail misses pull the mean up +while 95% of queries finish in one page. + +### Offline cost, k=3 + +| Approach | mean calls | p95 calls | ≤1 | ≤2 | ≤3 | mean tok | p95 tok | total tok | +|---|---:|---:|---:|---:|---:|---:|---:|---:| +| hybrid_codedb_weighted_bm25 | 1.01 | 1.0 | 98.7% | 100.0% | 100.0% | 236 | 337 | 35,445 | +| hybrid_rrf_weighted_trigram | 1.02 | 1.0 | 98.7% | 99.3% | 100.0% | 237 | 333 | 35,500 | +| hybrid_rrf_weighted_hashed | 1.01 | 1.0 | 98.7% | 100.0% | 100.0% | 239 | 325 | 35,885 | +| lexical_rerank_weighted_top20 | 1.03 | 1.0 | 97.3% | 100.0% | 100.0% | 241 | 373 | 36,149 | +| hashed_ngram_embed | 1.03 | 1.0 | 96.7% | 100.0% | 100.0% | 241 | 340 | 36,135 | +| bm25_weighted | 1.04 | 1.0 | 97.3% | 99.3% | 99.3% | 243 | 350 | 36,435 | +| decision_heuristic | 1.04 | 1.0 | 97.3% | 99.3% | 99.3% | 243 | 350 | 36,435 | +| hybrid_codedb_trigram_bm25 | 1.16 | 2.0 | 90.0% | 95.3% | 98.7% | 268 | 577 | 40,240 | +| hybrid_rrf_trigram_embed | 1.10 | 1.0 | 97.3% | 99.3% | 99.3% | 254 | 329 | 38,082 | +| hybrid_rrf_bm25_embed | 1.09 | 1.0 | 98.7% | 99.3% | 99.3% | 251 | 322 | 37,621 | +| hybrid_codedb_embed | 1.09 | 1.0 | 98.7% | 99.3% | 99.3% | 253 | 346 | 37,948 | +| embed_bge_small | 1.08 | 1.0 | 99.3% | 99.3% | 99.3% | 253 | 316 | 38,022 | +| hybrid_rrf_weighted_embed | 1.08 | 1.0 | 99.3% | 99.3% | 99.3% | 255 | 318 | 38,304 | +| query_rewrite_weighted_bm25 | 1.11 | 1.0 | 95.3% | 98.7% | 99.3% | 261 | 389 | 39,171 | +| hybrid_rrf_bm25_trigram | 1.34 | 1.0 | 98.0% | 98.7% | 99.3% | 314 | 333 | 47,157 | +| tfidf | 1.35 | 1.0 | 97.3% | 98.7% | 99.3% | 319 | 315 | 47,779 | +| bm25_tfidf_hybrid | 1.35 | 1.0 | 97.3% | 98.7% | 99.3% | 319 | 337 | 47,789 | +| bm25_ident_split | 1.34 | 1.0 | 98.0% | 98.7% | 99.3% | 316 | 333 | 47,382 | +| boolean_or | 1.20 | 1.0 | 98.0% | 99.3% | 99.3% | 309 | 393 | 46,369 | +| bm25_plain | 1.40 | 1.0 | 97.3% | 98.0% | 98.7% | 332 | 324 | 49,733 | +| char_trigram | 1.54 | 1.0 | 96.7% | 98.7% | 98.7% | 368 | 330 | 55,230 | +| codedb_search | 1.63 | 1.0 | 96.0% | 97.3% | 97.3% | 400 | 353 | 59,945 | +| bm25_desc_only | 1.90 | 2.0 | 92.7% | 96.7% | 97.3% | 467 | 441 | 70,036 | +| codedb_word | 2.51 | 3.0 | 87.3% | 93.3% | 96.7% | 622 | 668 | 93,371 | +| bm25_name_only | 3.77 | 3.5 | 90.0% | 94.0% | 94.7% | 924 | 976 | 138,610 | +| codedb_trigram_tokens | 4.06 | 30.1 | 80.7% | 87.3% | 90.7% | 1007 | 8227 | 151,069 | +| boolean_and | 10.73 | 59.7 | 77.3% | 77.3% | 77.3% | 2656 | 14505 | 398,471 | +| codedb_symbol | 15.01 | 85.6 | 53.3% | 62.0% | 70.0% | 3683 | 20396 | 552,471 | +| codedb_find | 28.31 | 101.2 | 50.7% | 50.7% | 51.3% | 6935 | 24073 | 1,040,178 | +| name_exact_substr | 63.75 | 116.5 | 0.7% | 0.7% | 1.3% | 15419 | 27302 | 2,312,788 | + +### Offline cost, k=5 + +| Approach | mean calls | p95 calls | ≤1 | ≤2 | ≤3 | mean tok | p95 tok | total tok | +|---|---:|---:|---:|---:|---:|---:|---:|---:| +| hybrid_codedb_weighted_bm25 | 1.01 | 1.0 | 99.3% | 100.0% | 100.0% | 373 | 523 | 55,917 | +| hybrid_rrf_weighted_trigram | 1.01 | 1.0 | 99.3% | 100.0% | 100.0% | 377 | 517 | 56,478 | +| hybrid_rrf_weighted_hashed | 1.01 | 1.0 | 99.3% | 100.0% | 100.0% | 379 | 511 | 56,851 | +| lexical_rerank_weighted_top20 | 1.01 | 1.0 | 98.7% | 100.0% | 100.0% | 380 | 511 | 57,062 | +| hashed_ngram_embed | 1.01 | 1.0 | 99.3% | 100.0% | 100.0% | 381 | 509 | 57,132 | +| bm25_weighted | 1.02 | 1.0 | 98.7% | 99.3% | 100.0% | 383 | 521 | 57,427 | +| decision_heuristic | 1.02 | 1.0 | 98.7% | 99.3% | 100.0% | 383 | 521 | 57,427 | +| hybrid_codedb_trigram_bm25 | 1.05 | 1.0 | 95.3% | 99.3% | 100.0% | 385 | 587 | 57,741 | +| hybrid_rrf_trigram_embed | 1.05 | 1.0 | 99.3% | 99.3% | 99.3% | 388 | 489 | 58,209 | +| hybrid_rrf_bm25_embed | 1.05 | 1.0 | 99.3% | 99.3% | 99.3% | 390 | 536 | 58,441 | +| hybrid_codedb_embed | 1.05 | 1.0 | 99.3% | 99.3% | 99.3% | 393 | 538 | 58,957 | +| embed_bge_small | 1.05 | 1.0 | 99.3% | 99.3% | 99.3% | 394 | 507 | 59,055 | +| hybrid_rrf_weighted_embed | 1.05 | 1.0 | 99.3% | 99.3% | 99.3% | 397 | 492 | 59,589 | +| query_rewrite_weighted_bm25 | 1.05 | 1.0 | 97.3% | 99.3% | 99.3% | 399 | 566 | 59,822 | +| hybrid_rrf_bm25_trigram | 1.20 | 1.0 | 98.7% | 99.3% | 99.3% | 447 | 518 | 67,039 | +| tfidf | 1.19 | 1.0 | 98.7% | 99.3% | 99.3% | 449 | 471 | 67,404 | +| bm25_tfidf_hybrid | 1.19 | 1.0 | 98.7% | 99.3% | 99.3% | 450 | 486 | 67,506 | +| bm25_ident_split | 1.20 | 1.0 | 98.7% | 99.3% | 99.3% | 452 | 515 | 67,853 | +| boolean_or | 1.11 | 1.0 | 99.3% | 99.3% | 99.3% | 465 | 601 | 69,734 | +| bm25_plain | 1.23 | 1.0 | 98.0% | 98.7% | 98.7% | 465 | 478 | 69,795 | +| char_trigram | 1.32 | 1.0 | 98.0% | 98.7% | 98.7% | 503 | 501 | 75,462 | +| codedb_search | 1.37 | 1.0 | 96.7% | 97.3% | 97.3% | 529 | 573 | 79,401 | +| bm25_desc_only | 1.52 | 1.0 | 96.0% | 97.3% | 97.3% | 597 | 598 | 89,594 | +| codedb_word | 1.87 | 2.0 | 91.3% | 96.7% | 96.7% | 748 | 839 | 112,253 | +| bm25_name_only | 2.63 | 2.5 | 92.7% | 94.7% | 95.3% | 1036 | 1115 | 155,394 | +| codedb_trigram_tokens | 2.78 | 18.5 | 85.3% | 92.7% | 92.7% | 1089 | 8117 | 163,319 | +| boolean_and | 6.79 | 35.6 | 77.3% | 77.3% | 77.3% | 2708 | 13944 | 406,143 | +| codedb_symbol | 9.29 | 52.1 | 61.3% | 72.7% | 73.3% | 3668 | 20001 | 550,193 | +| codedb_find | 17.26 | 61.1 | 50.7% | 52.0% | 52.0% | 6801 | 23343 | 1,020,221 | +| name_exact_substr | 38.48 | 70.0 | 0.7% | 1.3% | 2.0% | 14979 | 26505 | 2,246,838 | + +### Offline cost, k=10 + +| Approach | mean calls | p95 calls | ≤1 | ≤2 | ≤3 | mean tok | p95 tok | total tok | +|---|---:|---:|---:|---:|---:|---:|---:|---:| +| hybrid_codedb_weighted_bm25 | 1.00 | 1.0 | 100.0% | 100.0% | 100.0% | 705 | 946 | 105,777 | +| hybrid_rrf_weighted_trigram | 1.00 | 1.0 | 100.0% | 100.0% | 100.0% | 741 | 1001 | 111,088 | +| hybrid_rrf_weighted_hashed | 1.00 | 1.0 | 100.0% | 100.0% | 100.0% | 755 | 1008 | 113,236 | +| lexical_rerank_weighted_top20 | 1.00 | 1.0 | 100.0% | 100.0% | 100.0% | 731 | 960 | 109,649 | +| hashed_ngram_embed | 1.00 | 1.0 | 100.0% | 100.0% | 100.0% | 756 | 942 | 113,413 | +| bm25_weighted | 1.01 | 1.0 | 99.3% | 100.0% | 100.0% | 733 | 953 | 110,019 | +| decision_heuristic | 1.01 | 1.0 | 99.3% | 100.0% | 100.0% | 733 | 953 | 110,019 | +| hybrid_codedb_trigram_bm25 | 1.01 | 1.0 | 99.3% | 100.0% | 100.0% | 716 | 947 | 107,331 | +| hybrid_rrf_trigram_embed | 1.02 | 1.0 | 99.3% | 99.3% | 99.3% | 745 | 970 | 111,823 | +| hybrid_rrf_bm25_embed | 1.02 | 1.0 | 99.3% | 99.3% | 99.3% | 737 | 996 | 110,507 | +| hybrid_codedb_embed | 1.02 | 1.0 | 99.3% | 99.3% | 99.3% | 729 | 979 | 109,423 | +| embed_bge_small | 1.02 | 1.0 | 99.3% | 99.3% | 99.3% | 751 | 979 | 112,672 | +| hybrid_rrf_weighted_embed | 1.02 | 1.0 | 99.3% | 99.3% | 99.3% | 759 | 1008 | 113,793 | +| query_rewrite_weighted_bm25 | 1.01 | 1.0 | 99.3% | 99.3% | 100.0% | 750 | 988 | 112,464 | +| hybrid_rrf_bm25_trigram | 1.09 | 1.0 | 99.3% | 99.3% | 99.3% | 784 | 985 | 117,622 | +| tfidf | 1.09 | 1.0 | 99.3% | 99.3% | 99.3% | 813 | 954 | 121,949 | +| bm25_tfidf_hybrid | 1.09 | 1.0 | 99.3% | 99.3% | 99.3% | 812 | 962 | 121,875 | +| bm25_ident_split | 1.09 | 1.0 | 99.3% | 99.3% | 99.3% | 795 | 1001 | 119,308 | +| boolean_or | 1.05 | 1.0 | 99.3% | 99.3% | 99.3% | 876 | 1188 | 131,331 | +| bm25_plain | 1.11 | 1.0 | 98.7% | 98.7% | 99.3% | 804 | 989 | 120,565 | +| char_trigram | 1.15 | 1.0 | 98.7% | 98.7% | 98.7% | 851 | 994 | 127,618 | +| codedb_search | 1.17 | 1.0 | 97.3% | 97.3% | 98.7% | 844 | 1107 | 126,604 | +| bm25_desc_only | 1.24 | 1.0 | 97.3% | 97.3% | 97.3% | 932 | 1045 | 139,870 | +| codedb_word | 1.40 | 1.0 | 96.7% | 96.7% | 97.3% | 1084 | 1101 | 162,574 | +| bm25_name_only | 1.79 | 1.5 | 94.7% | 95.3% | 95.3% | 1366 | 1302 | 204,854 | +| codedb_trigram_tokens | 1.84 | 9.5 | 92.7% | 92.7% | 92.7% | 1371 | 8033 | 205,675 | +| boolean_and | 3.84 | 18.1 | 77.3% | 78.0% | 78.0% | 3004 | 13725 | 450,537 | +| codedb_symbol | 5.03 | 26.5 | 72.7% | 73.3% | 74.0% | 3851 | 19869 | 577,646 | +| codedb_find | 9.01 | 30.5 | 52.0% | 52.7% | 53.3% | 6907 | 22689 | 1,036,085 | +| name_exact_substr | 19.49 | 35.0 | 1.3% | 4.0% | 7.3% | 14759 | 25789 | 2,213,858 | + +### Live agent loop + +Skipped in this environment: no LLM API key +(`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `AZURE_OPENAI_API_KEY`, +`GOOGLE_API_KEY`, `GEMINI_API_KEY`, `COHERE_API_KEY`, `TOGETHER_API_KEY`, +`GROQ_API_KEY`, `MISTRAL_API_KEY`, `OPENROUTER_API_KEY`). + +Harness is ready: + +```bash +OPENAI_API_KEY=... python live_agent.py +# or +ANTHROPIC_API_KEY=... python live_agent.py +``` + +Default approaches: `bm25_plain`, `bm25_weighted`, `embed_bge_small`, +`hybrid_rrf_bm25_embed`, `codedb_search`, `char_trigram`. Records real +search-call counts, API input/output tokens, whether the named tool is +gold/alt, and wall time. Writes `results/live_agent.json`. + +Raw cost outputs: `results/cost.json`, `results/cost.csv`, +`results/cost_per_query.json`, `results/cost_per_query.csv`. +Per-approach JSON/CSV now include `hit_rank` and `k{3,5,10}_*` fields. + + +## Headline table (n=536 tools, 150 queries) + +| Approach | R@1 | R@3 | R@5 | MRR@10 | nDCG@10 | p50 ms | p95 ms | index s | model/API | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---| +| name_exact_substr | 0.0% | 0.0% | 0.0% | 0.000 | 0.000 | 0.31 | 0.36 | 0.000 | local | +| boolean_and | 76.7% | 77.3% | 77.3% | 0.770 | 0.767 | 0.24 | 0.28 | 0.000 | local | +| boolean_or | 94.7% | 98.0% | 99.3% | 0.964 | 0.959 | 0.30 | 0.37 | 0.000 | local | +| bm25_plain | 90.7% | 97.3% | 98.0% | 0.940 | 0.948 | 0.14 | 0.23 | 0.001 | local | +| bm25_ident_split | 90.7% | 98.0% | 98.7% | 0.942 | 0.952 | 0.14 | 0.26 | 0.001 | local | +| bm25_name_only | 82.0% | 90.0% | 92.7% | 0.870 | 0.882 | 0.10 | 0.14 | 0.001 | local | +| bm25_desc_only | 82.7% | 92.7% | 96.0% | 0.881 | 0.900 | 0.14 | 0.25 | 0.001 | local | +| bm25_weighted | 85.3% | 97.3% | 98.7% | 0.911 | 0.923 | 0.40 | 0.65 | 0.003 | local | +| tfidf | 80.0% | 97.3% | 98.7% | 0.880 | 0.902 | 0.17 | 0.28 | 0.001 | local | +| bm25_tfidf_hybrid | 82.0% | 97.3% | 98.7% | 0.892 | 0.913 | 0.38 | 0.65 | 0.002 | local | +| char_trigram | 85.3% | 96.7% | 98.0% | 0.912 | 0.926 | 0.35 | 0.47 | 0.005 | local | +| hashed_ngram_embed | 84.0% | 96.7% | 99.3% | 0.909 | 0.921 | 4.20 | 4.38 | 0.021 | local | +| hybrid_rrf_bm25_trigram | 92.0% | 98.0% | 98.7% | 0.949 | 0.956 | 0.55 | 0.79 | 0.006 | local | +| hybrid_rrf_weighted_trigram | 86.0% | 98.7% | 99.3% | 0.922 | 0.936 | 0.84 | 1.21 | 0.008 | local | +| hybrid_rrf_weighted_hashed | 86.0% | 98.7% | 99.3% | 0.924 | 0.936 | 4.69 | 4.95 | 0.024 | local | +| decision_heuristic | 85.3% | 97.3% | 98.7% | 0.911 | 0.923 | 0.81 | 1.07 | 0.003 | local | +| query_rewrite_weighted_bm25 | 83.3% | 95.3% | 97.3% | 0.895 | 0.913 | 0.40 | 0.56 | 0.003 | local | +| lexical_rerank_weighted_top20 | 88.0% | 97.3% | 98.7% | 0.929 | 0.940 | 0.58 | 0.82 | 0.003 | local | +| embed_bge_small | 92.7% | 99.3% | 99.3% | 0.957 | 0.960 | 2.14 | 2.64 | 1.960 | model | +| hybrid_rrf_bm25_embed | 93.3% | 98.7% | 99.3% | 0.959 | 0.967 | 2.39 | 3.31 | 1.961 | model | +| hybrid_rrf_weighted_embed | 88.7% | 99.3% | 99.3% | 0.938 | 0.950 | 2.68 | 3.20 | 1.963 | model | +| hybrid_rrf_trigram_embed | 90.0% | 97.3% | 99.3% | 0.940 | 0.950 | 2.59 | 3.03 | 1.965 | model | +| codedb_search | 91.3% | 96.0% | 96.7% | 0.938 | 0.943 | 10.28 | 10.41 | 0.442 | local | +| codedb_trigram_tokens | 60.7% | 80.7% | 85.3% | 0.718 | 0.759 | 30.78 | 41.26 | 0.442 | local | +| codedb_word | 67.3% | 87.3% | 91.3% | 0.783 | 0.819 | 30.81 | 32.20 | 0.442 | local | +| codedb_symbol | 30.7% | 53.3% | 61.3% | 0.432 | 0.497 | 20.62 | 20.85 | 0.442 | local | +| codedb_find | 48.7% | 50.7% | 50.7% | 0.495 | 0.494 | 10.25 | 10.37 | 0.442 | local | +| hybrid_codedb_weighted_bm25 | 93.3% | 98.7% | 99.3% | 0.960 | 0.965 | 0.01 | 0.02 | 0.445 | local | +| hybrid_codedb_trigram_bm25 | 70.7% | 90.0% | 95.3% | 0.809 | 0.845 | 0.01 | 0.01 | 0.445 | local | +| hybrid_codedb_embed | 93.3% | 98.7% | 99.3% | 0.959 | 0.966 | 0.01 | 0.01 | 2.401 | model | + +### Bootstrap 95% CIs for Recall@1 / MRR@10 + +| Approach | Recall@1 | MRR@10 | +|---|---|---| +| name_exact_substr | 0.0% [0.0%, 0.0%] | 0.000 [0.000, 0.000] | +| boolean_and | 76.7% [70.0%, 83.3%] | 0.770 [0.703, 0.840] | +| boolean_or | 94.7% [90.7%, 98.0%] | 0.964 [0.937, 0.987] | +| bm25_plain | 90.7% [86.0%, 95.3%] | 0.940 [0.907, 0.971] | +| bm25_ident_split | 90.7% [86.0%, 95.3%] | 0.942 [0.910, 0.971] | +| bm25_name_only | 82.0% [75.3%, 88.0%] | 0.870 [0.822, 0.916] | +| bm25_desc_only | 82.7% [76.0%, 88.7%] | 0.881 [0.836, 0.922] | +| bm25_weighted | 85.3% [79.3%, 90.7%] | 0.911 [0.874, 0.946] | +| tfidf | 80.0% [73.3%, 86.0%] | 0.880 [0.840, 0.917] | +| bm25_tfidf_hybrid | 82.0% [76.0%, 88.0%] | 0.892 [0.854, 0.927] | +| char_trigram | 85.3% [79.3%, 90.7%] | 0.912 [0.877, 0.946] | +| hashed_ngram_embed | 84.0% [78.0%, 90.0%] | 0.909 [0.874, 0.942] | +| hybrid_rrf_bm25_trigram | 92.0% [87.3%, 96.0%] | 0.949 [0.919, 0.977] | +| hybrid_rrf_weighted_trigram | 86.0% [80.7%, 91.3%] | 0.922 [0.889, 0.953] | +| hybrid_rrf_weighted_hashed | 86.0% [80.0%, 91.3%] | 0.924 [0.893, 0.953] | +| decision_heuristic | 85.3% [79.3%, 90.7%] | 0.911 [0.874, 0.946] | +| query_rewrite_weighted_bm25 | 83.3% [77.3%, 89.3%] | 0.895 [0.855, 0.933] | +| lexical_rerank_weighted_top20 | 88.0% [82.7%, 93.3%] | 0.929 [0.897, 0.960] | +| embed_bge_small | 92.7% [88.0%, 96.7%] | 0.957 [0.929, 0.980] | +| hybrid_rrf_bm25_embed | 93.3% [89.3%, 97.3%] | 0.959 [0.933, 0.983] | +| hybrid_rrf_weighted_embed | 88.7% [83.3%, 93.3%] | 0.938 [0.908, 0.966] | +| hybrid_rrf_trigram_embed | 90.0% [85.3%, 94.7%] | 0.940 [0.908, 0.968] | +| codedb_search | 91.3% [86.7%, 96.0%] | 0.938 [0.901, 0.970] | +| codedb_trigram_tokens | 60.7% [52.7%, 68.7%] | 0.718 [0.658, 0.778] | +| codedb_word | 67.3% [59.3%, 74.7%] | 0.783 [0.728, 0.835] | +| codedb_symbol | 30.7% [24.0%, 38.0%] | 0.432 [0.370, 0.497] | +| codedb_find | 48.7% [40.0%, 56.7%] | 0.495 [0.413, 0.577] | +| hybrid_codedb_weighted_bm25 | 93.3% [89.3%, 97.3%] | 0.960 [0.935, 0.983] | +| hybrid_codedb_trigram_bm25 | 70.7% [63.3%, 78.0%] | 0.809 [0.761, 0.857] | +| hybrid_codedb_embed | 93.3% [89.3%, 97.3%] | 0.959 [0.932, 0.983] | + +## Embedding vs trigram (same queries) + +Head-to-head on the 150-query / 536-tool set. `char_trigram` is an +in-process character-trigram cosine (codedb-like). `codedb_trigram_tokens` +is codedb's real trigram path (one search per token). Dense embedding is +`embed_bge_small` when fastembed is available, else hashed n-grams. + +| Approach | R@1 | R@3 | MRR@10 | nDCG@10 | p50 ms | +|---|---:|---:|---:|---:|---:| +| char_trigram | 85.3% | 96.7% | 0.912 | 0.926 | 0.35 | +| codedb_trigram_tokens | 60.7% | 80.7% | 0.718 | 0.759 | 30.78 | +| embed_bge_small | 92.7% | 99.3% | 0.957 | 0.960 | 2.14 | +| hashed_ngram_embed | 84.0% | 96.7% | 0.909 | 0.921 | 4.20 | +| hybrid_rrf_trigram_embed | 90.0% | 97.3% | 0.940 | 0.950 | 2.59 | +| hybrid_rrf_bm25_trigram | 92.0% | 98.0% | 0.949 | 0.956 | 0.55 | +| hybrid_rrf_weighted_embed | 88.7% | 99.3% | 0.938 | 0.950 | 2.68 | + +## Where codedb ranks + +codedb is a **code-intelligence** engine. Here it is asked to retrieve +MCP tools from a synthetic catalog (one Python file per tool). That is +off-label: no field weights, no verb down-weighting, ranking is for source +files (centrality / call-graph) rather than tool metadata. + +Best codedb-family method by MRR@10: **hybrid_codedb_weighted_bm25** (0.960). + +## Scaling (536 → ~2k → ~5k tools) + +Gold tools stay fixed; distractors reuse the same create/get/list verbs. + +| n tools | Approach | R@1 | R@5 | MRR@10 | p50 ms | +|---:|---|---:|---:|---:|---:| +| 536 | bm25_plain | 90.7% | 98.0% | 0.940 | 0.14 | +| 536 | bm25_weighted | 85.3% | 98.7% | 0.911 | 0.40 | +| 536 | tfidf | 80.0% | 98.7% | 0.880 | 0.16 | +| 536 | char_trigram | 85.3% | 98.0% | 0.912 | 0.35 | +| 536 | hybrid_rrf_weighted_trigram | 86.0% | 99.3% | 0.922 | 0.84 | +| 2000 | bm25_plain | 92.0% | 98.0% | 0.945 | 0.48 | +| 2000 | bm25_weighted | 84.0% | 99.3% | 0.899 | 1.43 | +| 2000 | tfidf | 79.3% | 96.7% | 0.868 | 0.61 | +| 2000 | char_trigram | 85.3% | 97.3% | 0.912 | 1.33 | +| 2000 | hybrid_rrf_weighted_trigram | 86.7% | 99.3% | 0.924 | 2.74 | +| 5000 | bm25_plain | 92.0% | 98.0% | 0.944 | 1.28 | +| 5000 | bm25_weighted | 83.3% | 97.3% | 0.887 | 3.42 | +| 5000 | tfidf | 78.0% | 96.7% | 0.854 | 1.51 | +| 5000 | char_trigram | 85.3% | 97.3% | 0.912 | 3.39 | +| 5000 | hybrid_rrf_weighted_trigram | 86.7% | 98.7% | 0.922 | 6.68 | + +## Approaches we could not run + +- Zed's exact 25-approach list, queries, tools, labels, field weights, and BM25 parameters are not public. +- LLM routing and trained decision models (LightGBM etc.) were not run: no API key / no labelled train split from Zed. +- Live agent loop was not run: no LLM API key. Harness is `python live_agent.py`. +- Cross-encoder rerankers and API embedding models were not run unless those packages/keys are present. +- A second local dense model (gte/nomic) was not downloaded to keep the one-command path small. + + +Full gap list: [`GAPS.md`](GAPS.md). + +## Raw outputs + +Per-approach JSON/CSV lives in `results/` (now includes `hit_rank` and +`k{3,5,10}_*` cost fields). Combined recall table: `results/summary.json`. +Cost tables: `results/cost.json`, `results/cost.csv`, +`results/cost_per_query.json`. Live skip/run: `results/live_agent.json`. + diff --git a/experiments/mcp-tool-search/results/bm25_desc_only.csv b/experiments/mcp-tool-search/results/bm25_desc_only.csv new file mode 100644 index 00000000..5fe2cb9f --- /dev/null +++ b/experiments/mcp-tool-search/results/bm25_desc_only.csv @@ -0,0 +1,151 @@ +id,query,kind,hit1,recall@1,recall@3,recall@5,mrr@10,ndcg@10,hit_rank,k3_calls,k3_tokens,k3_success1,k5_calls,k5_tokens,k5_success1,k10_calls,k10_tokens,k10_success1,ranking +q001,Create a GitHub issue,verb_product,gitlab.gitlab_create_issue,0.0,1.0,1.0,0.5000,0.6697,2,1,380,1.0,1,584,1.0,1,1011,1.0,gitlab.gitlab_create_issue github.create_issue github.issue_write github.issue_read github.add_issue_comment github.update_issue_comment github.create_gist bitbucket.bitbucket_create_issue github.create_repository gitlab.gitlab_get_issue +q002,Create a Jira ticket,verb_product,jira.jira_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,281,1.0,1,481,1.0,1,966,1.0,jira.jira_create_issue zendesk.zendesk_create_ticket zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket jira.jira_create_sprint jira.create_board jira.create_sprint jira.create_filter jira.create_webhook jira.create_attachment +q003,Send a Slack message,verb_product,discord.discord_send_message,0.0,1.0,1.0,0.5000,0.6309,2,1,247,1.0,1,427,1.0,1,771,1.0,discord.discord_send_message slack.slack_post_message slack.slack_add_reaction gmail.gmail_send_email twilio.twilio_send_sms gmail.gmail_get_email gmail.gmail_trash_email discord.discord_add_reaction gmail.gmail_modify_email slack.slack_join_channel +q004,List open pull requests,resource,bitbucket.bitbucket_list_prs,0.0,1.0,1.0,0.5000,0.6199,2,1,237,1.0,1,423,1.0,1,825,1.0,bitbucket.bitbucket_list_prs github.search_pull_requests github.list_pull_requests github.create_pull_request playwright.browser_tab_list gitlab.gitlab_list_merge_requests playwright.browser_tab_new bitbucket.bitbucket_create_pr github.merge_pull_request memory.open_nodes +q005,Open a pull request on GitHub,verb_product,github.create_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,379,1.0,1,612,1.0,1,1163,1.0,github.create_pull_request github.request_copilot_review github.merge_pull_request github.pull_request_read github.pull_request_review_write bitbucket.bitbucket_create_pr github.add_comment_to_pending_review github.update_pull_request github.search_pull_requests github.list_repository_collaborators +q006,Run a SQL query against postgres,verb_product,mysql.mysql_query,0.0,1.0,1.0,0.3333,0.5000,3,1,180,1.0,1,295,1.0,1,671,1.0,mysql.mysql_query sqlite.sqlite_query postgres.query salesforce.salesforce_query postgres.explain_query notion.notion_query_database prometheus.prom_query prometheus.prom_query_range mysql.mysql_list_tables docker.docker_run +q007,Read a local file from disk,paraphrase,filesystem.read_text_file,1.0,1.0,1.0,1.0000,0.8597,1,1,223,1.0,1,425,1.0,1,832,1.0,filesystem.read_text_file filesystem.read_file filesystem.read_media_file figma.figma_get_images github.delete_file gitlab.gitlab_get_file figma.figma_get_file_nodes github.get_file_contents memory.read_graph github.issue_read +q008,What files are in this folder,paraphrase,google_drive.gdrive_list_folder,0.0,0.0,0.0,0.1667,0.4244,6,2,413,0.0,2,727,0.0,1,705,1.0,google_drive.gdrive_list_folder slack.list_files google_drive.gdrive_search filesystem.move_file filesystem.search_files filesystem.list_directory github.push_files filesystem.list_directory_with_sizes filesystem.directory_tree elasticsearch.es_index_doc +q009,Remember this fact in the knowledge graph,paraphrase,memory.read_graph,0.0,1.0,1.0,0.5000,0.5339,2,1,154,1.0,1,261,1.0,1,537,1.0,memory.read_graph memory.create_entities memory.delete_relations memory.create_relations memory.add_observations memory.open_nodes memory.search_nodes memory.delete_entities time.get_current_time slack.slack_list_channels +q010,Search the web with Brave,verb_product,brave_search.brave_web_search,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,430,1.0,1,723,1.0,brave_search.brave_web_search jira.jira_search_issues brave_search.brave_local_search github.update_pull_request_branch puppeteer.puppeteer_screenshot puppeteer.puppeteer_navigate puppeteer.puppeteer_click puppeteer.puppeteer_fill puppeteer.puppeteer_evaluate memory.search_nodes +q011,Fetch the contents of a URL,verb_product,fetch.fetch,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,346,1.0,1,662,1.0,fetch.fetch filesystem.read_file playwright.browser_navigate git.git_add slack.slack_set_channel_topic docker.docker_logs puppeteer.puppeteer_navigate github.get_repository_tree git.git_status playwright.browser_snapshot +q012,Show git status,identifier,git.git_status,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,299,1.0,1,675,1.0,git.git_status git.git_show git.git_diff_unstaged git.git_diff_staged git.git_log git.git_diff sentry.sentry_update_issue jira.jira_transition_issue github.list_tags github.get_tag +q013,Commit staged changes,paraphrase,git.git_reset,0.0,1.0,1.0,0.5000,0.6309,2,1,193,1.0,1,305,1.0,1,663,1.0,git.git_reset git.git_commit git.git_log git.git_show git.git_diff_staged github.get_commit github.push_files +q014,Create a Linear issue,verb_product,linear.linear_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,357,1.0,1,498,1.0,1,956,1.0,linear.linear_create_issue linear.linear_update_issue bitbucket.bitbucket_create_issue linear.linear_create_comment linear.linear_assign_issue linear.linear_create_project gitlab.gitlab_create_issue linear.create_cycle linear.create_label linear.create_user +q015,Make a task in my project tracker Linear,paraphrase,linear.linear_create_project,0.0,0.0,0.0,0.0000,0.0000,146,49,12370,0.0,30,12110,0.0,15,11750,0.0,linear.linear_create_project mysql.mysql_list_tables supabase.supabase_list_tables vercel.vercel_create_project mysql.mysql_query asana.asana_create_task asana.asana_get_task asana.asana_update_task clickup.clickup_create_task clickup.clickup_get_task +q016,Post a comment on a GitHub issue,verb_product,github.update_issue_comment,0.0,1.0,1.0,0.3333,0.5000,3,1,307,1.0,1,482,1.0,1,887,1.0,github.update_issue_comment figma.figma_post_comment github.add_issue_comment trello.trello_add_comment github.assign_copilot_to_issue linear.linear_create_comment jira.jira_add_comment github.list_repository_collaborators github.get_discussion_comments github.issue_read +q017,Merge the pull request,ambiguous,github.merge_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,321,1.0,1,524,1.0,1,1103,1.0,github.merge_pull_request gitlab.gitlab_merge_merge_request gitlab.gitlab_create_merge_request gitlab.gitlab_get_merge_request bitbucket.bitbucket_create_pr github.update_pull_request_branch github.request_copilot_review github.create_pull_request github.pull_request_read github.pull_request_review_write +q018,Trigger a GitHub Actions workflow,verb_product,github.actions_run_trigger,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,458,1.0,1,831,1.0,github.actions_run_trigger github.actions_get github.actions_list github.actions_get_job_logs circleci.circleci_trigger_pipeline github.delete_repository github.fork_repository github.star_repository github.unstar_repository github.search_issues +q019,Get Dependabot alerts,resource,github.list_dependabot_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,287,1.0,1,420,1.0,1,739,1.0,github.list_dependabot_alerts github.get_dependabot_alert github.list_code_scanning_alerts github.list_secret_scanning_alerts notion.notion_get_block kubernetes.k8s_get_pod shopify.shopify_get_product shopify.shopify_get_order sentry.sentry_get_issue datadog.datadog_get_monitor +q020,List Sentry issues,verb_product,sentry.sentry_list_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,395,1.0,1,796,1.0,sentry.sentry_list_issues sentry.sentry_list_projects gitlab.gitlab_list_issues sentry.sentry_list_events linear.linear_list_issues sentry.sentry_get_issue github.list_issues github.search_issues sentry.sentry_update_issue linear.linear_search_issues +q021,Create a Stripe customer,verb_product,stripe.stripe_create_customer,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,334,1.0,1,669,1.0,stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_create_invoice stripe.stripe_create_product stripe.stripe_create_price stripe.stripe_create_refund stripe.stripe_create_payment_intent stripe.stripe_list_customers stripe.stripe_list_invoices stripe.stripe_list_subscriptions +q022,Refund a Stripe payment,paraphrase,stripe.stripe_create_refund,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,352,1.0,1,662,1.0,stripe.stripe_create_refund stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_list_customers stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_create_product stripe.stripe_create_price +q023,Send an email with Gmail,verb_product,gmail.gmail_send_email,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,395,1.0,1,707,1.0,gmail.gmail_send_email twilio.twilio_send_sms puppeteer.puppeteer_click puppeteer.puppeteer_fill gmail.gmail_search_emails gmail.gmail_create_draft gmail.gmail_list_labels discord.discord_send_message puppeteer.puppeteer_screenshot gmail.gmail_get_email +q024,Search my Gmail inbox,verb_product,gmail.gmail_search_emails,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,305,1.0,1,666,1.0,gmail.gmail_search_emails mysql.mysql_list_tables gmail.gmail_create_draft gmail.gmail_list_labels mysql.mysql_query gmail.gmail_send_email gmail.gmail_get_email gmail.gmail_trash_email brave_search.brave_web_search slack.slack_search_messages +q025,Upload a file to Google Drive,paraphrase,slack.slack_upload_file,0.0,0.0,1.0,0.2500,0.4307,4,2,555,0.0,1,446,1.0,1,747,1.0,slack.slack_upload_file google_drive.gdrive_update_file google_drive.gdrive_share_file google_drive.gdrive_create_file google_drive.gdrive_get_file aws_s3.s3_put_object google_drive.gdrive_search google_drive.gdrive_list_folder google_calendar.gcal_list_calendars git.git_add +q026,Create a calendar event,resource,google_calendar.gcal_create_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,461,1.0,1,813,1.0,google_calendar.gcal_create_event google_calendar.gcal_update_event google_calendar.gcal_delete_event google_calendar.gcal_list_events slack.slack_create_channel linear.linear_create_project notion.notion_create_database stripe.stripe_create_customer stripe.stripe_create_invoice stripe.stripe_create_product +q027,List Kubernetes pods,verb_product,kubernetes.k8s_list_pods,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,314,1.0,1,741,1.0,kubernetes.k8s_list_pods kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_get_pod kubernetes.k8s_apply kubernetes.k8s_delete kubernetes.k8s_scale_deployment kubernetes.k8s_pod_logs trello.trello_list_cards +q028,Get logs from a k8s pod,verb_product,kubernetes.k8s_pod_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,262,1.0,1,423,1.0,1,822,1.0,kubernetes.k8s_pod_logs kubernetes.k8s_get_pod docker.docker_logs git.git_log datadog.datadog_list_logs elasticsearch.es_get_doc aws_s3.s3_get_object github.actions_get_job_logs slack.slack_get_channel_history gitlab.gitlab_get_file +q029,Run a docker container,verb_product,docker.docker_run,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,345,1.0,1,594,1.0,docker.docker_run docker.docker_exec docker.docker_stop docker.docker_rm docker.docker_logs docker.docker_images docker.docker_build docker.docker_ps terraform.tf_plan terraform.tf_apply +q030,Show docker container logs,verb_product,docker.docker_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,343,1.0,1,665,1.0,docker.docker_logs git.git_log docker.docker_run docker.docker_stop docker.docker_rm docker.docker_exec datadog.datadog_list_logs git.git_show docker.docker_images docker.docker_build +q031,Navigate the browser to a URL,ambiguous,playwright.browser_navigate,1.0,1.0,1.0,1.0000,1.0000,1,1,179,1.0,1,293,1.0,1,568,1.0,playwright.browser_navigate puppeteer.puppeteer_navigate playwright.browser_tab_close fetch.fetch playwright.browser_tab_list playwright.browser_tab_new git.git_commit git.git_add filesystem.list_allowed_directories memory.add_observations +q032,Take a screenshot of the page,ambiguous,playwright.browser_take_screenshot,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,326,1.0,1,665,1.0,playwright.browser_take_screenshot puppeteer.puppeteer_screenshot playwright.browser_snapshot playwright.browser_press_key playwright.browser_click playwright.browser_evaluate slack.slack_set_channel_topic filesystem.read_file github.get_repository_tree git.git_status +q033,Create a GitLab merge request,verb_product,gitlab.gitlab_merge_merge_request,0.0,1.0,1.0,0.5000,0.6309,2,1,297,1.0,1,475,1.0,1,833,1.0,gitlab.gitlab_merge_merge_request gitlab.gitlab_create_merge_request gitlab.gitlab_get_merge_request gitlab.gitlab_list_merge_requests github.merge_pull_request gitlab.gitlab_create_issue gitlab.gitlab_create_pipeline gitlab.gitlab_list_issues gitlab.gitlab_get_issue gitlab.gitlab_list_projects +q034,Search Notion pages,verb_product,notion.notion_search,1.0,1.0,1.0,1.0000,1.0000,1,1,247,1.0,1,417,1.0,1,721,1.0,notion.notion_search brave_search.brave_web_search slack.slack_search_messages datadog.datadog_list_logs gmail.gmail_search_emails grafana.grafana_search_dashboards confluence.confluence_search notion.notion_get_page notion.notion_get_block notion.notion_query_database +q035,Create a Notion page,verb_product,notion.notion_get_page,0.0,1.0,1.0,0.5000,0.6309,2,1,230,1.0,1,376,1.0,1,822,1.0,notion.notion_get_page notion.notion_create_page notion.notion_update_page confluence.confluence_create_page notion.notion_get_comments notion.notion_create_database notion.create_comment notion.create_users_token notion.create_data_source confluence.confluence_get_page +q036,Query a Notion database,verb_product,notion.notion_query_database,1.0,1.0,1.0,1.0000,1.0000,1,1,279,1.0,1,430,1.0,1,752,1.0,notion.notion_query_database notion.notion_create_database notion.notion_create_page prometheus.prom_query prometheus.prom_query_range datadog.datadog_query_metrics postgres.explain_query salesforce.salesforce_query notion.notion_get_page notion.notion_get_block +q037,Create a Zendesk ticket,verb_product,zendesk.zendesk_create_ticket,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,426,1.0,1,791,1.0,zendesk.zendesk_create_ticket zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_list_tickets jira.jira_create_issue zendesk.zendesk_search slack.slack_create_channel linear.linear_create_project notion.notion_create_database stripe.stripe_create_customer +q038,Create a HubSpot contact,verb_product,hubspot.hubspot_create_contact,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,838,1.0,hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_create_deal hubspot.hubspot_list_contacts hubspot.hubspot_search_contacts github.create_gist github.create_repository github.create_branch github.create_issue github.projects_write +q039,Create a Salesforce lead,verb_product,salesforce.salesforce_create_lead,1.0,1.0,1.0,1.0000,1.0000,1,1,252,1.0,1,416,1.0,1,758,1.0,salesforce.salesforce_create_lead salesforce.salesforce_convert_lead salesforce.salesforce_create_record salesforce.salesforce_update_record salesforce.salesforce_delete_record salesforce.salesforce_list_opportunities salesforce.salesforce_query salesforce.salesforce_get_record slack.slack_create_channel linear.linear_create_project +q040,Run a SOQL query,identifier,salesforce.salesforce_query,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,324,1.0,1,673,1.0,salesforce.salesforce_query sqlite.sqlite_query mysql.mysql_query postgres.query notion.notion_query_database prometheus.prom_query prometheus.prom_query_range docker.docker_run terraform.tf_plan terraform.tf_apply +q041,List Shopify orders,verb_product,shopify.shopify_list_orders,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,354,1.0,1,675,1.0,shopify.shopify_list_orders shopify.shopify_list_products shopify.shopify_list_customers shopify.shopify_get_product shopify.shopify_create_product shopify.shopify_update_product shopify.shopify_get_order shopify.shopify_create_order trello.trello_list_cards git.git_branch +q042,Create a PagerDuty incident,verb_product,pagerduty.pagerduty_create_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,403,1.0,1,773,1.0,pagerduty.pagerduty_create_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls slack.slack_create_channel linear.linear_create_project notion.notion_create_database stripe.stripe_create_customer +q043,Who is on call in PagerDuty,paraphrase,pagerduty.pagerduty_list_oncalls,1.0,1.0,1.0,1.0000,1.0000,1,1,227,1.0,1,401,1.0,1,721,1.0,pagerduty.pagerduty_list_oncalls pagerduty.pagerduty_list_incidents pagerduty.pagerduty_get_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident twilio.twilio_make_call filesystem.list_allowed_directories trello.trello_add_comment notion.notion_get_comments +q044,Query Datadog metrics,verb_product,datadog.datadog_query_metrics,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,458,1.0,1,809,1.0,datadog.datadog_query_metrics datadog.datadog_list_monitors datadog.datadog_get_monitor datadog.datadog_create_monitor datadog.datadog_list_logs notion.notion_query_database prometheus.prom_query prometheus.prom_query_range postgres.explain_query salesforce.salesforce_query +q045,Search Datadog logs,verb_product,datadog.datadog_list_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,245,1.0,1,419,1.0,1,863,1.0,datadog.datadog_list_logs git.git_log datadog.datadog_list_monitors datadog.datadog_get_monitor datadog.datadog_create_monitor datadog.datadog_query_metrics docker.docker_logs kubernetes.k8s_pod_logs brave_search.brave_web_search github.actions_get_job_logs +q046,List S3 buckets,verb_product,aws_s3.s3_list_buckets,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,263,1.0,1,490,1.0,aws_s3.s3_list_buckets trello.trello_list_cards git.git_branch linear.linear_list_teams linear.linear_list_projects jira.jira_list_projects docker.docker_images kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_get_events +q047,Upload an object to S3,paraphrase,aws_s3.s3_put_object,1.0,1.0,1.0,1.0000,1.0000,1,1,262,1.0,1,423,1.0,1,788,1.0,aws_s3.s3_put_object aws_s3.s3_delete_object slack.slack_upload_file aws_s3.s3_get_object slack.slack_add_reaction github.assign_copilot_to_issue slack.slack_reply_to_thread elasticsearch.es_search okta.okta_get_user okta.okta_create_user +q048,Purge Cloudflare cache,verb_product,cloudflare.cf_purge_cache,1.0,1.0,1.0,1.0000,1.0000,1,1,231,1.0,1,340,1.0,1,746,1.0,cloudflare.cf_purge_cache cloudflare.cf_list_zones cloudflare.cf_create_dns cloudflare.cf_list_dns +q049,Create a Cloudflare DNS record,verb_product,cloudflare.cf_create_dns,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,356,1.0,1,732,1.0,cloudflare.cf_create_dns cloudflare.cf_list_dns salesforce.salesforce_create_record cloudflare.cf_list_zones cloudflare.cf_purge_cache salesforce.salesforce_update_record salesforce.salesforce_delete_record salesforce.salesforce_get_record git.git_commit slack.slack_create_channel +q050,List Vercel deployments,verb_product,vercel.vercel_list_deployments,1.0,1.0,1.0,1.0000,1.0000,1,1,173,1.0,1,291,1.0,1,527,1.0,vercel.vercel_list_deployments kubernetes.k8s_list_deployments vercel.vercel_list_projects vercel.vercel_get_deployment vercel.vercel_create_project trello.trello_list_cards git.git_branch linear.linear_list_teams linear.linear_list_projects jira.jira_list_projects +q051,Insert a row into Supabase,verb_product,supabase.supabase_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,357,1.0,1,665,1.0,supabase.supabase_insert mongodb.mongo_insert supabase.supabase_invoke_fn playwright.browser_type supabase.supabase_list_tables supabase.supabase_select supabase.supabase_update +q052,Export images from Figma,paraphrase,figma.figma_get_images,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,328,1.0,1,658,1.0,figma.figma_get_images figma.figma_get_file_nodes docker.docker_images figma.figma_get_file figma.figma_get_comments figma.figma_post_comment elasticsearch.es_get_doc aws_s3.s3_get_object memory.delete_observations docker.docker_logs +q053,Send an SMS with Twilio,verb_product,twilio.twilio_send_sms,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,377,1.0,1,738,1.0,twilio.twilio_send_sms twilio.twilio_list_messages gmail.gmail_send_email puppeteer.puppeteer_click puppeteer.puppeteer_fill twilio.twilio_make_call discord.discord_send_message puppeteer.puppeteer_screenshot jira.jira_search_issues puppeteer.puppeteer_navigate +q054,Create an Okta user,verb_product,okta.okta_create_user,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,329,1.0,1,743,1.0,okta.okta_create_user okta.okta_get_user okta.okta_deactivate_user linear.create_user okta.okta_list_users asana.asana_create_task asana.asana_create_project github.create_issue slack.slack_get_user_profile linear.get_user +q055,Instant query Prometheus,verb_product,prometheus.prom_query,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,407,1.0,1,680,1.0,prometheus.prom_query prometheus.prom_query_range prometheus.prom_list_metrics notion.notion_query_database datadog.datadog_query_metrics postgres.explain_query salesforce.salesforce_query sqlite.sqlite_query mysql.mysql_query postgres.query +q056,Search Grafana dashboards,verb_product,grafana.grafana_search_dashboards,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,375,1.0,1,735,1.0,grafana.grafana_search_dashboards grafana.grafana_get_dashboard grafana.grafana_create_annotation brave_search.brave_web_search slack.slack_search_messages datadog.datadog_list_logs gmail.gmail_search_emails confluence.confluence_search github.search_issues github.search_users +q057,Trigger a CircleCI pipeline,verb_product,circleci.circleci_trigger_pipeline,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,390,1.0,1,718,1.0,circleci.circleci_trigger_pipeline circleci.circleci_get_pipeline circleci.circleci_list_pipelines gitlab.gitlab_create_pipeline github.actions_run_trigger gitlab.create_pipeline gitlab.get_pipeline gitlab.update_pipeline gitlab.delete_pipeline +q058,Run terraform plan,identifier,terraform.tf_plan,1.0,1.0,1.0,1.0000,1.0000,1,1,179,1.0,1,320,1.0,1,658,1.0,terraform.tf_plan terraform.tf_apply postgres.explain_query terraform.tf_state_list docker.docker_run salesforce.salesforce_query github.create_check_run github.get_check_run github.update_check_run github.delete_check_run +q059,Create an Asana task,verb_product,asana.asana_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,429,1.0,1,825,1.0,asana.asana_create_task asana.asana_get_task asana.asana_update_task asana.asana_create_project clickup.clickup_create_task asana.asana_list_tasks okta.okta_create_user clickup.clickup_get_task clickup.clickup_update_task github.create_issue +q060,Create a Trello card,verb_product,trello.trello_create_card,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,388,1.0,1,740,1.0,trello.trello_create_card trello.trello_add_comment trello.trello_create_board trello.trello_list_cards slack.slack_create_channel linear.linear_create_project notion.notion_create_database stripe.stripe_create_customer stripe.stripe_create_invoice stripe.stripe_create_product +q061,Create a ClickUp task,verb_product,clickup.clickup_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,415,1.0,1,745,1.0,clickup.clickup_create_task clickup.clickup_get_task clickup.clickup_update_task clickup.clickup_list_tasks asana.asana_create_task asana.asana_get_task asana.asana_update_task puppeteer.puppeteer_click playwright.browser_click slack.slack_create_channel +q062,Create a Confluence page,verb_product,confluence.confluence_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,401,1.0,1,692,1.0,confluence.confluence_create_page confluence.confluence_get_page confluence.confluence_update_page notion.notion_create_page confluence.confluence_search notion.notion_get_page notion.notion_update_page notion.notion_get_comments playwright.browser_press_key playwright.browser_click +q063,Send a Discord message,verb_product,discord.discord_send_message,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,322,1.0,1,682,1.0,discord.discord_send_message discord.discord_add_reaction discord.discord_get_channel discord.discord_list_channels discord.discord_list_members gmail.gmail_send_email twilio.twilio_send_sms slack.slack_post_message gmail.gmail_get_email gmail.gmail_trash_email +q064,Search Elasticsearch,verb_product,elasticsearch.es_search,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,397,1.0,1,766,1.0,elasticsearch.es_search elasticsearch.es_index_doc elasticsearch.es_get_doc brave_search.brave_web_search slack.slack_search_messages datadog.datadog_list_logs gmail.gmail_search_emails grafana.grafana_search_dashboards confluence.confluence_search github.search_issues +q065,Get a Redis key,verb_product,redis.redis_get,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,285,1.0,1,610,1.0,redis.redis_get redis.redis_set jira.jira_get_issue redis.redis_delete playwright.browser_press_key redis.redis_keys github.get_dependabot_alert notion.notion_get_block kubernetes.k8s_get_pod shopify.shopify_get_product +q066,Find MongoDB documents,verb_product,mongodb.mongo_find,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,381,1.0,1,738,1.0,mongodb.mongo_find mongodb.mongo_insert mongodb.mongo_update mongodb.mongo_delete mongodb.mongo_list_collections +q067,List SQLite tables,verb_product,sqlite.sqlite_list_tables,1.0,1.0,1.0,1.0000,1.0000,1,1,156,1.0,1,243,1.0,1,494,1.0,sqlite.sqlite_list_tables sqlite.sqlite_describe_table sqlite.sqlite_query mysql.mysql_list_tables supabase.supabase_list_tables postgres.list_tables trello.trello_list_cards git.git_branch linear.linear_list_teams linear.linear_list_projects +q068,Describe a postgres table,verb_product,sqlite.sqlite_describe_table,0.0,1.0,1.0,0.5000,0.6309,2,1,200,1.0,1,349,1.0,1,706,1.0,sqlite.sqlite_describe_table postgres.describe_table supabase.supabase_select supabase.supabase_insert supabase.supabase_update +q069,github_create_issue,identifier,bitbucket.bitbucket_create_issue,0.0,0.0,0.0,0.0000,0.0000,39,13,3683,0.0,8,3724,0.0,4,3648,0.0,bitbucket.bitbucket_create_issue gitlab.gitlab_create_issue github.create_milestone github.create_hook github.create_environment github.create_package github.create_artifact github.create_runner linear.linear_create_issue github.create_check_run +q070,createIssue,identifier,bitbucket.bitbucket_create_issue,0.0,1.0,1.0,0.3333,0.5584,3,1,332,1.0,1,579,1.0,1,1039,1.0,bitbucket.bitbucket_create_issue gitlab.gitlab_create_issue linear.linear_create_issue github.create_issue jira.jira_create_issue linear.linear_update_issue jira.jira_assign_issue sentry.sentry_get_issue github.issue_write gitlab.gitlab_get_issue +q071,list-directory,identifier,filesystem.create_directory,0.0,0.0,0.0,0.0000,0.0000,90,30,7901,0.0,18,7685,0.0,9,7523,0.0,filesystem.create_directory filesystem.get_file_info github.get_file_contents trello.trello_list_cards git.git_branch linear.linear_list_teams linear.linear_list_projects jira.jira_list_projects docker.docker_images kubernetes.k8s_list_deployments +q072,sequential thinking next thought,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,116,39,9799,0.0,24,9727,0.0,12,9475,0.0, +q073,What time is it in Tokyo,paraphrase,time.convert_time,1.0,1.0,1.0,1.0000,0.8597,1,1,190,1.0,1,342,1.0,1,709,1.0,time.convert_time time.get_current_time filesystem.list_allowed_directories playwright.browser_wait_for elasticsearch.es_index_doc github.create_milestone github.get_milestone github.list_milestones github.update_milestone github.delete_milestone +q074,Convert time between timezones,verb_product,time.convert_time,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,335,1.0,1,636,1.0,time.convert_time salesforce.salesforce_convert_lead time.get_current_time git.git_diff playwright.browser_wait_for memory.create_relations +q075,Star a GitHub repo,verb_product,github.star_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,394,1.0,1,772,1.0,github.star_repository github.delete_repository github.fork_repository github.unstar_repository github.search_issues github.get_discussion github.create_gist github.update_gist github.projects_list github.projects_get +q076,Fork a repository,resource,github.fork_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,235,1.0,1,386,1.0,1,822,1.0,github.fork_repository github.delete_repository github.star_repository github.unstar_repository github.create_repository github.list_dependabot_alerts github.list_branches github.list_releases github.create_branch github.delete_file +q077,Get the latest GitHub release,verb_product,github.get_latest_release,1.0,1.0,1.0,1.0000,1.0000,1,1,291,1.0,1,383,1.0,1,759,1.0,github.get_latest_release github.get_release_by_tag github.get_repository_tree gitlab.get_release github.get_teams github.get_discussion github.projects_get github.get_label github.get_tag github.get_me +q078,Create a gist,resource,github.create_gist,1.0,1.0,1.0,1.0000,1.0000,1,1,229,1.0,1,383,1.0,1,733,1.0,github.create_gist github.update_gist github.get_gist slack.slack_create_channel linear.linear_create_project notion.notion_create_database stripe.stripe_create_customer stripe.stripe_create_invoice stripe.stripe_create_product stripe.stripe_create_price +q079,Search GitHub code,verb_product,github.search_code,1.0,1.0,1.0,1.0000,1.0000,1,1,283,1.0,1,419,1.0,1,725,1.0,github.search_code github.get_code_scanning_alert github.list_code_scanning_alerts github.search_issues github.search_users github.search_orgs github.search_commits github.search_pull_requests hubspot.hubspot_search_contacts github.search_repositories +q080,List GitHub notifications,verb_product,github.list_notifications,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,326,1.0,1,717,1.0,github.list_notifications github.mark_all_notifications_read github.projects_list github.actions_list github.list_tags github.get_team_members github.list_branches github.list_releases github.list_repository_collaborators github.list_issues +q081,Assign Copilot to an issue,verb_product,github.assign_copilot_to_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,421,1.0,1,819,1.0,github.assign_copilot_to_issue linear.linear_assign_issue jira.jira_assign_issue aws_s3.s3_put_object linear.linear_create_comment jira.jira_update_issue jira.jira_add_comment jira.jira_add_worklog github.add_issue_comment jira.jira_transition_issue +q082,Get secret scanning alerts,resource,github.list_secret_scanning_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,295,1.0,1,470,1.0,1,804,1.0,github.list_secret_scanning_alerts github.get_secret_scanning_alert github.list_code_scanning_alerts github.get_code_scanning_alert github.list_dependabot_alerts github.get_dependabot_alert notion.notion_get_block kubernetes.k8s_get_pod shopify.shopify_get_product shopify.shopify_get_order +q083,Create or update a file on GitHub,verb_product,github.create_or_update_file,1.0,1.0,1.0,1.0000,0.7602,1,1,369,1.0,1,610,1.0,1,1088,1.0,github.create_or_update_file github.projects_write github.label_write github.update_issue_comment github.issue_write github.get_file_contents filesystem.write_file github.pull_request_review_write github.update_gist github.list_repository_collaborators +q084,List collaborators on a repo,resource,github.list_repository_collaborators,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,411,1.0,1,751,1.0,github.list_repository_collaborators trello.trello_list_cards pagerduty.pagerduty_list_oncalls github.list_commits trello.trello_add_comment notion.notion_get_comments playwright.browser_press_key figma.figma_get_comments figma.figma_post_comment github.update_issue_comment +q085,Reply in a Slack thread,verb_product,slack.slack_reply_to_thread,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,397,1.0,1,767,1.0,slack.slack_reply_to_thread slack.slack_get_thread_replies github.get_notification_details slack.create_reminder slack.get_reminder slack.list_reminders slack.update_reminder slack.delete_reminder slack.create_canvase slack.get_canvase +q086,Add an emoji reaction on Slack,verb_product,slack.slack_add_reaction,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,461,1.0,1,827,1.0,slack.slack_add_reaction discord.discord_add_reaction gmail.gmail_modify_email playwright.browser_click github.assign_copilot_to_issue linear.linear_create_comment jira.jira_add_comment jira.jira_add_worklog trello.trello_add_comment aws_s3.s3_delete_object +q087,Search Slack messages,verb_product,slack.slack_search_messages,1.0,1.0,1.0,1.0000,1.0000,1,1,232,1.0,1,373,1.0,1,721,1.0,slack.slack_search_messages gmail.gmail_search_emails slack.slack_get_channel_history twilio.twilio_list_messages brave_search.brave_web_search datadog.datadog_list_logs grafana.grafana_search_dashboards confluence.confluence_search github.search_issues github.search_users +q088,Create a Slack channel,verb_product,slack.slack_create_channel,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,339,1.0,1,773,1.0,slack.slack_create_channel slack.slack_join_channel slack.slack_leave_channel slack.slack_post_message slack.slack_get_channel_history slack.slack_set_channel_topic discord.discord_get_channel slack.create_reminder slack.create_canvase slack.create_bookmark +q089,Transition a Jira issue,verb_product,jira.jira_transition_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,222,1.0,1,342,1.0,1,822,1.0,jira.jira_transition_issue jira.jira_assign_issue jira.jira_update_issue jira.jira_get_issue jira.jira_add_comment jira.jira_add_worklog jira.jira_create_issue linear.linear_update_issue sentry.sentry_get_issue bitbucket.bitbucket_create_issue +q090,Search Jira with JQL,identifier,jira.jira_search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,370,1.0,1,734,1.0,jira.jira_search_issues puppeteer.puppeteer_screenshot brave_search.brave_web_search puppeteer.puppeteer_navigate puppeteer.puppeteer_click puppeteer.puppeteer_fill puppeteer.puppeteer_evaluate slack.slack_search_messages datadog.datadog_list_logs gmail.gmail_search_emails +q091,Add a worklog to Jira,verb_product,jira.jira_add_worklog,1.0,1.0,1.0,1.0000,1.0000,1,1,247,1.0,1,440,1.0,1,874,1.0,jira.jira_add_worklog jira.jira_add_comment linear.linear_create_comment discord.discord_add_reaction github.add_issue_comment slack.slack_add_reaction git.git_add jira.jira_transition_issue github.add_comment_to_pending_review memory.add_observations +q092,List Linear teams,verb_product,linear.linear_list_teams,1.0,1.0,1.0,1.0000,1.0000,1,1,172,1.0,1,318,1.0,1,705,1.0,linear.linear_list_teams linear.linear_list_projects linear.list_cycles linear.list_labels linear.list_users linear.list_comments linear.list_attachments linear.linear_list_issues github.get_teams linear.linear_update_issue +q093,Comment on a Linear issue,verb_product,linear.linear_create_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,264,1.0,1,451,1.0,1,781,1.0,linear.linear_create_comment github.update_issue_comment trello.trello_add_comment linear.linear_update_issue figma.figma_post_comment linear.create_comment linear.get_comment linear.update_comment linear.delete_comment jira.jira_add_comment +q094,Move a local file,paraphrase,filesystem.read_text_file,0.0,0.0,1.0,0.2000,0.3869,5,2,434,0.0,1,338,1.0,1,712,1.0,filesystem.read_text_file gmail.gmail_trash_email filesystem.read_file filesystem.read_media_file filesystem.move_file brave_search.brave_local_search figma.figma_get_file filesystem.write_file slack.slack_upload_file google_drive.gdrive_update_file +q095,Recursively search files by pattern,paraphrase,filesystem.search_files,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,338,1.0,1,741,1.0,filesystem.search_files google_drive.gdrive_search linear.linear_search_issues redis.redis_keys github.search_repositories slack.list_files brave_search.brave_web_search slack.slack_search_messages datadog.datadog_list_logs gmail.gmail_search_emails +q096,Edit a file in place with old and new string,paraphrase,filesystem.edit_file,1.0,1.0,1.0,1.0000,1.0000,1,1,247,1.0,1,419,1.0,1,763,1.0,filesystem.edit_file filesystem.write_file filesystem.list_directory_with_sizes twilio.twilio_make_call git.git_create_branch playwright.browser_tab_new puppeteer.puppeteer_screenshot slack.create_file slack.get_file slack.update_file +q097,Read the whole knowledge graph,paraphrase,memory.read_graph,1.0,1.0,1.0,1.0000,1.0000,1,1,152,1.0,1,259,1.0,1,561,1.0,memory.read_graph memory.delete_relations memory.create_entities memory.create_relations memory.add_observations memory.delete_entities memory.open_nodes memory.search_nodes filesystem.read_text_file filesystem.read_file +q098,Search memory nodes,verb_product,memory.search_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,241,1.0,1,409,1.0,1,729,1.0,memory.search_nodes figma.figma_get_file_nodes brave_search.brave_web_search slack.slack_search_messages datadog.datadog_list_logs gmail.gmail_search_emails grafana.grafana_search_dashboards confluence.confluence_search memory.open_nodes github.search_issues +q099,Show unstaged git diff,verb_product,git.git_diff_unstaged,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,313,1.0,1,628,1.0,git.git_diff_unstaged git.git_diff git.git_show git.git_diff_staged git.git_log git.git_status github.list_tags github.get_tag git.git_checkout git.git_branch +q100,Stage files with git add,identifier,filesystem.list_directory_with_sizes,0.0,0.0,1.0,0.2000,0.3869,5,2,495,0.0,1,402,1.0,1,710,1.0,filesystem.list_directory_with_sizes github.add_issue_comment puppeteer.puppeteer_screenshot slack.list_files git.git_add jira.jira_search_issues puppeteer.puppeteer_navigate puppeteer.puppeteer_click puppeteer.puppeteer_fill puppeteer.puppeteer_evaluate +q101,Create a Shopify product,verb_product,shopify.shopify_create_product,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,377,1.0,1,694,1.0,shopify.shopify_create_product shopify.shopify_get_product shopify.shopify_update_product stripe.stripe_create_product shopify.shopify_create_order shopify.shopify_list_products shopify.shopify_list_orders shopify.shopify_get_order shopify.shopify_list_customers slack.slack_create_channel +q102,Cancel a Stripe subscription,verb_product,stripe.stripe_cancel_subscription,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,301,1.0,1,632,1.0,stripe.stripe_cancel_subscription stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_list_customers stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_create_product stripe.stripe_create_price stripe.stripe_list_subscriptions stripe.stripe_create_refund +q103,Create a payment intent,resource,stripe.stripe_create_payment_intent,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,405,1.0,1,738,1.0,stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent slack.slack_create_channel linear.linear_create_project notion.notion_create_database stripe.stripe_create_customer stripe.stripe_create_invoice stripe.stripe_create_product stripe.stripe_create_price stripe.stripe_create_refund +q104,Acknowledge a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,261,1.0,1,405,1.0,1,717,1.0,pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls +q105,Create a Datadog monitor,verb_product,datadog.datadog_create_monitor,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,459,1.0,1,815,1.0,datadog.datadog_create_monitor datadog.datadog_get_monitor datadog.datadog_list_monitors datadog.datadog_list_logs datadog.datadog_query_metrics slack.slack_create_channel linear.linear_create_project notion.notion_create_database stripe.stripe_create_customer stripe.stripe_create_invoice +q106,Resolve a Sentry issue,paraphrase,sentry.sentry_get_issue,0.0,1.0,1.0,0.3333,0.5000,3,1,251,1.0,1,368,1.0,1,826,1.0,sentry.sentry_get_issue sentry.sentry_list_events sentry.sentry_update_issue pagerduty.pagerduty_resolve_incident sentry.sentry_list_projects sentry.sentry_list_issues linear.linear_update_issue jira.jira_assign_issue bitbucket.bitbucket_create_issue gitlab.gitlab_create_issue +q107,Scale a Kubernetes deployment,verb_product,kubernetes.k8s_scale_deployment,1.0,1.0,1.0,1.0000,1.0000,1,1,256,1.0,1,411,1.0,1,726,1.0,kubernetes.k8s_scale_deployment vercel.vercel_get_deployment kubernetes.k8s_get_pod kubernetes.k8s_apply kubernetes.k8s_delete kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_list_pods kubernetes.k8s_pod_logs +q108,Apply a Kubernetes manifest,verb_product,kubernetes.k8s_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,226,1.0,1,370,1.0,1,727,1.0,kubernetes.k8s_apply terraform.tf_apply kubernetes.k8s_get_pod kubernetes.k8s_delete kubernetes.k8s_list_deployments kubernetes.k8s_scale_deployment kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_list_pods kubernetes.k8s_pod_logs +q109,Stop a docker container,verb_product,docker.docker_stop,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,345,1.0,1,581,1.0,docker.docker_stop docker.docker_run docker.docker_rm docker.docker_logs docker.docker_exec docker.docker_images docker.docker_build docker.docker_ps +q110,Build a docker image,verb_product,docker.docker_build,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,317,1.0,1,601,1.0,docker.docker_build docker.docker_run docker.docker_stop docker.docker_rm docker.docker_images docker.docker_ps docker.docker_logs docker.docker_exec filesystem.read_media_file +q111,Click an element in Playwright,verb_product,puppeteer.puppeteer_click,0.0,1.0,1.0,0.5000,0.6309,2,1,216,1.0,1,381,1.0,1,721,1.0,puppeteer.puppeteer_click playwright.browser_click playwright.browser_type playwright.browser_take_screenshot clickup.clickup_create_task clickup.clickup_list_tasks clickup.clickup_get_task clickup.clickup_update_task aws_s3.s3_list_objects aws_s3.s3_delete_object +q112,Type text into a browser input,paraphrase,playwright.browser_type,1.0,1.0,1.0,1.0000,0.9502,1,1,217,1.0,1,303,1.0,1,636,1.0,playwright.browser_type playwright.browser_tab_close puppeteer.puppeteer_fill playwright.browser_tab_list playwright.browser_tab_new mongodb.mongo_insert supabase.supabase_insert playwright.browser_navigate linear.linear_search_issues filesystem.read_text_file +q113,List GitLab projects,verb_product,gitlab.gitlab_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,209,1.0,1,289,1.0,1,615,1.0,gitlab.gitlab_list_projects github.projects_list gitlab.gitlab_list_issues linear.linear_list_projects jira.jira_list_projects sentry.sentry_list_projects vercel.vercel_list_projects gitlab.gitlab_list_merge_requests gitlab.gitlab_create_issue gitlab.gitlab_get_issue +q114,Create a Bitbucket pull request,verb_product,bitbucket.bitbucket_create_pr,1.0,1.0,1.0,1.0000,1.0000,1,1,313,1.0,1,519,1.0,1,1050,1.0,bitbucket.bitbucket_create_pr bitbucket.bitbucket_list_prs bitbucket.bitbucket_create_issue github.merge_pull_request github.request_copilot_review github.pull_request_review_write github.create_pull_request github.pull_request_read bitbucket.bitbucket_list_repos github.update_pull_request_branch +q115,Create a Gmail draft,verb_product,gmail.gmail_create_draft,1.0,1.0,1.0,1.0000,1.0000,1,1,205,1.0,1,352,1.0,1,742,1.0,gmail.gmail_create_draft gmail.gmail_search_emails gmail.gmail_list_labels gmail.gmail_send_email gmail.gmail_get_email gmail.gmail_trash_email gmail.gmail_modify_email slack.slack_create_channel linear.linear_create_project notion.notion_create_database +q116,Share a Google Drive file,verb_product,google_drive.gdrive_share_file,1.0,1.0,1.0,1.0000,1.0000,1,1,306,1.0,1,417,1.0,1,808,1.0,google_drive.gdrive_share_file google_drive.gdrive_update_file google_drive.gdrive_create_file google_drive.gdrive_get_file google_drive.gdrive_search google_drive.gdrive_list_folder google_calendar.gcal_list_calendars google_calendar.gcal_list_events google_calendar.gcal_create_event google_calendar.gcal_update_event +q117,List Google Calendar events,verb_product,google_calendar.gcal_list_events,1.0,1.0,1.0,1.0000,1.0000,1,1,317,1.0,1,441,1.0,1,784,1.0,google_calendar.gcal_list_events google_calendar.gcal_create_event google_calendar.gcal_update_event google_calendar.gcal_delete_event kubernetes.k8s_get_events google_calendar.gcal_list_calendars sentry.sentry_list_events google_drive.gdrive_list_folder google_drive.gdrive_update_file google_drive.gdrive_share_file +q118,Delete a calendar event,resource,google_calendar.gcal_delete_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,477,1.0,1,787,1.0,google_calendar.gcal_delete_event google_calendar.gcal_create_event google_calendar.gcal_update_event google_calendar.gcal_list_events kubernetes.k8s_delete salesforce.salesforce_delete_record redis.redis_delete aws_s3.s3_delete_object github.delete_repository github.delete_milestone +q119,List Vercel projects,verb_product,vercel.vercel_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,163,1.0,1,248,1.0,1,559,1.0,vercel.vercel_list_projects vercel.vercel_list_deployments linear.linear_list_projects jira.jira_list_projects sentry.sentry_list_projects github.projects_list gitlab.gitlab_list_projects vercel.vercel_get_deployment vercel.vercel_create_project trello.trello_list_cards +q120,Select rows from Supabase,verb_product,supabase.supabase_select,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,354,1.0,1,709,1.0,supabase.supabase_select supabase.supabase_insert supabase.supabase_update supabase.supabase_invoke_fn supabase.supabase_list_tables elasticsearch.es_get_doc aws_s3.s3_get_object memory.delete_observations docker.docker_logs kubernetes.k8s_pod_logs +q121,Post a Figma comment,verb_product,figma.figma_post_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,362,1.0,1,720,1.0,figma.figma_post_comment figma.figma_get_file slack.slack_post_message figma.figma_get_comments figma.figma_get_images figma.figma_get_file_nodes trello.trello_add_comment linear.create_comment linear.get_comment linear.update_comment +q122,List Okta users,verb_product,okta.okta_list_users,1.0,1.0,1.0,1.0000,1.0000,1,1,174,1.0,1,301,1.0,1,618,1.0,okta.okta_list_users slack.slack_list_users notion.notion_list_users linear.list_users okta.okta_get_user okta.okta_create_user okta.okta_deactivate_user slack.slack_get_users notion.list_users_tokens pagerduty.pagerduty_list_oncalls +q123,Range query Prometheus,verb_product,prometheus.prom_query_range,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,407,1.0,1,680,1.0,prometheus.prom_query_range prometheus.prom_query prometheus.prom_list_metrics notion.notion_query_database datadog.datadog_query_metrics postgres.explain_query salesforce.salesforce_query sqlite.sqlite_query mysql.mysql_query postgres.query +q124,Create a Grafana annotation,verb_product,grafana.grafana_create_annotation,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,372,1.0,1,722,1.0,grafana.grafana_create_annotation grafana.grafana_search_dashboards grafana.grafana_get_dashboard slack.slack_create_channel linear.linear_create_project notion.notion_create_database stripe.stripe_create_customer stripe.stripe_create_invoice stripe.stripe_create_product stripe.stripe_create_price +q125,Apply terraform,identifier,terraform.tf_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,292,1.0,1,698,1.0,terraform.tf_apply kubernetes.k8s_apply terraform.tf_plan terraform.tf_state_list +q126,List Asana tasks,verb_product,asana.asana_list_tasks,1.0,1.0,1.0,1.0000,1.0000,1,1,245,1.0,1,386,1.0,1,659,1.0,asana.asana_list_tasks clickup.clickup_list_tasks asana.asana_create_task asana.asana_get_task asana.asana_update_task asana.asana_create_project trello.trello_list_cards git.git_branch linear.linear_list_teams linear.linear_list_projects +q127,Comment on a Trello card,verb_product,trello.trello_add_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,438,1.0,1,757,1.0,trello.trello_add_comment trello.trello_create_card figma.figma_post_comment trello.trello_list_cards github.update_issue_comment trello.trello_create_board linear.create_comment linear.get_comment linear.update_comment linear.delete_comment +q128,Update a ClickUp task,verb_product,clickup.clickup_update_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,399,1.0,1,787,1.0,clickup.clickup_update_task clickup.clickup_create_task clickup.clickup_get_task clickup.clickup_list_tasks asana.asana_update_task asana.asana_create_task asana.asana_get_task puppeteer.puppeteer_click playwright.browser_click linear.linear_update_issue +q129,Search Confluence,verb_product,confluence.confluence_search,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,401,1.0,1,785,1.0,confluence.confluence_search confluence.confluence_get_page confluence.confluence_create_page confluence.confluence_update_page brave_search.brave_web_search slack.slack_search_messages datadog.datadog_list_logs gmail.gmail_search_emails grafana.grafana_search_dashboards github.search_issues +q130,Create a HubSpot deal,verb_product,hubspot.hubspot_create_deal,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,838,1.0,hubspot.hubspot_create_deal hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_list_contacts hubspot.hubspot_search_contacts github.create_gist github.create_repository github.create_branch github.create_issue github.projects_write +q131,List Zendesk tickets,verb_product,zendesk.zendesk_list_tickets,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,367,1.0,1,603,1.0,zendesk.zendesk_list_tickets zendesk.zendesk_search zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_create_ticket trello.trello_list_cards git.git_branch linear.linear_list_teams linear.linear_list_projects jira.jira_list_projects +q132,List Discord channels,verb_product,discord.discord_list_channels,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,300,1.0,1,590,1.0,discord.discord_list_channels discord.discord_list_members slack.slack_list_channels discord.discord_get_channel discord.discord_send_message discord.discord_add_reaction trello.trello_list_cards git.git_branch linear.linear_list_teams linear.linear_list_projects +q133,Set a Redis key,verb_product,redis.redis_set,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,307,1.0,1,564,1.0,redis.redis_set redis.redis_get redis.redis_delete slack.slack_set_channel_topic jira.jira_get_issue playwright.browser_press_key redis.redis_keys +q134,Insert MongoDB documents,verb_product,mongodb.mongo_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,256,1.0,1,381,1.0,1,698,1.0,mongodb.mongo_insert mongodb.mongo_update mongodb.mongo_delete mongodb.mongo_find mongodb.mongo_list_collections supabase.supabase_insert +q135,Run a MySQL query,verb_product,mysql.mysql_query,1.0,1.0,1.0,1.0000,1.0000,1,1,179,1.0,1,281,1.0,1,670,1.0,mysql.mysql_query postgres.query sqlite.sqlite_query mysql.mysql_list_tables postgres.explain_query salesforce.salesforce_query notion.notion_query_database prometheus.prom_query prometheus.prom_query_range docker.docker_run +q136,Index a document in Elasticsearch,verb_product,elasticsearch.es_index_doc,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,402,1.0,1,755,1.0,elasticsearch.es_index_doc elasticsearch.es_search elasticsearch.es_get_doc github.create_milestone github.get_milestone github.list_milestones github.update_milestone github.delete_milestone github.create_hook github.get_hook +q137,Get an S3 object,verb_product,aws_s3.s3_get_object,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,355,1.0,1,723,1.0,aws_s3.s3_get_object aws_s3.s3_delete_object aws_s3.s3_put_object okta.okta_get_user asana.asana_get_task elasticsearch.es_search okta.okta_create_user okta.okta_deactivate_user asana.asana_create_task asana.asana_update_task +q138,List Cloudflare zones,verb_product,cloudflare.cf_list_zones,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,361,1.0,1,554,1.0,cloudflare.cf_list_zones cloudflare.cf_purge_cache cloudflare.cf_list_dns cloudflare.cf_create_dns trello.trello_list_cards git.git_branch linear.linear_list_teams linear.linear_list_projects jira.jira_list_projects docker.docker_images +q139,Create a Stripe invoice,verb_product,stripe.stripe_create_invoice,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,353,1.0,1,669,1.0,stripe.stripe_create_invoice stripe.stripe_create_customer stripe.stripe_create_product stripe.stripe_create_price stripe.stripe_create_refund stripe.stripe_create_payment_intent stripe.stripe_get_customer stripe.stripe_list_customers stripe.stripe_list_invoices stripe.stripe_list_subscriptions +q140,List GitHub Actions workflows,verb_product,github.actions_list,1.0,1.0,1.0,1.0000,1.0000,1,1,293,1.0,1,453,1.0,1,812,1.0,github.actions_list github.actions_run_trigger github.actions_get github.actions_get_job_logs github.projects_list github.list_tags github.get_team_members github.list_branches github.list_releases github.list_repository_collaborators +q141,Get GitHub Actions job logs,verb_product,github.actions_get_job_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,442,1.0,1,839,1.0,github.actions_get_job_logs github.actions_get github.actions_list github.actions_run_trigger gitlab.get_job kubernetes.k8s_pod_logs github.get_discussion github.projects_get github.get_label git.git_log +q142,Create a GitHub repository,verb_product,github.create_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,268,1.0,1,417,1.0,1,920,1.0,github.create_repository github.create_branch github.delete_repository github.fork_repository github.star_repository github.unstar_repository github.create_issue github.list_tags github.get_tag github.create_or_update_file +q143,Checkout a git branch,verb_product,git.git_create_branch,0.0,0.0,0.0,0.1429,0.3333,7,3,766,0.0,2,852,0.0,1,832,1.0,git.git_create_branch github.create_branch github.update_pull_request_branch github.list_commits github.list_tags github.get_tag git.git_checkout git.git_show git.git_branch github.get_repository_tree +q144,Show git log,identifier,git.git_show,0.0,0.0,1.0,0.2500,0.4307,4,2,382,0.0,1,311,1.0,1,626,1.0,git.git_show git.git_diff_unstaged git.git_diff_staged git.git_log git.git_diff git.git_status github.list_tags github.get_tag git.git_checkout git.git_branch +q145,List postgres schemas,verb_product,postgres.list_schemas,1.0,1.0,1.0,1.0000,1.0000,1,1,178,1.0,1,258,1.0,1,485,1.0,postgres.list_schemas trello.trello_list_cards git.git_branch linear.linear_list_teams linear.linear_list_projects jira.jira_list_projects docker.docker_images kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_get_events +q146,Open nodes in the knowledge graph,verb_product,memory.open_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,163,1.0,1,260,1.0,1,505,1.0,memory.open_nodes memory.search_nodes memory.read_graph memory.create_entities memory.delete_relations memory.create_relations memory.add_observations memory.delete_entities playwright.browser_tab_list playwright.browser_tab_new +q147,Get file metadata from the filesystem,paraphrase,filesystem.read_text_file,0.0,0.0,1.0,0.2000,0.3869,5,2,447,0.0,1,334,1.0,1,750,1.0,filesystem.read_text_file filesystem.read_file filesystem.read_media_file google_drive.gdrive_get_file filesystem.get_file_info gitlab.gitlab_get_file figma.figma_get_file_nodes figma.figma_get_images github.get_file_contents figma.figma_get_file +q148,Write a new local file,paraphrase,filesystem.read_text_file,0.0,1.0,1.0,0.5000,0.6309,2,1,234,1.0,1,376,1.0,1,803,1.0,filesystem.read_text_file filesystem.write_file filesystem.read_file filesystem.read_media_file git.git_create_branch playwright.browser_tab_new filesystem.edit_file github.issue_write github.create_repository jira.jira_transition_issue +q149,Search GitHub issues,verb_product,github.search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,335,1.0,1,683,1.0,github.search_issues github.search_users github.search_orgs linear.linear_search_issues jira.jira_search_issues github.list_issues github.search_commits github.search_pull_requests gitlab.gitlab_list_issues github.search_code +q150,Get the authenticated GitHub user,paraphrase,github.get_teams,0.0,1.0,1.0,0.3333,0.5000,3,1,179,1.0,1,356,1.0,1,699,1.0,github.get_teams github.list_notifications github.get_me github.get_latest_release github.get_repository_tree github.list_gists github.list_starred_repositories github.get_discussion github.projects_get github.get_label diff --git a/experiments/mcp-tool-search/results/bm25_desc_only.json b/experiments/mcp-tool-search/results/bm25_desc_only.json new file mode 100644 index 00000000..5311c63e --- /dev/null +++ b/experiments/mcp-tool-search/results/bm25_desc_only.json @@ -0,0 +1,6955 @@ +{ + "summary": { + "n": 150, + "index_s": 0.0007229240000015125, + "rss_mb": 0.0390625, + "p50_ms": 0.1389104999702795, + "p95_ms": 0.2481368999781352, + "recall@1": 0.8266666666666667, + "recall@1_lo": 0.76, + "recall@1_hi": 0.8866666666666667, + "recall@3": 0.9266666666666666, + "recall@3_lo": 0.8866666666666667, + "recall@3_hi": 0.9666666666666667, + "recall@5": 0.96, + "recall@5_lo": 0.9266666666666666, + "recall@5_hi": 0.9866666666666667, + "mrr@10": 0.8805079365079365, + "mrr@10_lo": 0.8359047619047619, + "mrr@10_hi": 0.9216666666666666, + "ndcg@10": 0.9000526376116909, + "ndcg@10_lo": 0.862655362957362, + "ndcg@10_hi": 0.9347402167331347 + }, + "meta": { + "name": "bm25_desc_only", + "needs_model": false, + "needs_api": false, + "notes": "BM25 over description field" + }, + "rows": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "gitlab.gitlab_create_issue", + "github.create_issue", + "github.issue_write", + "github.issue_read", + "github.add_issue_comment", + "github.update_issue_comment", + "github.create_gist", + "bitbucket.bitbucket_create_issue", + "github.create_repository", + "gitlab.gitlab_get_issue" + ], + "hit1": "gitlab.gitlab_create_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.66967181649423, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 380, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 584, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1011, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "kind": "verb_product", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_create_issue", + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "jira.jira_create_sprint", + "jira.create_board", + "jira.create_sprint", + "jira.create_filter", + "jira.create_webhook", + "jira.create_attachment" + ], + "hit1": "jira.jira_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 281, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 481, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 966, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "kind": "verb_product", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "slack.slack_post_message", + "slack.slack_add_reaction", + "gmail.gmail_send_email", + "twilio.twilio_send_sms", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "discord.discord_add_reaction", + "gmail.gmail_modify_email", + "slack.slack_join_channel" + ], + "hit1": "discord.discord_send_message", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 771, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "kind": "resource", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "ranking": [ + "bitbucket.bitbucket_list_prs", + "github.search_pull_requests", + "github.list_pull_requests", + "github.create_pull_request", + "playwright.browser_tab_list", + "gitlab.gitlab_list_merge_requests", + "playwright.browser_tab_new", + "bitbucket.bitbucket_create_pr", + "github.merge_pull_request", + "memory.open_nodes" + ], + "hit1": "bitbucket.bitbucket_list_prs", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6199062332840657, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 423, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "ranking": [ + "github.create_pull_request", + "github.request_copilot_review", + "github.merge_pull_request", + "github.pull_request_read", + "github.pull_request_review_write", + "bitbucket.bitbucket_create_pr", + "github.add_comment_to_pending_review", + "github.update_pull_request", + "github.search_pull_requests", + "github.list_repository_collaborators" + ], + "hit1": "github.create_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 379, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 612, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1163, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "kind": "verb_product", + "gold": [ + "postgres.query" + ], + "alts": [], + "ranking": [ + "mysql.mysql_query", + "sqlite.sqlite_query", + "postgres.query", + "salesforce.salesforce_query", + "postgres.explain_query", + "notion.notion_query_database", + "prometheus.prom_query", + "prometheus.prom_query_range", + "mysql.mysql_list_tables", + "docker.docker_run" + ], + "hit1": "mysql.mysql_query", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 671, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "kind": "paraphrase", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "ranking": [ + "filesystem.read_text_file", + "filesystem.read_file", + "filesystem.read_media_file", + "figma.figma_get_images", + "github.delete_file", + "gitlab.gitlab_get_file", + "figma.figma_get_file_nodes", + "github.get_file_contents", + "memory.read_graph", + "github.issue_read" + ], + "hit1": "filesystem.read_text_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8597186998521972, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 425, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 832, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "kind": "paraphrase", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "ranking": [ + "google_drive.gdrive_list_folder", + "slack.list_files", + "google_drive.gdrive_search", + "filesystem.move_file", + "filesystem.search_files", + "filesystem.list_directory", + "github.push_files", + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree", + "elasticsearch.es_index_doc" + ], + "hit1": "google_drive.gdrive_list_folder", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.16666666666666666, + "ndcg@10": 0.424445564500406, + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 413, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 727, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "ranking": [ + "memory.read_graph", + "memory.create_entities", + "memory.delete_relations", + "memory.create_relations", + "memory.add_observations", + "memory.open_nodes", + "memory.search_nodes", + "memory.delete_entities", + "time.get_current_time", + "slack.slack_list_channels" + ], + "hit1": "memory.read_graph", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.5338931479009518, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 154, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 261, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 537, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "kind": "verb_product", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "ranking": [ + "brave_search.brave_web_search", + "jira.jira_search_issues", + "brave_search.brave_local_search", + "github.update_pull_request_branch", + "puppeteer.puppeteer_screenshot", + "puppeteer.puppeteer_navigate", + "puppeteer.puppeteer_click", + "puppeteer.puppeteer_fill", + "puppeteer.puppeteer_evaluate", + "memory.search_nodes" + ], + "hit1": "brave_search.brave_web_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 430, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 723, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "kind": "verb_product", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "ranking": [ + "fetch.fetch", + "filesystem.read_file", + "playwright.browser_navigate", + "git.git_add", + "slack.slack_set_channel_topic", + "docker.docker_logs", + "puppeteer.puppeteer_navigate", + "github.get_repository_tree", + "git.git_status", + "playwright.browser_snapshot" + ], + "hit1": "fetch.fetch", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 346, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "kind": "identifier", + "gold": [ + "git.git_status" + ], + "alts": [], + "ranking": [ + "git.git_status", + "git.git_show", + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_log", + "git.git_diff", + "sentry.sentry_update_issue", + "jira.jira_transition_issue", + "github.list_tags", + "github.get_tag" + ], + "hit1": "git.git_status", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "kind": "paraphrase", + "gold": [ + "git.git_commit" + ], + "alts": [], + "ranking": [ + "git.git_reset", + "git.git_commit", + "git.git_log", + "git.git_show", + "git.git_diff_staged", + "github.get_commit", + "github.push_files" + ], + "hit1": "git.git_reset", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 305, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_issue", + "linear.linear_update_issue", + "bitbucket.bitbucket_create_issue", + "linear.linear_create_comment", + "linear.linear_assign_issue", + "linear.linear_create_project", + "gitlab.gitlab_create_issue", + "linear.create_cycle", + "linear.create_label", + "linear.create_user" + ], + "hit1": "linear.linear_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 357, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 498, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 956, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "kind": "paraphrase", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_project", + "mysql.mysql_list_tables", + "supabase.supabase_list_tables", + "vercel.vercel_create_project", + "mysql.mysql_query", + "asana.asana_create_task", + "asana.asana_get_task", + "asana.asana_update_task", + "clickup.clickup_create_task", + "clickup.clickup_get_task" + ], + "hit1": "linear.linear_create_project", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 146, + "k3_calls": 49, + "k3_tokens": 12370, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 30, + "k5_tokens": 12110, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11750, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "ranking": [ + "github.update_issue_comment", + "figma.figma_post_comment", + "github.add_issue_comment", + "trello.trello_add_comment", + "github.assign_copilot_to_issue", + "linear.linear_create_comment", + "jira.jira_add_comment", + "github.list_repository_collaborators", + "github.get_discussion_comments", + "github.issue_read" + ], + "hit1": "github.update_issue_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 307, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 482, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 887, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "kind": "ambiguous", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "ranking": [ + "github.merge_pull_request", + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_create_merge_request", + "gitlab.gitlab_get_merge_request", + "bitbucket.bitbucket_create_pr", + "github.update_pull_request_branch", + "github.request_copilot_review", + "github.create_pull_request", + "github.pull_request_read", + "github.pull_request_review_write" + ], + "hit1": "github.merge_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 321, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 524, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1103, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "kind": "verb_product", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "ranking": [ + "github.actions_run_trigger", + "github.actions_get", + "github.actions_list", + "github.actions_get_job_logs", + "circleci.circleci_trigger_pipeline", + "github.delete_repository", + "github.fork_repository", + "github.star_repository", + "github.unstar_repository", + "github.search_issues" + ], + "hit1": "github.actions_run_trigger", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 831, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "kind": "resource", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "ranking": [ + "github.list_dependabot_alerts", + "github.get_dependabot_alert", + "github.list_code_scanning_alerts", + "github.list_secret_scanning_alerts", + "notion.notion_get_block", + "kubernetes.k8s_get_pod", + "shopify.shopify_get_product", + "shopify.shopify_get_order", + "sentry.sentry_get_issue", + "datadog.datadog_get_monitor" + ], + "hit1": "github.list_dependabot_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 287, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 420, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 739, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "kind": "verb_product", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_issues", + "sentry.sentry_list_projects", + "gitlab.gitlab_list_issues", + "sentry.sentry_list_events", + "linear.linear_list_issues", + "sentry.sentry_get_issue", + "github.list_issues", + "github.search_issues", + "sentry.sentry_update_issue", + "linear.linear_search_issues" + ], + "hit1": "sentry.sentry_list_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 395, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_create_invoice", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "stripe.stripe_create_refund", + "stripe.stripe_create_payment_intent", + "stripe.stripe_list_customers", + "stripe.stripe_list_invoices", + "stripe.stripe_list_subscriptions" + ], + "hit1": "stripe.stripe_create_customer", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "kind": "paraphrase", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_refund", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_list_customers", + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_create_product", + "stripe.stripe_create_price" + ], + "hit1": "stripe.stripe_create_refund", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "kind": "verb_product", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "ranking": [ + "gmail.gmail_send_email", + "twilio.twilio_send_sms", + "puppeteer.puppeteer_click", + "puppeteer.puppeteer_fill", + "gmail.gmail_search_emails", + "gmail.gmail_create_draft", + "gmail.gmail_list_labels", + "discord.discord_send_message", + "puppeteer.puppeteer_screenshot", + "gmail.gmail_get_email" + ], + "hit1": "gmail.gmail_send_email", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 395, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "kind": "verb_product", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "ranking": [ + "gmail.gmail_search_emails", + "mysql.mysql_list_tables", + "gmail.gmail_create_draft", + "gmail.gmail_list_labels", + "mysql.mysql_query", + "gmail.gmail_send_email", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "brave_search.brave_web_search", + "slack.slack_search_messages" + ], + "hit1": "gmail.gmail_search_emails", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 305, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 666, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "kind": "paraphrase", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "ranking": [ + "slack.slack_upload_file", + "google_drive.gdrive_update_file", + "google_drive.gdrive_share_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_get_file", + "aws_s3.s3_put_object", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "google_calendar.gcal_list_calendars", + "git.git_add" + ], + "hit1": "slack.slack_upload_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 555, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 446, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 747, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_create_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_list_events", + "slack.slack_create_channel", + "linear.linear_create_project", + "notion.notion_create_database", + "stripe.stripe_create_customer", + "stripe.stripe_create_invoice", + "stripe.stripe_create_product" + ], + "hit1": "google_calendar.gcal_create_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 461, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 813, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_list_pods", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_apply", + "kubernetes.k8s_delete", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_pod_logs", + "trello.trello_list_cards" + ], + "hit1": "kubernetes.k8s_list_pods", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 741, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_get_pod", + "docker.docker_logs", + "git.git_log", + "datadog.datadog_list_logs", + "elasticsearch.es_get_doc", + "aws_s3.s3_get_object", + "github.actions_get_job_logs", + "slack.slack_get_channel_history", + "gitlab.gitlab_get_file" + ], + "hit1": "kubernetes.k8s_pod_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 423, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 822, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "ranking": [ + "docker.docker_run", + "docker.docker_exec", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_images", + "docker.docker_build", + "docker.docker_ps", + "terraform.tf_plan", + "terraform.tf_apply" + ], + "hit1": "docker.docker_run", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 594, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "kind": "verb_product", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "ranking": [ + "docker.docker_logs", + "git.git_log", + "docker.docker_run", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_exec", + "datadog.datadog_list_logs", + "git.git_show", + "docker.docker_images", + "docker.docker_build" + ], + "hit1": "docker.docker_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "kind": "ambiguous", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "ranking": [ + "playwright.browser_navigate", + "puppeteer.puppeteer_navigate", + "playwright.browser_tab_close", + "fetch.fetch", + "playwright.browser_tab_list", + "playwright.browser_tab_new", + "git.git_commit", + "git.git_add", + "filesystem.list_allowed_directories", + "memory.add_observations" + ], + "hit1": "playwright.browser_navigate", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 293, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 568, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "kind": "ambiguous", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "ranking": [ + "playwright.browser_take_screenshot", + "puppeteer.puppeteer_screenshot", + "playwright.browser_snapshot", + "playwright.browser_press_key", + "playwright.browser_click", + "playwright.browser_evaluate", + "slack.slack_set_channel_topic", + "filesystem.read_file", + "github.get_repository_tree", + "git.git_status" + ], + "hit1": "playwright.browser_take_screenshot", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 326, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_create_merge_request", + "gitlab.gitlab_get_merge_request", + "gitlab.gitlab_list_merge_requests", + "github.merge_pull_request", + "gitlab.gitlab_create_issue", + "gitlab.gitlab_create_pipeline", + "gitlab.gitlab_list_issues", + "gitlab.gitlab_get_issue", + "gitlab.gitlab_list_projects" + ], + "hit1": "gitlab.gitlab_merge_merge_request", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 833, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "kind": "verb_product", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "ranking": [ + "notion.notion_search", + "brave_search.brave_web_search", + "slack.slack_search_messages", + "datadog.datadog_list_logs", + "gmail.gmail_search_emails", + "grafana.grafana_search_dashboards", + "confluence.confluence_search", + "notion.notion_get_page", + "notion.notion_get_block", + "notion.notion_query_database" + ], + "hit1": "notion.notion_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "kind": "verb_product", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "ranking": [ + "notion.notion_get_page", + "notion.notion_create_page", + "notion.notion_update_page", + "confluence.confluence_create_page", + "notion.notion_get_comments", + "notion.notion_create_database", + "notion.create_comment", + "notion.create_users_token", + "notion.create_data_source", + "confluence.confluence_get_page" + ], + "hit1": "notion.notion_get_page", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 376, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 822, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "kind": "verb_product", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "ranking": [ + "notion.notion_query_database", + "notion.notion_create_database", + "notion.notion_create_page", + "prometheus.prom_query", + "prometheus.prom_query_range", + "datadog.datadog_query_metrics", + "postgres.explain_query", + "salesforce.salesforce_query", + "notion.notion_get_page", + "notion.notion_get_block" + ], + "hit1": "notion.notion_query_database", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 279, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 430, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 752, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_list_tickets", + "jira.jira_create_issue", + "zendesk.zendesk_search", + "slack.slack_create_channel", + "linear.linear_create_project", + "notion.notion_create_database", + "stripe.stripe_create_customer" + ], + "hit1": "zendesk.zendesk_create_ticket", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 426, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 791, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_create_deal", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_search_contacts", + "github.create_gist", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.projects_write" + ], + "hit1": "hubspot.hubspot_create_contact", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 838, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "kind": "verb_product", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_create_lead", + "salesforce.salesforce_convert_lead", + "salesforce.salesforce_create_record", + "salesforce.salesforce_update_record", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_list_opportunities", + "salesforce.salesforce_query", + "salesforce.salesforce_get_record", + "slack.slack_create_channel", + "linear.linear_create_project" + ], + "hit1": "salesforce.salesforce_create_lead", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "kind": "identifier", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "postgres.query", + "notion.notion_query_database", + "prometheus.prom_query", + "prometheus.prom_query_range", + "docker.docker_run", + "terraform.tf_plan", + "terraform.tf_apply" + ], + "hit1": "salesforce.salesforce_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "kind": "verb_product", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "ranking": [ + "shopify.shopify_list_orders", + "shopify.shopify_list_products", + "shopify.shopify_list_customers", + "shopify.shopify_get_product", + "shopify.shopify_create_product", + "shopify.shopify_update_product", + "shopify.shopify_get_order", + "shopify.shopify_create_order", + "trello.trello_list_cards", + "git.git_branch" + ], + "hit1": "shopify.shopify_list_orders", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "slack.slack_create_channel", + "linear.linear_create_project", + "notion.notion_create_database", + "stripe.stripe_create_customer" + ], + "hit1": "pagerduty.pagerduty_create_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 773, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "kind": "paraphrase", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_list_oncalls", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "twilio.twilio_make_call", + "filesystem.list_allowed_directories", + "trello.trello_add_comment", + "notion.notion_get_comments" + ], + "hit1": "pagerduty.pagerduty_list_oncalls", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "kind": "verb_product", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "ranking": [ + "datadog.datadog_query_metrics", + "datadog.datadog_list_monitors", + "datadog.datadog_get_monitor", + "datadog.datadog_create_monitor", + "datadog.datadog_list_logs", + "notion.notion_query_database", + "prometheus.prom_query", + "prometheus.prom_query_range", + "postgres.explain_query", + "salesforce.salesforce_query" + ], + "hit1": "datadog.datadog_query_metrics", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 809, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "kind": "verb_product", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "ranking": [ + "datadog.datadog_list_logs", + "git.git_log", + "datadog.datadog_list_monitors", + "datadog.datadog_get_monitor", + "datadog.datadog_create_monitor", + "datadog.datadog_query_metrics", + "docker.docker_logs", + "kubernetes.k8s_pod_logs", + "brave_search.brave_web_search", + "github.actions_get_job_logs" + ], + "hit1": "datadog.datadog_list_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 419, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 863, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_list_buckets", + "trello.trello_list_cards", + "git.git_branch", + "linear.linear_list_teams", + "linear.linear_list_projects", + "jira.jira_list_projects", + "docker.docker_images", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events" + ], + "hit1": "aws_s3.s3_list_buckets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 263, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 490, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "kind": "paraphrase", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_put_object", + "aws_s3.s3_delete_object", + "slack.slack_upload_file", + "aws_s3.s3_get_object", + "slack.slack_add_reaction", + "github.assign_copilot_to_issue", + "slack.slack_reply_to_thread", + "elasticsearch.es_search", + "okta.okta_get_user", + "okta.okta_create_user" + ], + "hit1": "aws_s3.s3_put_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 423, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 788, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns" + ], + "hit1": "cloudflare.cf_purge_cache", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "salesforce.salesforce_create_record", + "cloudflare.cf_list_zones", + "cloudflare.cf_purge_cache", + "salesforce.salesforce_update_record", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_get_record", + "git.git_commit", + "slack.slack_create_channel" + ], + "hit1": "cloudflare.cf_create_dns", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 732, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_deployments", + "kubernetes.k8s_list_deployments", + "vercel.vercel_list_projects", + "vercel.vercel_get_deployment", + "vercel.vercel_create_project", + "trello.trello_list_cards", + "git.git_branch", + "linear.linear_list_teams", + "linear.linear_list_projects", + "jira.jira_list_projects" + ], + "hit1": "vercel.vercel_list_deployments", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 173, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 527, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "ranking": [ + "supabase.supabase_insert", + "mongodb.mongo_insert", + "supabase.supabase_invoke_fn", + "playwright.browser_type", + "supabase.supabase_list_tables", + "supabase.supabase_select", + "supabase.supabase_update" + ], + "hit1": "supabase.supabase_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 357, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "kind": "paraphrase", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "ranking": [ + "figma.figma_get_images", + "figma.figma_get_file_nodes", + "docker.docker_images", + "figma.figma_get_file", + "figma.figma_get_comments", + "figma.figma_post_comment", + "elasticsearch.es_get_doc", + "aws_s3.s3_get_object", + "memory.delete_observations", + "docker.docker_logs" + ], + "hit1": "figma.figma_get_images", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "kind": "verb_product", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "ranking": [ + "twilio.twilio_send_sms", + "twilio.twilio_list_messages", + "gmail.gmail_send_email", + "puppeteer.puppeteer_click", + "puppeteer.puppeteer_fill", + "twilio.twilio_make_call", + "discord.discord_send_message", + "puppeteer.puppeteer_screenshot", + "jira.jira_search_issues", + "puppeteer.puppeteer_navigate" + ], + "hit1": "twilio.twilio_send_sms", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "kind": "verb_product", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "ranking": [ + "okta.okta_create_user", + "okta.okta_get_user", + "okta.okta_deactivate_user", + "linear.create_user", + "okta.okta_list_users", + "asana.asana_create_task", + "asana.asana_create_project", + "github.create_issue", + "slack.slack_get_user_profile", + "linear.get_user" + ], + "hit1": "okta.okta_create_user", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 743, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query", + "prometheus.prom_query_range", + "prometheus.prom_list_metrics", + "notion.notion_query_database", + "datadog.datadog_query_metrics", + "postgres.explain_query", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "postgres.query" + ], + "hit1": "prometheus.prom_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "kind": "verb_product", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "ranking": [ + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "grafana.grafana_create_annotation", + "brave_search.brave_web_search", + "slack.slack_search_messages", + "datadog.datadog_list_logs", + "gmail.gmail_search_emails", + "confluence.confluence_search", + "github.search_issues", + "github.search_users" + ], + "hit1": "grafana.grafana_search_dashboards", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 735, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "kind": "verb_product", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "ranking": [ + "circleci.circleci_trigger_pipeline", + "circleci.circleci_get_pipeline", + "circleci.circleci_list_pipelines", + "gitlab.gitlab_create_pipeline", + "github.actions_run_trigger", + "gitlab.create_pipeline", + "gitlab.get_pipeline", + "gitlab.update_pipeline", + "gitlab.delete_pipeline" + ], + "hit1": "circleci.circleci_trigger_pipeline", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "kind": "identifier", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "ranking": [ + "terraform.tf_plan", + "terraform.tf_apply", + "postgres.explain_query", + "terraform.tf_state_list", + "docker.docker_run", + "salesforce.salesforce_query", + "github.create_check_run", + "github.get_check_run", + "github.update_check_run", + "github.delete_check_run" + ], + "hit1": "terraform.tf_plan", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "kind": "verb_product", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "ranking": [ + "asana.asana_create_task", + "asana.asana_get_task", + "asana.asana_update_task", + "asana.asana_create_project", + "clickup.clickup_create_task", + "asana.asana_list_tasks", + "okta.okta_create_user", + "clickup.clickup_get_task", + "clickup.clickup_update_task", + "github.create_issue" + ], + "hit1": "asana.asana_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "ranking": [ + "trello.trello_create_card", + "trello.trello_add_comment", + "trello.trello_create_board", + "trello.trello_list_cards", + "slack.slack_create_channel", + "linear.linear_create_project", + "notion.notion_create_database", + "stripe.stripe_create_customer", + "stripe.stripe_create_invoice", + "stripe.stripe_create_product" + ], + "hit1": "trello.trello_create_card", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_create_task", + "clickup.clickup_get_task", + "clickup.clickup_update_task", + "clickup.clickup_list_tasks", + "asana.asana_create_task", + "asana.asana_get_task", + "asana.asana_update_task", + "puppeteer.puppeteer_click", + "playwright.browser_click", + "slack.slack_create_channel" + ], + "hit1": "clickup.clickup_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 415, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 745, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "kind": "verb_product", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "ranking": [ + "confluence.confluence_create_page", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "notion.notion_create_page", + "confluence.confluence_search", + "notion.notion_get_page", + "notion.notion_update_page", + "notion.notion_get_comments", + "playwright.browser_press_key", + "playwright.browser_click" + ], + "hit1": "confluence.confluence_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 692, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "kind": "verb_product", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "discord.discord_add_reaction", + "discord.discord_get_channel", + "discord.discord_list_channels", + "discord.discord_list_members", + "gmail.gmail_send_email", + "twilio.twilio_send_sms", + "slack.slack_post_message", + "gmail.gmail_get_email", + "gmail.gmail_trash_email" + ], + "hit1": "discord.discord_send_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_search", + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "brave_search.brave_web_search", + "slack.slack_search_messages", + "datadog.datadog_list_logs", + "gmail.gmail_search_emails", + "grafana.grafana_search_dashboards", + "confluence.confluence_search", + "github.search_issues" + ], + "hit1": "elasticsearch.es_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 766, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "redis.redis_set", + "jira.jira_get_issue", + "redis.redis_delete", + "playwright.browser_press_key", + "redis.redis_keys", + "github.get_dependabot_alert", + "notion.notion_get_block", + "kubernetes.k8s_get_pod", + "shopify.shopify_get_product" + ], + "hit1": "redis.redis_get", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 285, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 610, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_find", + "mongodb.mongo_insert", + "mongodb.mongo_update", + "mongodb.mongo_delete", + "mongodb.mongo_list_collections" + ], + "hit1": "mongodb.mongo_find", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "kind": "verb_product", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "ranking": [ + "sqlite.sqlite_list_tables", + "sqlite.sqlite_describe_table", + "sqlite.sqlite_query", + "mysql.mysql_list_tables", + "supabase.supabase_list_tables", + "postgres.list_tables", + "trello.trello_list_cards", + "git.git_branch", + "linear.linear_list_teams", + "linear.linear_list_projects" + ], + "hit1": "sqlite.sqlite_list_tables", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 156, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 243, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 494, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "kind": "verb_product", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "ranking": [ + "sqlite.sqlite_describe_table", + "postgres.describe_table", + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update" + ], + "hit1": "sqlite.sqlite_describe_table", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "bitbucket.bitbucket_create_issue", + "gitlab.gitlab_create_issue", + "github.create_milestone", + "github.create_hook", + "github.create_environment", + "github.create_package", + "github.create_artifact", + "github.create_runner", + "linear.linear_create_issue", + "github.create_check_run" + ], + "hit1": "bitbucket.bitbucket_create_issue", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 39, + "k3_calls": 13, + "k3_tokens": 3683, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 8, + "k5_tokens": 3724, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3648, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q070", + "query": "createIssue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "ranking": [ + "bitbucket.bitbucket_create_issue", + "gitlab.gitlab_create_issue", + "linear.linear_create_issue", + "github.create_issue", + "jira.jira_create_issue", + "linear.linear_update_issue", + "jira.jira_assign_issue", + "sentry.sentry_get_issue", + "github.issue_write", + "gitlab.gitlab_get_issue" + ], + "hit1": "bitbucket.bitbucket_create_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5583663834639362, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 332, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 579, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1039, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "kind": "identifier", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "ranking": [ + "filesystem.create_directory", + "filesystem.get_file_info", + "github.get_file_contents", + "trello.trello_list_cards", + "git.git_branch", + "linear.linear_list_teams", + "linear.linear_list_projects", + "jira.jira_list_projects", + "docker.docker_images", + "kubernetes.k8s_list_deployments" + ], + "hit1": "filesystem.create_directory", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 90, + "k3_calls": 30, + "k3_tokens": 7901, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7685, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7523, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "kind": "paraphrase", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 116, + "k3_calls": 39, + "k3_tokens": 9799, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 24, + "k5_tokens": 9727, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9475, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "kind": "paraphrase", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "ranking": [ + "time.convert_time", + "time.get_current_time", + "filesystem.list_allowed_directories", + "playwright.browser_wait_for", + "elasticsearch.es_index_doc", + "github.create_milestone", + "github.get_milestone", + "github.list_milestones", + "github.update_milestone", + "github.delete_milestone" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8597186998521972, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "kind": "verb_product", + "gold": [ + "time.convert_time" + ], + "alts": [], + "ranking": [ + "time.convert_time", + "salesforce.salesforce_convert_lead", + "time.get_current_time", + "git.git_diff", + "playwright.browser_wait_for", + "memory.create_relations" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 636, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "kind": "verb_product", + "gold": [ + "github.star_repository" + ], + "alts": [], + "ranking": [ + "github.star_repository", + "github.delete_repository", + "github.fork_repository", + "github.unstar_repository", + "github.search_issues", + "github.get_discussion", + "github.create_gist", + "github.update_gist", + "github.projects_list", + "github.projects_get" + ], + "hit1": "github.star_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 394, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 772, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "kind": "resource", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "ranking": [ + "github.fork_repository", + "github.delete_repository", + "github.star_repository", + "github.unstar_repository", + "github.create_repository", + "github.list_dependabot_alerts", + "github.list_branches", + "github.list_releases", + "github.create_branch", + "github.delete_file" + ], + "hit1": "github.fork_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 822, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "kind": "verb_product", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "ranking": [ + "github.get_latest_release", + "github.get_release_by_tag", + "github.get_repository_tree", + "gitlab.get_release", + "github.get_teams", + "github.get_discussion", + "github.projects_get", + "github.get_label", + "github.get_tag", + "github.get_me" + ], + "hit1": "github.get_latest_release", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 291, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 383, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "kind": "resource", + "gold": [ + "github.create_gist" + ], + "alts": [], + "ranking": [ + "github.create_gist", + "github.update_gist", + "github.get_gist", + "slack.slack_create_channel", + "linear.linear_create_project", + "notion.notion_create_database", + "stripe.stripe_create_customer", + "stripe.stripe_create_invoice", + "stripe.stripe_create_product", + "stripe.stripe_create_price" + ], + "hit1": "github.create_gist", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 383, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "kind": "verb_product", + "gold": [ + "github.search_code" + ], + "alts": [], + "ranking": [ + "github.search_code", + "github.get_code_scanning_alert", + "github.list_code_scanning_alerts", + "github.search_issues", + "github.search_users", + "github.search_orgs", + "github.search_commits", + "github.search_pull_requests", + "hubspot.hubspot_search_contacts", + "github.search_repositories" + ], + "hit1": "github.search_code", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 419, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "kind": "verb_product", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "ranking": [ + "github.list_notifications", + "github.mark_all_notifications_read", + "github.projects_list", + "github.actions_list", + "github.list_tags", + "github.get_team_members", + "github.list_branches", + "github.list_releases", + "github.list_repository_collaborators", + "github.list_issues" + ], + "hit1": "github.list_notifications", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 326, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "kind": "verb_product", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "ranking": [ + "github.assign_copilot_to_issue", + "linear.linear_assign_issue", + "jira.jira_assign_issue", + "aws_s3.s3_put_object", + "linear.linear_create_comment", + "jira.jira_update_issue", + "jira.jira_add_comment", + "jira.jira_add_worklog", + "github.add_issue_comment", + "jira.jira_transition_issue" + ], + "hit1": "github.assign_copilot_to_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 421, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 819, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "kind": "resource", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "ranking": [ + "github.list_secret_scanning_alerts", + "github.get_secret_scanning_alert", + "github.list_code_scanning_alerts", + "github.get_code_scanning_alert", + "github.list_dependabot_alerts", + "github.get_dependabot_alert", + "notion.notion_get_block", + "kubernetes.k8s_get_pod", + "shopify.shopify_get_product", + "shopify.shopify_get_order" + ], + "hit1": "github.list_secret_scanning_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 804, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "ranking": [ + "github.create_or_update_file", + "github.projects_write", + "github.label_write", + "github.update_issue_comment", + "github.issue_write", + "github.get_file_contents", + "filesystem.write_file", + "github.pull_request_review_write", + "github.update_gist", + "github.list_repository_collaborators" + ], + "hit1": "github.create_or_update_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 369, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 610, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1088, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "kind": "resource", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "ranking": [ + "github.list_repository_collaborators", + "trello.trello_list_cards", + "pagerduty.pagerduty_list_oncalls", + "github.list_commits", + "trello.trello_add_comment", + "notion.notion_get_comments", + "playwright.browser_press_key", + "figma.figma_get_comments", + "figma.figma_post_comment", + "github.update_issue_comment" + ], + "hit1": "github.list_repository_collaborators", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 411, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "kind": "verb_product", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "ranking": [ + "slack.slack_reply_to_thread", + "slack.slack_get_thread_replies", + "github.get_notification_details", + "slack.create_reminder", + "slack.get_reminder", + "slack.list_reminders", + "slack.update_reminder", + "slack.delete_reminder", + "slack.create_canvase", + "slack.get_canvase" + ], + "hit1": "slack.slack_reply_to_thread", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 767, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "kind": "verb_product", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "ranking": [ + "slack.slack_add_reaction", + "discord.discord_add_reaction", + "gmail.gmail_modify_email", + "playwright.browser_click", + "github.assign_copilot_to_issue", + "linear.linear_create_comment", + "jira.jira_add_comment", + "jira.jira_add_worklog", + "trello.trello_add_comment", + "aws_s3.s3_delete_object" + ], + "hit1": "slack.slack_add_reaction", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 461, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 827, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "kind": "verb_product", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "ranking": [ + "slack.slack_search_messages", + "gmail.gmail_search_emails", + "slack.slack_get_channel_history", + "twilio.twilio_list_messages", + "brave_search.brave_web_search", + "datadog.datadog_list_logs", + "grafana.grafana_search_dashboards", + "confluence.confluence_search", + "github.search_issues", + "github.search_users" + ], + "hit1": "slack.slack_search_messages", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "kind": "verb_product", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "ranking": [ + "slack.slack_create_channel", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_post_message", + "slack.slack_get_channel_history", + "slack.slack_set_channel_topic", + "discord.discord_get_channel", + "slack.create_reminder", + "slack.create_canvase", + "slack.create_bookmark" + ], + "hit1": "slack.slack_create_channel", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 773, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "kind": "verb_product", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_transition_issue", + "jira.jira_assign_issue", + "jira.jira_update_issue", + "jira.jira_get_issue", + "jira.jira_add_comment", + "jira.jira_add_worklog", + "jira.jira_create_issue", + "linear.linear_update_issue", + "sentry.sentry_get_issue", + "bitbucket.bitbucket_create_issue" + ], + "hit1": "jira.jira_transition_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 222, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 822, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "kind": "identifier", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "ranking": [ + "jira.jira_search_issues", + "puppeteer.puppeteer_screenshot", + "brave_search.brave_web_search", + "puppeteer.puppeteer_navigate", + "puppeteer.puppeteer_click", + "puppeteer.puppeteer_fill", + "puppeteer.puppeteer_evaluate", + "slack.slack_search_messages", + "datadog.datadog_list_logs", + "gmail.gmail_search_emails" + ], + "hit1": "jira.jira_search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 734, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "kind": "verb_product", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "ranking": [ + "jira.jira_add_worklog", + "jira.jira_add_comment", + "linear.linear_create_comment", + "discord.discord_add_reaction", + "github.add_issue_comment", + "slack.slack_add_reaction", + "git.git_add", + "jira.jira_transition_issue", + "github.add_comment_to_pending_review", + "memory.add_observations" + ], + "hit1": "jira.jira_add_worklog", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 440, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 874, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "kind": "verb_product", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "ranking": [ + "linear.linear_list_teams", + "linear.linear_list_projects", + "linear.list_cycles", + "linear.list_labels", + "linear.list_users", + "linear.list_comments", + "linear.list_attachments", + "linear.linear_list_issues", + "github.get_teams", + "linear.linear_update_issue" + ], + "hit1": "linear.linear_list_teams", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 318, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "ranking": [ + "linear.linear_create_comment", + "github.update_issue_comment", + "trello.trello_add_comment", + "linear.linear_update_issue", + "figma.figma_post_comment", + "linear.create_comment", + "linear.get_comment", + "linear.update_comment", + "linear.delete_comment", + "jira.jira_add_comment" + ], + "hit1": "linear.linear_create_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 264, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 451, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "kind": "paraphrase", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "ranking": [ + "filesystem.read_text_file", + "gmail.gmail_trash_email", + "filesystem.read_file", + "filesystem.read_media_file", + "filesystem.move_file", + "brave_search.brave_local_search", + "figma.figma_get_file", + "filesystem.write_file", + "slack.slack_upload_file", + "google_drive.gdrive_update_file" + ], + "hit1": "filesystem.read_text_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.2, + "ndcg@10": 0.38685280723454163, + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 434, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 712, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "kind": "paraphrase", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "ranking": [ + "filesystem.search_files", + "google_drive.gdrive_search", + "linear.linear_search_issues", + "redis.redis_keys", + "github.search_repositories", + "slack.list_files", + "brave_search.brave_web_search", + "slack.slack_search_messages", + "datadog.datadog_list_logs", + "gmail.gmail_search_emails" + ], + "hit1": "filesystem.search_files", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 741, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "kind": "paraphrase", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "ranking": [ + "filesystem.edit_file", + "filesystem.write_file", + "filesystem.list_directory_with_sizes", + "twilio.twilio_make_call", + "git.git_create_branch", + "playwright.browser_tab_new", + "puppeteer.puppeteer_screenshot", + "slack.create_file", + "slack.get_file", + "slack.update_file" + ], + "hit1": "filesystem.edit_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 419, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "ranking": [ + "memory.read_graph", + "memory.delete_relations", + "memory.create_entities", + "memory.create_relations", + "memory.add_observations", + "memory.delete_entities", + "memory.open_nodes", + "memory.search_nodes", + "filesystem.read_text_file", + "filesystem.read_file" + ], + "hit1": "memory.read_graph", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 259, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 561, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "kind": "verb_product", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "figma.figma_get_file_nodes", + "brave_search.brave_web_search", + "slack.slack_search_messages", + "datadog.datadog_list_logs", + "gmail.gmail_search_emails", + "grafana.grafana_search_dashboards", + "confluence.confluence_search", + "memory.open_nodes", + "github.search_issues" + ], + "hit1": "memory.search_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 241, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "kind": "verb_product", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "ranking": [ + "git.git_diff_unstaged", + "git.git_diff", + "git.git_show", + "git.git_diff_staged", + "git.git_log", + "git.git_status", + "github.list_tags", + "github.get_tag", + "git.git_checkout", + "git.git_branch" + ], + "hit1": "git.git_diff_unstaged", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 313, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 628, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "kind": "identifier", + "gold": [ + "git.git_add" + ], + "alts": [], + "ranking": [ + "filesystem.list_directory_with_sizes", + "github.add_issue_comment", + "puppeteer.puppeteer_screenshot", + "slack.list_files", + "git.git_add", + "jira.jira_search_issues", + "puppeteer.puppeteer_navigate", + "puppeteer.puppeteer_click", + "puppeteer.puppeteer_fill", + "puppeteer.puppeteer_evaluate" + ], + "hit1": "filesystem.list_directory_with_sizes", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.2, + "ndcg@10": 0.38685280723454163, + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 495, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 710, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "kind": "verb_product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "ranking": [ + "shopify.shopify_create_product", + "shopify.shopify_get_product", + "shopify.shopify_update_product", + "stripe.stripe_create_product", + "shopify.shopify_create_order", + "shopify.shopify_list_products", + "shopify.shopify_list_orders", + "shopify.shopify_get_order", + "shopify.shopify_list_customers", + "slack.slack_create_channel" + ], + "hit1": "shopify.shopify_create_product", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "kind": "verb_product", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "ranking": [ + "stripe.stripe_cancel_subscription", + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_list_customers", + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "stripe.stripe_list_subscriptions", + "stripe.stripe_create_refund" + ], + "hit1": "stripe.stripe_cancel_subscription", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "kind": "resource", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent", + "slack.slack_create_channel", + "linear.linear_create_project", + "notion.notion_create_database", + "stripe.stripe_create_customer", + "stripe.stripe_create_invoice", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "stripe.stripe_create_refund" + ], + "hit1": "stripe.stripe_create_payment_intent", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "kind": "verb_product", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "ranking": [ + "datadog.datadog_create_monitor", + "datadog.datadog_get_monitor", + "datadog.datadog_list_monitors", + "datadog.datadog_list_logs", + "datadog.datadog_query_metrics", + "slack.slack_create_channel", + "linear.linear_create_project", + "notion.notion_create_database", + "stripe.stripe_create_customer", + "stripe.stripe_create_invoice" + ], + "hit1": "datadog.datadog_create_monitor", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 815, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "kind": "paraphrase", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "ranking": [ + "sentry.sentry_get_issue", + "sentry.sentry_list_events", + "sentry.sentry_update_issue", + "pagerduty.pagerduty_resolve_incident", + "sentry.sentry_list_projects", + "sentry.sentry_list_issues", + "linear.linear_update_issue", + "jira.jira_assign_issue", + "bitbucket.bitbucket_create_issue", + "gitlab.gitlab_create_issue" + ], + "hit1": "sentry.sentry_get_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 826, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_scale_deployment", + "vercel.vercel_get_deployment", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_apply", + "kubernetes.k8s_delete", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_pod_logs" + ], + "hit1": "kubernetes.k8s_scale_deployment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 411, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_apply", + "terraform.tf_apply", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_delete", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_pod_logs" + ], + "hit1": "kubernetes.k8s_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "ranking": [ + "docker.docker_stop", + "docker.docker_run", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_exec", + "docker.docker_images", + "docker.docker_build", + "docker.docker_ps" + ], + "hit1": "docker.docker_stop", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 581, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "kind": "verb_product", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "ranking": [ + "docker.docker_build", + "docker.docker_run", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_images", + "docker.docker_ps", + "docker.docker_logs", + "docker.docker_exec", + "filesystem.read_media_file" + ], + "hit1": "docker.docker_build", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "kind": "verb_product", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "ranking": [ + "puppeteer.puppeteer_click", + "playwright.browser_click", + "playwright.browser_type", + "playwright.browser_take_screenshot", + "clickup.clickup_create_task", + "clickup.clickup_list_tasks", + "clickup.clickup_get_task", + "clickup.clickup_update_task", + "aws_s3.s3_list_objects", + "aws_s3.s3_delete_object" + ], + "hit1": "puppeteer.puppeteer_click", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "kind": "paraphrase", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "ranking": [ + "playwright.browser_type", + "playwright.browser_tab_close", + "puppeteer.puppeteer_fill", + "playwright.browser_tab_list", + "playwright.browser_tab_new", + "mongodb.mongo_insert", + "supabase.supabase_insert", + "playwright.browser_navigate", + "linear.linear_search_issues", + "filesystem.read_text_file" + ], + "hit1": "playwright.browser_type", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 636, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_list_projects", + "github.projects_list", + "gitlab.gitlab_list_issues", + "linear.linear_list_projects", + "jira.jira_list_projects", + "sentry.sentry_list_projects", + "vercel.vercel_list_projects", + "gitlab.gitlab_list_merge_requests", + "gitlab.gitlab_create_issue", + "gitlab.gitlab_get_issue" + ], + "hit1": "gitlab.gitlab_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 209, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 289, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 615, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "kind": "verb_product", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "ranking": [ + "bitbucket.bitbucket_create_pr", + "bitbucket.bitbucket_list_prs", + "bitbucket.bitbucket_create_issue", + "github.merge_pull_request", + "github.request_copilot_review", + "github.pull_request_review_write", + "github.create_pull_request", + "github.pull_request_read", + "bitbucket.bitbucket_list_repos", + "github.update_pull_request_branch" + ], + "hit1": "bitbucket.bitbucket_create_pr", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 519, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1050, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "kind": "verb_product", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "ranking": [ + "gmail.gmail_create_draft", + "gmail.gmail_search_emails", + "gmail.gmail_list_labels", + "gmail.gmail_send_email", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "gmail.gmail_modify_email", + "slack.slack_create_channel", + "linear.linear_create_project", + "notion.notion_create_database" + ], + "hit1": "gmail.gmail_create_draft", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "kind": "verb_product", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_share_file", + "google_drive.gdrive_update_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "google_calendar.gcal_list_calendars", + "google_calendar.gcal_list_events", + "google_calendar.gcal_create_event", + "google_calendar.gcal_update_event" + ], + "hit1": "google_drive.gdrive_share_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 306, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 808, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "kind": "verb_product", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_list_events", + "google_calendar.gcal_create_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_delete_event", + "kubernetes.k8s_get_events", + "google_calendar.gcal_list_calendars", + "sentry.sentry_list_events", + "google_drive.gdrive_list_folder", + "google_drive.gdrive_update_file", + "google_drive.gdrive_share_file" + ], + "hit1": "google_calendar.gcal_list_events", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 317, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 441, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 784, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_list_events", + "kubernetes.k8s_delete", + "salesforce.salesforce_delete_record", + "redis.redis_delete", + "aws_s3.s3_delete_object", + "github.delete_repository", + "github.delete_milestone" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 477, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 787, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_projects", + "vercel.vercel_list_deployments", + "linear.linear_list_projects", + "jira.jira_list_projects", + "sentry.sentry_list_projects", + "github.projects_list", + "gitlab.gitlab_list_projects", + "vercel.vercel_get_deployment", + "vercel.vercel_create_project", + "trello.trello_list_cards" + ], + "hit1": "vercel.vercel_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 248, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 559, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "ranking": [ + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update", + "supabase.supabase_invoke_fn", + "supabase.supabase_list_tables", + "elasticsearch.es_get_doc", + "aws_s3.s3_get_object", + "memory.delete_observations", + "docker.docker_logs", + "kubernetes.k8s_pod_logs" + ], + "hit1": "supabase.supabase_select", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "kind": "verb_product", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "figma.figma_get_file", + "slack.slack_post_message", + "figma.figma_get_comments", + "figma.figma_get_images", + "figma.figma_get_file_nodes", + "trello.trello_add_comment", + "linear.create_comment", + "linear.get_comment", + "linear.update_comment" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 720, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "kind": "verb_product", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "ranking": [ + "okta.okta_list_users", + "slack.slack_list_users", + "notion.notion_list_users", + "linear.list_users", + "okta.okta_get_user", + "okta.okta_create_user", + "okta.okta_deactivate_user", + "slack.slack_get_users", + "notion.list_users_tokens", + "pagerduty.pagerduty_list_oncalls" + ], + "hit1": "okta.okta_list_users", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 174, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 618, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query_range", + "prometheus.prom_query", + "prometheus.prom_list_metrics", + "notion.notion_query_database", + "datadog.datadog_query_metrics", + "postgres.explain_query", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "postgres.query" + ], + "hit1": "prometheus.prom_query_range", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "kind": "verb_product", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "ranking": [ + "grafana.grafana_create_annotation", + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "slack.slack_create_channel", + "linear.linear_create_project", + "notion.notion_create_database", + "stripe.stripe_create_customer", + "stripe.stripe_create_invoice", + "stripe.stripe_create_product", + "stripe.stripe_create_price" + ], + "hit1": "grafana.grafana_create_annotation", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 372, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 722, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "kind": "identifier", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "ranking": [ + "terraform.tf_apply", + "kubernetes.k8s_apply", + "terraform.tf_plan", + "terraform.tf_state_list" + ], + "hit1": "terraform.tf_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "kind": "verb_product", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "ranking": [ + "asana.asana_list_tasks", + "clickup.clickup_list_tasks", + "asana.asana_create_task", + "asana.asana_get_task", + "asana.asana_update_task", + "asana.asana_create_project", + "trello.trello_list_cards", + "git.git_branch", + "linear.linear_list_teams", + "linear.linear_list_projects" + ], + "hit1": "asana.asana_list_tasks", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 659, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "ranking": [ + "trello.trello_add_comment", + "trello.trello_create_card", + "figma.figma_post_comment", + "trello.trello_list_cards", + "github.update_issue_comment", + "trello.trello_create_board", + "linear.create_comment", + "linear.get_comment", + "linear.update_comment", + "linear.delete_comment" + ], + "hit1": "trello.trello_add_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 438, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_update_task", + "clickup.clickup_create_task", + "clickup.clickup_get_task", + "clickup.clickup_list_tasks", + "asana.asana_update_task", + "asana.asana_create_task", + "asana.asana_get_task", + "puppeteer.puppeteer_click", + "playwright.browser_click", + "linear.linear_update_issue" + ], + "hit1": "clickup.clickup_update_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 787, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "kind": "verb_product", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "ranking": [ + "confluence.confluence_search", + "confluence.confluence_get_page", + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "brave_search.brave_web_search", + "slack.slack_search_messages", + "datadog.datadog_list_logs", + "gmail.gmail_search_emails", + "grafana.grafana_search_dashboards", + "github.search_issues" + ], + "hit1": "confluence.confluence_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 785, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_deal", + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_search_contacts", + "github.create_gist", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.projects_write" + ], + "hit1": "hubspot.hubspot_create_deal", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 838, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_search", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_create_ticket", + "trello.trello_list_cards", + "git.git_branch", + "linear.linear_list_teams", + "linear.linear_list_projects", + "jira.jira_list_projects" + ], + "hit1": "zendesk.zendesk_list_tickets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 603, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "kind": "verb_product", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "ranking": [ + "discord.discord_list_channels", + "discord.discord_list_members", + "slack.slack_list_channels", + "discord.discord_get_channel", + "discord.discord_send_message", + "discord.discord_add_reaction", + "trello.trello_list_cards", + "git.git_branch", + "linear.linear_list_teams", + "linear.linear_list_projects" + ], + "hit1": "discord.discord_list_channels", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 590, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "ranking": [ + "redis.redis_set", + "redis.redis_get", + "redis.redis_delete", + "slack.slack_set_channel_topic", + "jira.jira_get_issue", + "playwright.browser_press_key", + "redis.redis_keys" + ], + "hit1": "redis.redis_set", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 307, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 564, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert", + "mongodb.mongo_update", + "mongodb.mongo_delete", + "mongodb.mongo_find", + "mongodb.mongo_list_collections", + "supabase.supabase_insert" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "kind": "verb_product", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "ranking": [ + "mysql.mysql_query", + "postgres.query", + "sqlite.sqlite_query", + "mysql.mysql_list_tables", + "postgres.explain_query", + "salesforce.salesforce_query", + "notion.notion_query_database", + "prometheus.prom_query", + "prometheus.prom_query_range", + "docker.docker_run" + ], + "hit1": "mysql.mysql_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 670, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_index_doc", + "elasticsearch.es_search", + "elasticsearch.es_get_doc", + "github.create_milestone", + "github.get_milestone", + "github.list_milestones", + "github.update_milestone", + "github.delete_milestone", + "github.create_hook", + "github.get_hook" + ], + "hit1": "elasticsearch.es_index_doc", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_put_object", + "okta.okta_get_user", + "asana.asana_get_task", + "elasticsearch.es_search", + "okta.okta_create_user", + "okta.okta_deactivate_user", + "asana.asana_create_task", + "asana.asana_update_task" + ], + "hit1": "aws_s3.s3_get_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 723, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones", + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_dns", + "cloudflare.cf_create_dns", + "trello.trello_list_cards", + "git.git_branch", + "linear.linear_list_teams", + "linear.linear_list_projects", + "jira.jira_list_projects", + "docker.docker_images" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 361, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 554, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_create_customer", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "stripe.stripe_create_refund", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_customer", + "stripe.stripe_list_customers", + "stripe.stripe_list_invoices", + "stripe.stripe_list_subscriptions" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "kind": "verb_product", + "gold": [ + "github.actions_list" + ], + "alts": [], + "ranking": [ + "github.actions_list", + "github.actions_run_trigger", + "github.actions_get", + "github.actions_get_job_logs", + "github.projects_list", + "github.list_tags", + "github.get_team_members", + "github.list_branches", + "github.list_releases", + "github.list_repository_collaborators" + ], + "hit1": "github.actions_list", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 453, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 812, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "kind": "verb_product", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "ranking": [ + "github.actions_get_job_logs", + "github.actions_get", + "github.actions_list", + "github.actions_run_trigger", + "gitlab.get_job", + "kubernetes.k8s_pod_logs", + "github.get_discussion", + "github.projects_get", + "github.get_label", + "git.git_log" + ], + "hit1": "github.actions_get_job_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 442, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 839, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "kind": "verb_product", + "gold": [ + "github.create_repository" + ], + "alts": [], + "ranking": [ + "github.create_repository", + "github.create_branch", + "github.delete_repository", + "github.fork_repository", + "github.star_repository", + "github.unstar_repository", + "github.create_issue", + "github.list_tags", + "github.get_tag", + "github.create_or_update_file" + ], + "hit1": "github.create_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 920, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "kind": "verb_product", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "ranking": [ + "git.git_create_branch", + "github.create_branch", + "github.update_pull_request_branch", + "github.list_commits", + "github.list_tags", + "github.get_tag", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "github.get_repository_tree" + ], + "hit1": "git.git_create_branch", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.14285714285714285, + "ndcg@10": 0.3333333333333333, + "hit_rank": 7, + "k3_calls": 3, + "k3_tokens": 766, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 852, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 832, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "kind": "identifier", + "gold": [ + "git.git_log" + ], + "alts": [], + "ranking": [ + "git.git_show", + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_log", + "git.git_diff", + "git.git_status", + "github.list_tags", + "github.get_tag", + "git.git_checkout", + "git.git_branch" + ], + "hit1": "git.git_show", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 382, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 311, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 626, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "kind": "verb_product", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "ranking": [ + "postgres.list_schemas", + "trello.trello_list_cards", + "git.git_branch", + "linear.linear_list_teams", + "linear.linear_list_projects", + "jira.jira_list_projects", + "docker.docker_images", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events" + ], + "hit1": "postgres.list_schemas", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 258, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 485, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "kind": "verb_product", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "ranking": [ + "memory.open_nodes", + "memory.search_nodes", + "memory.read_graph", + "memory.create_entities", + "memory.delete_relations", + "memory.create_relations", + "memory.add_observations", + "memory.delete_entities", + "playwright.browser_tab_list", + "playwright.browser_tab_new" + ], + "hit1": "memory.open_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 260, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 505, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "kind": "paraphrase", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "ranking": [ + "filesystem.read_text_file", + "filesystem.read_file", + "filesystem.read_media_file", + "google_drive.gdrive_get_file", + "filesystem.get_file_info", + "gitlab.gitlab_get_file", + "figma.figma_get_file_nodes", + "figma.figma_get_images", + "github.get_file_contents", + "figma.figma_get_file" + ], + "hit1": "filesystem.read_text_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.2, + "ndcg@10": 0.38685280723454163, + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 447, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 750, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "kind": "paraphrase", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "ranking": [ + "filesystem.read_text_file", + "filesystem.write_file", + "filesystem.read_file", + "filesystem.read_media_file", + "git.git_create_branch", + "playwright.browser_tab_new", + "filesystem.edit_file", + "github.issue_write", + "github.create_repository", + "jira.jira_transition_issue" + ], + "hit1": "filesystem.read_text_file", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 376, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 803, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "kind": "verb_product", + "gold": [ + "github.search_issues" + ], + "alts": [], + "ranking": [ + "github.search_issues", + "github.search_users", + "github.search_orgs", + "linear.linear_search_issues", + "jira.jira_search_issues", + "github.list_issues", + "github.search_commits", + "github.search_pull_requests", + "gitlab.gitlab_list_issues", + "github.search_code" + ], + "hit1": "github.search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 683, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "kind": "paraphrase", + "gold": [ + "github.get_me" + ], + "alts": [], + "ranking": [ + "github.get_teams", + "github.list_notifications", + "github.get_me", + "github.get_latest_release", + "github.get_repository_tree", + "github.list_gists", + "github.list_starred_repositories", + "github.get_discussion", + "github.projects_get", + "github.get_label" + ], + "hit1": "github.get_teams", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "cost": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.9, + "p95_calls": 2.0, + "success_within_1": 0.9266666666666666, + "success_within_2": 0.9666666666666667, + "success_within_3": 0.9733333333333334, + "mean_tokens": 466.9066666666667, + "p95_tokens": 441.14999999999975, + "total_tokens": 70036, + "tokens_per_successful_query": 466.9066666666667, + "total_tokens_successful": 70036 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.52, + "p95_calls": 1.0, + "success_within_1": 0.96, + "success_within_2": 0.9733333333333334, + "success_within_3": 0.9733333333333334, + "mean_tokens": 597.2933333333333, + "p95_tokens": 598.2999999999995, + "total_tokens": 89594, + "tokens_per_successful_query": 597.2933333333333, + "total_tokens_successful": 89594 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.24, + "p95_calls": 1.0, + "success_within_1": 0.9733333333333334, + "success_within_2": 0.9733333333333334, + "success_within_3": 0.9733333333333334, + "mean_tokens": 932.4666666666667, + "p95_tokens": 1045.0499999999997, + "total_tokens": 139870, + "tokens_per_successful_query": 932.4666666666667, + "total_tokens_successful": 139870 + } + } +} diff --git a/experiments/mcp-tool-search/results/bm25_ident_split.csv b/experiments/mcp-tool-search/results/bm25_ident_split.csv new file mode 100644 index 00000000..82e6ff56 --- /dev/null +++ b/experiments/mcp-tool-search/results/bm25_ident_split.csv @@ -0,0 +1,151 @@ +id,query,kind,hit1,recall@1,recall@3,recall@5,mrr@10,ndcg@10,hit_rank,k3_calls,k3_tokens,k3_success1,k5_calls,k5_tokens,k5_success1,k10_calls,k10_tokens,k10_success1,ranking +q001,Create a GitHub issue,verb_product,github.create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,380,1.0,1,584,1.0,1,1035,1.0,github.create_issue github.issue_write gitlab.gitlab_create_issue github.issue_read github.add_issue_comment github.assign_copilot_to_issue bitbucket.bitbucket_create_issue github.update_issue_comment gitlab.gitlab_get_issue github.create_gist +q002,Create a Jira ticket,verb_product,jira.jira_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,281,1.0,1,466,1.0,1,951,1.0,jira.jira_create_issue zendesk.zendesk_create_ticket zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket jira.create_board jira.create_sprint jira.create_filter jira.create_webhook jira.create_attachment jira.create_version +q003,Send a Slack message,verb_product,discord.discord_send_message,0.0,1.0,1.0,0.5000,0.6309,2,1,244,1.0,1,427,1.0,1,760,1.0,discord.discord_send_message slack.slack_post_message twilio.twilio_send_sms slack.slack_add_reaction gmail.gmail_send_email discord.discord_add_reaction slack.slack_join_channel slack.slack_leave_channel slack.slack_search_messages slack.slack_create_channel +q004,List open pull requests,resource,github.list_pull_requests,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,460,1.0,1,836,1.0,github.list_pull_requests github.search_pull_requests bitbucket.bitbucket_list_prs gitlab.gitlab_list_merge_requests github.create_pull_request playwright.browser_tab_list memory.open_nodes github.pull_request_read github.merge_pull_request github.update_pull_request_branch +q005,Open a pull request on GitHub,verb_product,github.create_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,364,1.0,1,612,1.0,1,1144,1.0,github.create_pull_request github.request_copilot_review github.pull_request_read github.merge_pull_request github.pull_request_review_write github.update_pull_request github.update_pull_request_branch github.search_pull_requests github.add_comment_to_pending_review github.list_pull_requests +q006,Run a SQL query against postgres,verb_product,postgres.query,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,295,1.0,1,592,1.0,postgres.query mysql.mysql_query sqlite.sqlite_query postgres.explain_query salesforce.salesforce_query postgres.list_schemas postgres.list_tables postgres.describe_table docker.docker_run github.get_check_run +q007,Read a local file from disk,paraphrase,filesystem.read_file,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,390,1.0,1,809,1.0,filesystem.read_file filesystem.read_text_file filesystem.read_media_file figma.figma_get_file_nodes gitlab.gitlab_get_file figma.figma_get_images github.delete_file memory.read_graph brave_search.brave_local_search github.get_file_contents +q008,What files are in this folder,paraphrase,google_drive.gdrive_list_folder,0.0,0.0,0.0,0.1111,0.1923,9,3,717,0.0,2,756,0.0,1,734,1.0,google_drive.gdrive_list_folder slack.list_files github.push_files filesystem.search_files google_drive.gdrive_search github.create_gist cloudflare.cf_purge_cache github.update_gist filesystem.list_directory filesystem.move_file +q009,Remember this fact in the knowledge graph,paraphrase,memory.read_graph,0.0,1.0,1.0,0.5000,0.5339,2,1,154,1.0,1,261,1.0,1,507,1.0,memory.read_graph memory.create_entities memory.delete_relations memory.create_relations memory.add_observations memory.open_nodes memory.search_nodes memory.delete_entities postgres.list_schemas time.get_current_time +q010,Search the web with Brave,verb_product,brave_search.brave_web_search,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,410,1.0,1,684,1.0,brave_search.brave_web_search brave_search.brave_local_search jira.jira_search_issues memory.search_nodes github.update_pull_request_branch filesystem.list_directory_with_sizes confluence.confluence_search github.search_users github.search_orgs grafana.grafana_search_dashboards +q011,Fetch the contents of a URL,verb_product,fetch.fetch,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,330,1.0,1,663,1.0,fetch.fetch filesystem.read_file playwright.browser_navigate puppeteer.puppeteer_navigate git.git_add playwright.browser_snapshot github.get_file_contents git.git_status docker.docker_logs slack.slack_set_channel_topic +q012,Show git status,identifier,git.git_status,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,299,1.0,1,613,1.0,git.git_status git.git_show git.git_diff_unstaged git.git_diff_staged git.git_log git.git_diff sentry.sentry_update_issue git.git_branch git.git_checkout zendesk.zendesk_list_tickets +q013,Commit staged changes,paraphrase,git.git_commit,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,327,1.0,1,663,1.0,git.git_commit git.git_reset git.git_diff_staged github.get_commit git.git_show git.git_log github.push_files +q014,Create a Linear issue,verb_product,linear.linear_create_comment,0.0,1.0,1.0,0.5000,0.6309,2,1,333,1.0,1,463,1.0,1,917,1.0,linear.linear_create_comment linear.linear_create_issue linear.linear_update_issue linear.linear_get_issue linear.linear_assign_issue linear.linear_create_project bitbucket.bitbucket_create_issue gitlab.gitlab_create_issue linear.create_cycle linear.create_label +q015,Make a task in my project tracker Linear,paraphrase,linear.linear_create_project,0.0,0.0,0.0,0.0000,0.0000,146,49,12414,0.0,30,12154,0.0,15,11794,0.0,linear.linear_create_project asana.asana_get_task clickup.clickup_get_task asana.asana_update_task clickup.clickup_update_task asana.asana_create_task clickup.clickup_create_task mysql.mysql_list_tables mysql.mysql_query linear.get_cycle +q016,Post a comment on a GitHub issue,verb_product,figma.figma_post_comment,0.0,1.0,1.0,0.3333,0.5000,3,1,307,1.0,1,478,1.0,1,922,1.0,figma.figma_post_comment github.update_issue_comment github.add_issue_comment github.assign_copilot_to_issue linear.linear_create_comment jira.jira_add_comment trello.trello_add_comment github.issue_read slack.slack_post_message github.issue_write +q017,Merge the pull request,ambiguous,github.merge_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,283,1.0,1,468,1.0,1,1121,1.0,github.merge_pull_request gitlab.gitlab_merge_merge_request gitlab.gitlab_get_merge_request github.update_pull_request_branch github.pull_request_read github.request_copilot_review gitlab.gitlab_create_merge_request github.pull_request_review_write github.update_pull_request github.create_pull_request +q018,Trigger a GitHub Actions workflow,verb_product,github.actions_run_trigger,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,458,1.0,1,768,1.0,github.actions_run_trigger github.actions_get github.actions_list github.actions_get_job_logs circleci.circleci_trigger_pipeline github.list_tags github.search_users github.search_orgs github.get_tag github.search_commits +q019,Get Dependabot alerts,resource,github.list_dependabot_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,412,1.0,1,715,1.0,github.list_dependabot_alerts github.get_dependabot_alert github.list_secret_scanning_alerts github.list_code_scanning_alerts redis.redis_get kubernetes.k8s_get_pod shopify.shopify_get_product shopify.shopify_get_order vercel.vercel_get_deployment grafana.grafana_get_dashboard +q020,List Sentry issues,verb_product,sentry.sentry_list_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,395,1.0,1,796,1.0,sentry.sentry_list_issues sentry.sentry_list_projects sentry.sentry_list_events gitlab.gitlab_list_issues linear.linear_list_issues sentry.sentry_get_issue github.list_issues sentry.sentry_update_issue github.search_issues linear.linear_search_issues +q021,Create a Stripe customer,verb_product,stripe.stripe_create_customer,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,322,1.0,1,669,1.0,stripe.stripe_create_customer stripe.stripe_create_invoice stripe.stripe_get_customer stripe.stripe_create_payment_intent stripe.stripe_list_invoices stripe.stripe_list_subscriptions stripe.stripe_create_product stripe.stripe_create_refund stripe.stripe_create_price stripe.stripe_list_customers +q022,Refund a Stripe payment,paraphrase,stripe.stripe_create_refund,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,331,1.0,1,647,1.0,stripe.stripe_create_refund stripe.stripe_get_payment_intent stripe.stripe_create_payment_intent stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_list_customers stripe.stripe_create_product stripe.stripe_list_subscriptions +q023,Send an email with Gmail,verb_product,gmail.gmail_send_email,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,405,1.0,1,713,1.0,gmail.gmail_send_email gmail.gmail_get_email gmail.gmail_trash_email twilio.twilio_send_sms gmail.gmail_modify_email gmail.gmail_list_labels puppeteer.puppeteer_click puppeteer.puppeteer_fill gmail.gmail_search_emails gmail.gmail_create_draft +q024,Search my Gmail inbox,verb_product,gmail.gmail_search_emails,1.0,1.0,1.0,1.0000,1.0000,1,1,178,1.0,1,315,1.0,1,691,1.0,gmail.gmail_search_emails gmail.gmail_list_labels gmail.gmail_get_email gmail.gmail_create_draft gmail.gmail_trash_email gmail.gmail_send_email gmail.gmail_modify_email mysql.mysql_list_tables mysql.mysql_query brave_search.brave_web_search +q025,Upload a file to Google Drive,paraphrase,google_drive.gdrive_update_file,0.0,0.0,1.0,0.2500,0.4307,4,2,523,0.0,1,446,1.0,1,774,1.0,google_drive.gdrive_update_file google_drive.gdrive_share_file google_drive.gdrive_get_file google_drive.gdrive_create_file slack.slack_upload_file google_drive.gdrive_search google_drive.gdrive_list_folder aws_s3.s3_put_object google_calendar.gcal_list_calendars filesystem.edit_file +q026,Create a calendar event,resource,google_calendar.gcal_create_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,427,1.0,1,801,1.0,google_calendar.gcal_create_event google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars github.pull_request_review_write stripe.stripe_create_invoice vercel.vercel_create_project trello.trello_create_board stripe.stripe_create_customer +q027,List Kubernetes pods,verb_product,kubernetes.k8s_list_pods,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,303,1.0,1,718,1.0,kubernetes.k8s_list_pods kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_apply kubernetes.k8s_get_pod kubernetes.k8s_delete kubernetes.k8s_scale_deployment kubernetes.k8s_pod_logs clickup.clickup_list_tasks +q028,Get logs from a k8s pod,verb_product,kubernetes.k8s_pod_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,262,1.0,1,455,1.0,1,830,1.0,kubernetes.k8s_pod_logs kubernetes.k8s_get_pod docker.docker_logs datadog.datadog_list_logs github.actions_get_job_logs aws_s3.s3_get_object elasticsearch.es_get_doc github.get_tag slack.slack_get_channel_history figma.figma_get_file_nodes +q029,Run a docker container,verb_product,docker.docker_run,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,345,1.0,1,597,1.0,docker.docker_run docker.docker_exec docker.docker_stop docker.docker_rm docker.docker_logs docker.docker_images docker.docker_ps docker.docker_build github.get_check_run github.delete_check_run +q030,Show docker container logs,verb_product,docker.docker_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,345,1.0,1,649,1.0,docker.docker_logs docker.docker_stop docker.docker_rm docker.docker_exec docker.docker_run kubernetes.k8s_pod_logs git.git_log docker.docker_images docker.docker_ps docker.docker_build +q031,Navigate the browser to a URL,ambiguous,playwright.browser_navigate,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,346,1.0,1,617,1.0,playwright.browser_navigate puppeteer.puppeteer_navigate playwright.browser_tab_new fetch.fetch twilio.twilio_make_call playwright.browser_evaluate playwright.browser_press_key playwright.browser_snapshot playwright.browser_click playwright.browser_wait_for +q032,Take a screenshot of the page,ambiguous,playwright.browser_take_screenshot,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,320,1.0,1,608,1.0,playwright.browser_take_screenshot puppeteer.puppeteer_screenshot playwright.browser_snapshot filesystem.read_file playwright.browser_evaluate playwright.browser_press_key playwright.browser_click notion.notion_get_page notion.notion_update_page git.git_status +q033,Create a GitLab merge request,verb_product,gitlab.gitlab_merge_merge_request,0.0,1.0,1.0,0.3333,0.5000,3,1,297,1.0,1,475,1.0,1,1031,1.0,gitlab.gitlab_merge_merge_request gitlab.gitlab_get_merge_request gitlab.gitlab_create_merge_request github.merge_pull_request gitlab.gitlab_list_merge_requests gitlab.gitlab_create_issue gitlab.gitlab_create_pipeline github.create_pull_request github.request_copilot_review github.pull_request_review_write +q034,Search Notion pages,verb_product,notion.notion_search,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,322,1.0,1,671,1.0,notion.notion_search notion.notion_list_users notion.notion_get_page notion.notion_get_block brave_search.brave_web_search notion.notion_create_database notion.notion_update_page notion.notion_query_database notion.notion_get_comments confluence.confluence_search +q035,Create a Notion page,verb_product,notion.notion_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,230,1.0,1,371,1.0,1,822,1.0,notion.notion_create_page notion.notion_get_page notion.notion_update_page notion.notion_get_comments notion.notion_create_database confluence.confluence_create_page notion.create_comment notion.create_users_token notion.create_data_source confluence.confluence_get_page +q036,Query a Notion database,verb_product,notion.notion_query_database,1.0,1.0,1.0,1.0000,1.0000,1,1,261,1.0,1,386,1.0,1,668,1.0,notion.notion_query_database notion.notion_create_database notion.notion_search notion.notion_create_page notion.notion_list_users notion.notion_get_page notion.notion_get_block notion.notion_update_page notion.notion_get_comments prometheus.prom_query +q037,Create a Zendesk ticket,verb_product,zendesk.zendesk_create_ticket,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,368,1.0,1,714,1.0,zendesk.zendesk_create_ticket zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_list_tickets zendesk.zendesk_search jira.jira_create_issue stripe.stripe_create_invoice vercel.vercel_create_project trello.trello_create_board stripe.stripe_create_customer +q038,Create a HubSpot contact,verb_product,hubspot.hubspot_create_contact,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,899,1.0,hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_create_deal hubspot.hubspot_list_contacts hubspot.hubspot_search_contacts github.create_gist github.create_repository github.create_branch github.create_issue github.create_or_update_file +q039,Create a Salesforce lead,verb_product,salesforce.salesforce_convert_lead,0.0,1.0,1.0,0.5000,0.6309,2,1,252,1.0,1,364,1.0,1,710,1.0,salesforce.salesforce_convert_lead salesforce.salesforce_create_lead salesforce.salesforce_create_record salesforce.salesforce_list_opportunities salesforce.salesforce_query salesforce.salesforce_delete_record salesforce.salesforce_update_record salesforce.salesforce_get_record stripe.stripe_create_invoice vercel.vercel_create_project +q040,Run a SOQL query,identifier,salesforce.salesforce_query,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,332,1.0,1,662,1.0,salesforce.salesforce_query sqlite.sqlite_query mysql.mysql_query postgres.query docker.docker_run github.get_check_run github.delete_check_run github.update_check_run prometheus.prom_query github.create_check_run +q041,List Shopify orders,verb_product,shopify.shopify_list_orders,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,320,1.0,1,682,1.0,shopify.shopify_list_orders shopify.shopify_list_customers shopify.shopify_list_products shopify.shopify_get_product shopify.shopify_get_order shopify.shopify_update_product shopify.shopify_create_order shopify.shopify_create_product clickup.clickup_list_tasks trello.trello_list_cards +q042,Create a PagerDuty incident,verb_product,pagerduty.pagerduty_create_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,403,1.0,1,696,1.0,pagerduty.pagerduty_create_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls stripe.stripe_create_invoice vercel.vercel_create_project trello.trello_create_board stripe.stripe_create_customer +q043,Who is on call in PagerDuty,paraphrase,pagerduty.pagerduty_list_oncalls,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,363,1.0,1,724,1.0,pagerduty.pagerduty_list_oncalls pagerduty.pagerduty_get_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_create_incident twilio.twilio_make_call slack.slack_create_channel filesystem.list_allowed_directories memory.search_nodes +q044,Query Datadog metrics,verb_product,datadog.datadog_query_metrics,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,458,1.0,1,774,1.0,datadog.datadog_query_metrics datadog.datadog_list_monitors datadog.datadog_create_monitor datadog.datadog_list_logs datadog.datadog_get_monitor prometheus.prom_list_metrics prometheus.prom_query prometheus.prom_query_range postgres.explain_query salesforce.salesforce_query +q045,Search Datadog logs,verb_product,datadog.datadog_list_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,458,1.0,1,855,1.0,datadog.datadog_list_logs datadog.datadog_list_monitors datadog.datadog_get_monitor datadog.datadog_create_monitor datadog.datadog_query_metrics docker.docker_logs kubernetes.k8s_pod_logs github.actions_get_job_logs brave_search.brave_web_search confluence.confluence_search +q046,List S3 buckets,verb_product,aws_s3.s3_list_buckets,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,258,1.0,1,474,1.0,aws_s3.s3_list_buckets clickup.clickup_list_tasks trello.trello_list_cards linear.linear_list_teams jira.jira_list_projects kubernetes.k8s_list_deployments kubernetes.k8s_list_services gmail.gmail_list_labels cloudflare.cf_list_zones vercel.vercel_list_projects +q047,Upload an object to S3,paraphrase,aws_s3.s3_put_object,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,437,1.0,1,850,1.0,aws_s3.s3_put_object aws_s3.s3_delete_object aws_s3.s3_get_object slack.slack_upload_file github.assign_copilot_to_issue twilio.twilio_send_sms gmail.gmail_send_email slack.slack_add_reaction slack.slack_reply_to_thread puppeteer.puppeteer_click +q048,Purge Cloudflare cache,verb_product,cloudflare.cf_purge_cache,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,340,1.0,1,746,1.0,cloudflare.cf_purge_cache cloudflare.cf_list_zones cloudflare.cf_list_dns cloudflare.cf_create_dns +q049,Create a Cloudflare DNS record,verb_product,cloudflare.cf_create_dns,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,356,1.0,1,708,1.0,cloudflare.cf_create_dns cloudflare.cf_list_dns salesforce.salesforce_create_record cloudflare.cf_list_zones cloudflare.cf_purge_cache salesforce.salesforce_delete_record salesforce.salesforce_update_record salesforce.salesforce_get_record git.git_commit stripe.stripe_create_invoice +q050,List Vercel deployments,verb_product,vercel.vercel_list_deployments,1.0,1.0,1.0,1.0000,1.0000,1,1,173,1.0,1,291,1.0,1,529,1.0,vercel.vercel_list_deployments vercel.vercel_list_projects kubernetes.k8s_list_deployments vercel.vercel_get_deployment vercel.vercel_create_project clickup.clickup_list_tasks trello.trello_list_cards aws_s3.s3_list_buckets linear.linear_list_teams jira.jira_list_projects +q051,Insert a row into Supabase,verb_product,supabase.supabase_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,203,1.0,1,347,1.0,1,665,1.0,supabase.supabase_insert mongodb.mongo_insert supabase.supabase_list_tables supabase.supabase_select supabase.supabase_invoke_fn supabase.supabase_update playwright.browser_type +q052,Export images from Figma,paraphrase,figma.figma_get_images,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,328,1.0,1,637,1.0,figma.figma_get_images figma.figma_get_file_nodes docker.docker_images figma.figma_get_file figma.figma_get_comments figma.figma_post_comment memory.delete_observations aws_s3.s3_get_object memory.delete_relations docker.docker_logs +q053,Send an SMS with Twilio,verb_product,twilio.twilio_send_sms,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,396,1.0,1,725,1.0,twilio.twilio_send_sms twilio.twilio_list_messages gmail.gmail_send_email twilio.twilio_make_call puppeteer.puppeteer_click puppeteer.puppeteer_fill discord.discord_send_message filesystem.list_directory_with_sizes puppeteer.puppeteer_navigate puppeteer.puppeteer_screenshot +q054,Create an Okta user,verb_product,okta.okta_create_user,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,329,1.0,1,622,1.0,okta.okta_create_user okta.okta_get_user okta.okta_deactivate_user okta.okta_list_users linear.create_user slack.slack_get_user_profile asana.asana_create_project linear.get_user linear.delete_user linear.update_user +q055,Instant query Prometheus,verb_product,prometheus.prom_query,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,377,1.0,1,680,1.0,prometheus.prom_query prometheus.prom_query_range prometheus.prom_list_metrics datadog.datadog_query_metrics postgres.explain_query salesforce.salesforce_query sqlite.sqlite_query mysql.mysql_query postgres.query notion.notion_query_database +q056,Search Grafana dashboards,verb_product,grafana.grafana_search_dashboards,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,360,1.0,1,660,1.0,grafana.grafana_search_dashboards grafana.grafana_get_dashboard grafana.grafana_create_annotation brave_search.brave_web_search confluence.confluence_search github.search_users github.search_orgs brave_search.brave_local_search github.search_commits slack.slack_search_messages +q057,Trigger a CircleCI pipeline,verb_product,circleci.circleci_trigger_pipeline,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,390,1.0,1,755,1.0,circleci.circleci_trigger_pipeline circleci.circleci_get_pipeline gitlab.gitlab_create_pipeline circleci.circleci_list_pipelines github.actions_run_trigger gitlab.get_pipeline gitlab.delete_pipeline gitlab.update_pipeline gitlab.create_pipeline hubspot.hubspot_create_deal +q058,Run terraform plan,identifier,terraform.tf_plan,1.0,1.0,1.0,1.0000,1.0000,1,1,172,1.0,1,320,1.0,1,658,1.0,terraform.tf_plan terraform.tf_apply terraform.tf_state_list postgres.explain_query docker.docker_run github.get_check_run github.delete_check_run github.update_check_run github.create_check_run salesforce.salesforce_query +q059,Create an Asana task,verb_product,asana.asana_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,429,1.0,1,825,1.0,asana.asana_create_task asana.asana_get_task asana.asana_update_task asana.asana_create_project clickup.clickup_create_task asana.asana_list_tasks clickup.clickup_get_task clickup.clickup_update_task okta.okta_create_user github.create_issue +q060,Create a Trello card,verb_product,trello.trello_create_card,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,364,1.0,1,679,1.0,trello.trello_create_card trello.trello_add_comment trello.trello_create_board trello.trello_list_cards stripe.stripe_create_invoice vercel.vercel_create_project stripe.stripe_create_customer stripe.stripe_create_product salesforce.salesforce_create_record okta.okta_create_user +q061,Create a ClickUp task,verb_product,clickup.clickup_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,415,1.0,1,721,1.0,clickup.clickup_create_task clickup.clickup_get_task clickup.clickup_update_task clickup.clickup_list_tasks asana.asana_create_task asana.asana_get_task asana.asana_update_task puppeteer.puppeteer_click playwright.browser_click stripe.stripe_create_invoice +q062,Create a Confluence page,verb_product,confluence.confluence_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,401,1.0,1,699,1.0,confluence.confluence_create_page confluence.confluence_get_page confluence.confluence_update_page confluence.confluence_search notion.notion_create_page notion.notion_get_page notion.notion_update_page playwright.browser_take_screenshot playwright.browser_evaluate playwright.browser_press_key +q063,Send a Discord message,verb_product,discord.discord_send_message,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,322,1.0,1,682,1.0,discord.discord_send_message discord.discord_add_reaction discord.discord_get_channel discord.discord_list_channels discord.discord_list_members twilio.twilio_send_sms gmail.gmail_send_email slack.slack_post_message gmail.gmail_get_email gmail.gmail_trash_email +q064,Search Elasticsearch,verb_product,elasticsearch.es_search,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,382,1.0,1,672,1.0,elasticsearch.es_search elasticsearch.es_get_doc elasticsearch.es_index_doc brave_search.brave_web_search confluence.confluence_search github.search_users github.search_orgs grafana.grafana_search_dashboards brave_search.brave_local_search github.search_commits +q065,Get a Redis key,verb_product,redis.redis_get,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,281,1.0,1,589,1.0,redis.redis_get redis.redis_set redis.redis_delete redis.redis_keys jira.jira_get_issue aws_s3.s3_get_object figma.figma_get_file playwright.browser_press_key figma.figma_get_comments figma.figma_get_file_nodes +q066,Find MongoDB documents,verb_product,mongodb.mongo_find,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,381,1.0,1,738,1.0,mongodb.mongo_find mongodb.mongo_insert mongodb.mongo_delete mongodb.mongo_update mongodb.mongo_list_collections +q067,List SQLite tables,verb_product,sqlite.sqlite_list_tables,1.0,1.0,1.0,1.0000,1.0000,1,1,156,1.0,1,243,1.0,1,496,1.0,sqlite.sqlite_list_tables sqlite.sqlite_describe_table sqlite.sqlite_query mysql.mysql_list_tables supabase.supabase_list_tables postgres.list_tables clickup.clickup_list_tasks trello.trello_list_cards aws_s3.s3_list_buckets linear.linear_list_teams +q068,Describe a postgres table,verb_product,postgres.describe_table,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,349,1.0,1,597,1.0,postgres.describe_table sqlite.sqlite_describe_table supabase.supabase_select supabase.supabase_insert supabase.supabase_update postgres.explain_query postgres.list_schemas postgres.query postgres.list_tables +q069,github_create_issue,identifier,github.create_issue,1.0,1.0,1.0,1.0000,0.9502,1,1,378,1.0,1,568,1.0,1,1091,1.0,github.create_issue bitbucket.bitbucket_create_issue github.issue_write gitlab.gitlab_create_issue github.issue_read github.add_issue_comment linear.linear_create_issue jira.jira_create_issue linear.linear_create_comment github.assign_copilot_to_issue +q070,createIssue,identifier,bitbucket.bitbucket_create_issue,0.0,1.0,1.0,0.3333,0.5805,3,1,332,1.0,1,579,1.0,1,971,1.0,bitbucket.bitbucket_create_issue gitlab.gitlab_create_issue github.create_issue linear.linear_create_issue jira.jira_create_issue linear.linear_create_comment github.issue_write sentry.sentry_get_issue jira.jira_get_issue jira.jira_assign_issue +q071,list-directory,identifier,filesystem.create_directory,0.0,1.0,1.0,0.5000,0.6309,2,1,186,1.0,1,296,1.0,1,607,1.0,filesystem.create_directory filesystem.list_directory filesystem.list_directory_with_sizes filesystem.get_file_info filesystem.directory_tree github.get_file_contents clickup.clickup_list_tasks trello.trello_list_cards aws_s3.s3_list_buckets linear.linear_list_teams +q072,sequential thinking next thought,paraphrase,sequential_thinking.sequentialthinking,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,382,1.0,1,797,1.0,sequential_thinking.sequentialthinking +q073,What time is it in Tokyo,paraphrase,time.get_current_time,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,378,1.0,1,703,1.0,time.get_current_time time.convert_time playwright.browser_wait_for google_calendar.gcal_list_events slack.slack_create_channel filesystem.list_allowed_directories grafana.grafana_create_annotation jira.jira_add_worklog github.get_milestone github.delete_milestone +q074,Convert time between timezones,verb_product,time.convert_time,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,360,1.0,1,704,1.0,time.convert_time salesforce.salesforce_convert_lead time.get_current_time playwright.browser_wait_for google_calendar.gcal_list_events memory.create_relations git.git_diff grafana.grafana_create_annotation jira.jira_add_worklog +q075,Star a GitHub repo,verb_product,github.star_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,225,1.0,1,381,1.0,1,771,1.0,github.star_repository github.list_tags github.delete_repository github.unstar_repository github.get_tag github.fork_repository github.search_issues github.actions_list github.get_label github.list_branches +q076,Fork a repository,resource,github.fork_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,235,1.0,1,386,1.0,1,792,1.0,github.fork_repository github.delete_repository github.star_repository github.unstar_repository github.create_repository github.list_repository_collaborators github.get_repository_tree github.list_branches github.list_releases github.list_dependabot_alerts +q077,Get the latest GitHub release,verb_product,github.get_latest_release,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,383,1.0,1,742,1.0,github.get_latest_release github.get_release_by_tag gitlab.get_release github.get_teams github.get_repository_tree github.get_me gitlab.delete_release gitlab.update_release github.get_tag gitlab.create_release +q078,Create a gist,resource,github.create_gist,1.0,1.0,1.0,1.0000,1.0000,1,1,229,1.0,1,335,1.0,1,649,1.0,github.create_gist github.get_gist github.update_gist stripe.stripe_create_invoice vercel.vercel_create_project trello.trello_create_board stripe.stripe_create_customer stripe.stripe_create_product salesforce.salesforce_create_record okta.okta_create_user +q079,Search GitHub code,verb_product,github.search_code,1.0,1.0,1.0,1.0000,1.0000,1,1,283,1.0,1,385,1.0,1,725,1.0,github.search_code github.list_code_scanning_alerts github.get_code_scanning_alert github.search_users github.search_orgs github.search_commits github.search_issues github.search_pull_requests github.search_repositories hubspot.hubspot_search_contacts +q080,List GitHub notifications,verb_product,github.list_notifications,1.0,1.0,1.0,1.0000,1.0000,1,1,193,1.0,1,311,1.0,1,662,1.0,github.list_notifications github.mark_all_notifications_read github.list_tags github.actions_list github.list_gists github.projects_list github.list_branches github.list_releases github.list_discussions github.list_labels +q081,Assign Copilot to an issue,verb_product,github.assign_copilot_to_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,402,1.0,1,833,1.0,github.assign_copilot_to_issue linear.linear_assign_issue jira.jira_assign_issue jira.jira_update_issue jira.jira_transition_issue github.add_issue_comment linear.linear_create_comment jira.jira_add_comment github.request_copilot_review jira.jira_add_worklog +q082,Get secret scanning alerts,resource,github.list_secret_scanning_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,295,1.0,1,470,1.0,1,765,1.0,github.list_secret_scanning_alerts github.get_secret_scanning_alert github.list_code_scanning_alerts github.get_code_scanning_alert github.list_dependabot_alerts redis.redis_get kubernetes.k8s_get_pod shopify.shopify_get_product shopify.shopify_get_order vercel.vercel_get_deployment +q083,Create or update a file on GitHub,verb_product,github.create_or_update_file,1.0,1.0,1.0,1.0000,0.7602,1,1,369,1.0,1,546,1.0,1,1001,1.0,github.create_or_update_file github.projects_write github.label_write github.get_file_contents filesystem.write_file github.update_issue_comment github.issue_write slack.update_file google_drive.gdrive_update_file figma.figma_get_comments +q084,List collaborators on a repo,resource,github.list_repository_collaborators,1.0,1.0,1.0,1.0000,1.0000,1,1,278,1.0,1,433,1.0,1,932,1.0,github.list_repository_collaborators github.list_commits trello.trello_list_cards pagerduty.pagerduty_list_oncalls github.get_discussion_comments github.update_issue_comment github.request_copilot_review github.actions_list bitbucket.bitbucket_list_prs github.create_pull_request +q085,Reply in a Slack thread,verb_product,slack.slack_reply_to_thread,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,350,1.0,1,632,1.0,slack.slack_reply_to_thread slack.slack_get_thread_replies github.get_notification_details slack.get_reminder slack.delete_reminder slack.get_canvase slack.delete_canvase slack.get_bookmark slack.delete_bookmark slack.get_usergroup +q086,Add an emoji reaction on Slack,verb_product,slack.slack_add_reaction,1.0,1.0,1.0,1.0000,1.0000,1,1,271,1.0,1,428,1.0,1,811,1.0,slack.slack_add_reaction discord.discord_add_reaction trello.trello_add_comment gmail.gmail_modify_email playwright.browser_click jira.jira_add_comment memory.add_observations git.git_add jira.jira_add_worklog github.assign_copilot_to_issue +q087,Search Slack messages,verb_product,slack.slack_search_messages,1.0,1.0,1.0,1.0000,1.0000,1,1,232,1.0,1,373,1.0,1,664,1.0,slack.slack_search_messages gmail.gmail_search_emails slack.slack_get_channel_history twilio.twilio_list_messages brave_search.brave_web_search confluence.confluence_search github.search_users github.search_orgs grafana.grafana_search_dashboards brave_search.brave_local_search +q088,Create a Slack channel,verb_product,slack.slack_create_channel,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,342,1.0,1,751,1.0,slack.slack_create_channel slack.slack_join_channel slack.slack_leave_channel slack.slack_get_channel_history slack.slack_set_channel_topic slack.slack_post_message slack.slack_upload_file slack.slack_reply_to_thread slack.slack_get_thread_replies slack.slack_add_reaction +q089,Transition a Jira issue,verb_product,jira.jira_transition_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,342,1.0,1,778,1.0,jira.jira_transition_issue jira.jira_get_issue jira.jira_assign_issue jira.jira_update_issue jira.jira_add_comment jira.jira_add_worklog jira.jira_create_issue sentry.sentry_get_issue gitlab.gitlab_get_issue sentry.sentry_update_issue +q090,Search Jira with JQL,identifier,jira.jira_search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,241,1.0,1,345,1.0,1,619,1.0,jira.jira_search_issues filesystem.list_directory_with_sizes brave_search.brave_web_search confluence.confluence_search github.search_users github.search_orgs grafana.grafana_search_dashboards jira.jira_list_projects brave_search.brave_local_search github.search_commits +q091,Add a worklog to Jira,verb_product,jira.jira_add_worklog,1.0,1.0,1.0,1.0000,1.0000,1,1,236,1.0,1,389,1.0,1,874,1.0,jira.jira_add_worklog jira.jira_add_comment memory.add_observations git.git_add discord.discord_add_reaction github.add_issue_comment slack.slack_add_reaction github.add_comment_to_pending_review jira.jira_transition_issue linear.linear_create_comment +q092,List Linear teams,verb_product,linear.linear_list_teams,1.0,1.0,1.0,1.0000,1.0000,1,1,137,1.0,1,295,1.0,1,646,1.0,linear.linear_list_teams github.get_teams linear.linear_list_projects linear.linear_list_issues linear.list_cycles linear.list_labels linear.list_users linear.list_comments linear.list_attachments linear.linear_search_issues +q093,Comment on a Linear issue,verb_product,linear.linear_create_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,360,1.0,1,777,1.0,linear.linear_create_comment github.update_issue_comment linear.get_comment linear.delete_comment linear.update_comment github.add_issue_comment jira.jira_add_comment linear.linear_update_issue linear.linear_get_issue linear.linear_assign_issue +q094,Move a local file,paraphrase,filesystem.move_file,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,358,1.0,1,698,1.0,filesystem.move_file filesystem.read_file filesystem.read_text_file filesystem.read_media_file brave_search.brave_local_search gmail.gmail_trash_email figma.figma_get_file google_drive.gdrive_update_file google_drive.gdrive_share_file slack.get_file +q095,Recursively search files by pattern,paraphrase,filesystem.search_files,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,338,1.0,1,721,1.0,filesystem.search_files linear.linear_search_issues google_drive.gdrive_search redis.redis_keys github.search_repositories slack.list_files brave_search.brave_web_search github.push_files confluence.confluence_search github.search_users +q096,Edit a file in place with old and new string,paraphrase,filesystem.edit_file,1.0,1.0,1.0,1.0000,1.0000,1,1,247,1.0,1,358,1.0,1,673,1.0,filesystem.edit_file filesystem.list_directory_with_sizes filesystem.write_file filesystem.move_file playwright.browser_tab_new slack.get_file slack.delete_file twilio.twilio_make_call slack.update_file figma.figma_get_file +q097,Read the whole knowledge graph,paraphrase,memory.read_graph,1.0,1.0,1.0,1.0000,1.0000,1,1,152,1.0,1,259,1.0,1,561,1.0,memory.read_graph memory.delete_relations memory.create_entities memory.create_relations memory.add_observations memory.open_nodes memory.delete_entities memory.search_nodes filesystem.read_file filesystem.read_text_file +q098,Search memory nodes,verb_product,memory.search_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,202,1.0,1,345,1.0,1,635,1.0,memory.search_nodes memory.open_nodes figma.figma_get_file_nodes brave_search.brave_web_search confluence.confluence_search github.search_users github.search_orgs grafana.grafana_search_dashboards brave_search.brave_local_search github.search_commits +q099,Show unstaged git diff,verb_product,git.git_diff_unstaged,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,313,1.0,1,602,1.0,git.git_diff_unstaged git.git_diff git.git_diff_staged git.git_show git.git_log git.git_status git.git_branch git.git_checkout git.git_commit git.git_add +q100,Stage files with git add,identifier,git.git_add,1.0,1.0,1.0,1.0000,1.0000,1,1,197,1.0,1,417,1.0,1,872,1.0,git.git_add filesystem.list_directory_with_sizes salesforce.salesforce_list_opportunities github.push_files github.add_issue_comment github.create_gist slack.list_files github.update_gist jira.jira_add_comment github.add_comment_to_pending_review +q101,Create a Shopify product,verb_product,shopify.shopify_create_product,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,377,1.0,1,702,1.0,shopify.shopify_create_product shopify.shopify_get_product shopify.shopify_update_product shopify.shopify_create_order stripe.stripe_create_product shopify.shopify_get_order shopify.shopify_list_customers shopify.shopify_list_products shopify.shopify_list_orders stripe.stripe_create_price +q102,Cancel a Stripe subscription,verb_product,stripe.stripe_cancel_subscription,1.0,1.0,1.0,1.0000,1.0000,1,1,167,1.0,1,285,1.0,1,610,1.0,stripe.stripe_cancel_subscription stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_list_customers stripe.stripe_create_product stripe.stripe_list_subscriptions stripe.stripe_get_payment_intent stripe.stripe_create_refund +q103,Create a payment intent,resource,stripe.stripe_create_payment_intent,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,340,1.0,1,654,1.0,stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent stripe.stripe_create_refund stripe.stripe_create_invoice vercel.vercel_create_project trello.trello_create_board stripe.stripe_create_customer stripe.stripe_create_product salesforce.salesforce_create_record okta.okta_create_user +q104,Acknowledge a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,405,1.0,1,717,1.0,pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls +q105,Create a Datadog monitor,verb_product,datadog.datadog_create_monitor,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,459,1.0,1,758,1.0,datadog.datadog_create_monitor datadog.datadog_get_monitor datadog.datadog_list_monitors datadog.datadog_list_logs datadog.datadog_query_metrics stripe.stripe_create_invoice vercel.vercel_create_project trello.trello_create_board stripe.stripe_create_customer stripe.stripe_create_product +q106,Resolve a Sentry issue,paraphrase,sentry.sentry_get_issue,0.0,1.0,1.0,0.5000,0.6309,2,1,251,1.0,1,368,1.0,1,721,1.0,sentry.sentry_get_issue sentry.sentry_update_issue sentry.sentry_list_events pagerduty.pagerduty_resolve_incident sentry.sentry_list_projects sentry.sentry_list_issues jira.jira_get_issue jira.jira_assign_issue jira.jira_update_issue gitlab.gitlab_get_issue +q107,Scale a Kubernetes deployment,verb_product,kubernetes.k8s_scale_deployment,1.0,1.0,1.0,1.0000,1.0000,1,1,245,1.0,1,353,1.0,1,726,1.0,kubernetes.k8s_scale_deployment vercel.vercel_get_deployment kubernetes.k8s_apply kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_get_pod kubernetes.k8s_delete kubernetes.k8s_list_pods kubernetes.k8s_pod_logs +q108,Apply a Kubernetes manifest,verb_product,kubernetes.k8s_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,308,1.0,1,727,1.0,kubernetes.k8s_apply terraform.tf_apply kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_get_pod kubernetes.k8s_delete kubernetes.k8s_list_pods kubernetes.k8s_scale_deployment kubernetes.k8s_pod_logs +q109,Stop a docker container,verb_product,docker.docker_stop,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,345,1.0,1,641,1.0,docker.docker_stop docker.docker_rm docker.docker_logs docker.docker_exec docker.docker_run docker.docker_images docker.docker_ps docker.docker_build kubernetes.k8s_pod_logs +q110,Build a docker image,verb_product,docker.docker_build,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,303,1.0,1,601,1.0,docker.docker_build docker.docker_run docker.docker_images docker.docker_stop docker.docker_ps docker.docker_rm docker.docker_logs docker.docker_exec filesystem.read_media_file +q111,Click an element in Playwright,verb_product,playwright.browser_click,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,329,1.0,1,612,1.0,playwright.browser_click puppeteer.puppeteer_click playwright.browser_type playwright.browser_take_screenshot playwright.browser_snapshot clickup.clickup_list_tasks clickup.clickup_get_task playwright.browser_tab_list playwright.browser_tab_close clickup.clickup_create_task +q112,Type text into a browser input,paraphrase,playwright.browser_type,1.0,1.0,1.0,1.0000,0.9502,1,1,235,1.0,1,320,1.0,1,648,1.0,playwright.browser_type playwright.browser_wait_for puppeteer.puppeteer_fill playwright.browser_tab_list playwright.browser_tab_close playwright.browser_navigate playwright.browser_tab_new filesystem.read_text_file supabase.supabase_insert mongodb.mongo_insert +q113,List GitLab projects,verb_product,gitlab.gitlab_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,171,1.0,1,255,1.0,1,609,1.0,gitlab.gitlab_list_projects github.projects_list jira.jira_list_projects vercel.vercel_list_projects linear.linear_list_projects sentry.sentry_list_projects gitlab.gitlab_list_issues gitlab.gitlab_list_merge_requests github.projects_get gitlab.gitlab_create_issue +q114,Create a Bitbucket pull request,verb_product,bitbucket.bitbucket_create_pr,1.0,1.0,1.0,1.0000,1.0000,1,1,313,1.0,1,546,1.0,1,1139,1.0,bitbucket.bitbucket_create_pr bitbucket.bitbucket_list_prs bitbucket.bitbucket_create_issue github.pull_request_review_write github.pull_request_read github.merge_pull_request github.request_copilot_review github.create_pull_request github.update_pull_request_branch github.update_pull_request +q115,Create a Gmail draft,verb_product,gmail.gmail_create_draft,1.0,1.0,1.0,1.0000,1.0000,1,1,205,1.0,1,315,1.0,1,758,1.0,gmail.gmail_create_draft gmail.gmail_list_labels gmail.gmail_search_emails gmail.gmail_get_email gmail.gmail_trash_email gmail.gmail_send_email gmail.gmail_modify_email github.create_pull_request stripe.stripe_create_invoice vercel.vercel_create_project +q116,Share a Google Drive file,verb_product,google_drive.gdrive_share_file,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,417,1.0,1,725,1.0,google_drive.gdrive_share_file google_drive.gdrive_update_file google_drive.gdrive_get_file google_drive.gdrive_create_file google_drive.gdrive_search google_drive.gdrive_list_folder google_calendar.gcal_list_calendars figma.figma_get_file google_calendar.gcal_delete_event google_calendar.gcal_update_event +q117,List Google Calendar events,verb_product,google_calendar.gcal_list_events,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,427,1.0,1,753,1.0,google_calendar.gcal_list_events google_calendar.gcal_list_calendars google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event kubernetes.k8s_get_events sentry.sentry_list_events google_drive.gdrive_list_folder google_drive.gdrive_search google_drive.gdrive_update_file +q118,Delete a calendar event,resource,google_calendar.gcal_delete_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,427,1.0,1,735,1.0,google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars redis.redis_delete kubernetes.k8s_delete aws_s3.s3_delete_object github.delete_milestone github.delete_hook +q119,List Vercel projects,verb_product,vercel.vercel_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,151,1.0,1,248,1.0,1,550,1.0,vercel.vercel_list_projects vercel.vercel_list_deployments jira.jira_list_projects linear.linear_list_projects sentry.sentry_list_projects vercel.vercel_get_deployment vercel.vercel_create_project gitlab.gitlab_list_projects github.projects_list github.projects_get +q120,Select rows from Supabase,verb_product,supabase.supabase_select,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,354,1.0,1,660,1.0,supabase.supabase_select supabase.supabase_insert supabase.supabase_update supabase.supabase_list_tables supabase.supabase_invoke_fn memory.delete_observations aws_s3.s3_get_object memory.delete_relations docker.docker_logs elasticsearch.es_get_doc +q121,Post a Figma comment,verb_product,figma.figma_post_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,362,1.0,1,646,1.0,figma.figma_post_comment figma.figma_get_file figma.figma_get_comments slack.slack_post_message figma.figma_get_images figma.figma_get_file_nodes linear.get_comment linear.delete_comment notion.get_comment notion.delete_comment +q122,List Okta users,verb_product,okta.okta_list_users,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,271,1.0,1,606,1.0,okta.okta_list_users okta.okta_create_user okta.okta_get_user okta.okta_deactivate_user notion.notion_list_users slack.slack_list_users linear.list_users notion.list_users_tokens slack.slack_get_users github.search_users +q123,Range query Prometheus,verb_product,prometheus.prom_query_range,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,377,1.0,1,680,1.0,prometheus.prom_query_range prometheus.prom_query prometheus.prom_list_metrics datadog.datadog_query_metrics postgres.explain_query salesforce.salesforce_query sqlite.sqlite_query mysql.mysql_query postgres.query notion.notion_query_database +q124,Create a Grafana annotation,verb_product,grafana.grafana_create_annotation,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,324,1.0,1,638,1.0,grafana.grafana_create_annotation grafana.grafana_search_dashboards grafana.grafana_get_dashboard stripe.stripe_create_invoice vercel.vercel_create_project trello.trello_create_board stripe.stripe_create_customer stripe.stripe_create_product salesforce.salesforce_create_record okta.okta_create_user +q125,Apply terraform,identifier,terraform.tf_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,292,1.0,1,698,1.0,terraform.tf_apply kubernetes.k8s_apply terraform.tf_plan terraform.tf_state_list +q126,List Asana tasks,verb_product,asana.asana_list_tasks,1.0,1.0,1.0,1.0000,1.0000,1,1,192,1.0,1,368,1.0,1,642,1.0,asana.asana_list_tasks clickup.clickup_list_tasks asana.asana_get_task asana.asana_create_project asana.asana_create_task asana.asana_update_task trello.trello_list_cards aws_s3.s3_list_buckets linear.linear_list_teams jira.jira_list_projects +q127,Comment on a Trello card,verb_product,trello.trello_add_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,422,1.0,1,703,1.0,trello.trello_add_comment trello.trello_create_card trello.trello_list_cards trello.trello_create_board github.update_issue_comment figma.figma_post_comment linear.get_comment linear.delete_comment notion.get_comment notion.delete_comment +q128,Update a ClickUp task,verb_product,clickup.clickup_update_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,399,1.0,1,761,1.0,clickup.clickup_update_task clickup.clickup_get_task clickup.clickup_create_task asana.asana_update_task clickup.clickup_list_tasks asana.asana_get_task puppeteer.puppeteer_click playwright.browser_click asana.asana_create_task mongodb.mongo_update +q129,Search Confluence,verb_product,confluence.confluence_search,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,401,1.0,1,691,1.0,confluence.confluence_search confluence.confluence_get_page confluence.confluence_update_page confluence.confluence_create_page brave_search.brave_web_search github.search_users github.search_orgs grafana.grafana_search_dashboards brave_search.brave_local_search github.search_commits +q130,Create a HubSpot deal,verb_product,hubspot.hubspot_create_deal,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,899,1.0,hubspot.hubspot_create_deal hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_list_contacts hubspot.hubspot_search_contacts github.create_gist github.create_repository github.create_branch github.create_issue github.create_or_update_file +q131,List Zendesk tickets,verb_product,zendesk.zendesk_list_tickets,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,367,1.0,1,605,1.0,zendesk.zendesk_list_tickets zendesk.zendesk_search zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_create_ticket clickup.clickup_list_tasks trello.trello_list_cards aws_s3.s3_list_buckets linear.linear_list_teams jira.jira_list_projects +q132,List Discord channels,verb_product,discord.discord_list_channels,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,300,1.0,1,592,1.0,discord.discord_list_channels discord.discord_list_members slack.slack_list_channels discord.discord_get_channel discord.discord_send_message discord.discord_add_reaction clickup.clickup_list_tasks trello.trello_list_cards aws_s3.s3_list_buckets linear.linear_list_teams +q133,Set a Redis key,verb_product,redis.redis_set,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,302,1.0,1,600,1.0,redis.redis_set redis.redis_get redis.redis_delete redis.redis_keys slack.slack_set_channel_topic playwright.browser_press_key jira.jira_get_issue aws_s3.s3_delete_object aws_s3.s3_get_object figma.figma_get_file +q134,Insert MongoDB documents,verb_product,mongodb.mongo_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,381,1.0,1,698,1.0,mongodb.mongo_insert mongodb.mongo_delete mongodb.mongo_find mongodb.mongo_update mongodb.mongo_list_collections supabase.supabase_insert +q135,Run a MySQL query,verb_product,mysql.mysql_query,1.0,1.0,1.0,1.0000,1.0000,1,1,179,1.0,1,281,1.0,1,611,1.0,mysql.mysql_query postgres.query sqlite.sqlite_query postgres.explain_query mysql.mysql_list_tables salesforce.salesforce_query docker.docker_run github.get_check_run github.delete_check_run github.update_check_run +q136,Index a document in Elasticsearch,verb_product,elasticsearch.es_index_doc,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,346,1.0,1,605,1.0,elasticsearch.es_index_doc elasticsearch.es_get_doc elasticsearch.es_search playwright.browser_tab_close github.get_milestone github.delete_milestone github.get_hook github.delete_hook github.get_environment github.delete_environment +q137,Get an S3 object,verb_product,aws_s3.s3_get_object,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,355,1.0,1,668,1.0,aws_s3.s3_get_object aws_s3.s3_delete_object aws_s3.s3_put_object okta.okta_get_user asana.asana_get_task puppeteer.puppeteer_click elasticsearch.es_search okta.okta_create_user puppeteer.puppeteer_fill aws_s3.s3_list_objects +q138,List Cloudflare zones,verb_product,cloudflare.cf_list_zones,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,338,1.0,1,579,1.0,cloudflare.cf_list_zones cloudflare.cf_list_dns cloudflare.cf_purge_cache cloudflare.cf_create_dns clickup.clickup_list_tasks trello.trello_list_cards aws_s3.s3_list_buckets linear.linear_list_teams jira.jira_list_projects kubernetes.k8s_list_deployments +q139,Create a Stripe invoice,verb_product,stripe.stripe_create_invoice,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,353,1.0,1,669,1.0,stripe.stripe_create_invoice stripe.stripe_create_customer stripe.stripe_create_product stripe.stripe_create_refund stripe.stripe_create_price stripe.stripe_create_payment_intent stripe.stripe_list_invoices stripe.stripe_get_customer stripe.stripe_list_customers stripe.stripe_list_subscriptions +q140,List GitHub Actions workflows,verb_product,github.actions_list,1.0,1.0,1.0,1.0000,1.0000,1,1,293,1.0,1,457,1.0,1,790,1.0,github.actions_list github.actions_get github.actions_run_trigger github.actions_get_job_logs github.list_tags github.list_gists github.projects_list github.list_branches github.list_releases github.list_discussions +q141,Get GitHub Actions job logs,verb_product,github.actions_get_job_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,442,1.0,1,786,1.0,github.actions_get_job_logs github.actions_get github.actions_list gitlab.get_job github.actions_run_trigger kubernetes.k8s_pod_logs git.git_log gitlab.delete_job gitlab.update_job docker.docker_logs +q142,Create a GitHub repository,verb_product,github.create_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,235,1.0,1,406,1.0,1,874,1.0,github.create_repository github.delete_repository github.star_repository github.unstar_repository github.create_branch github.fork_repository github.create_issue github.list_repository_collaborators github.get_repository_tree github.list_tags +q143,Checkout a git branch,verb_product,git.git_checkout,1.0,1.0,1.0,1.0000,1.0000,1,1,266,1.0,1,429,1.0,1,1001,1.0,git.git_checkout git.git_create_branch github.create_branch git.git_branch gitlab.gitlab_create_merge_request github.update_pull_request_branch github.list_commits github.delete_file github.push_files github.create_or_update_file +q144,Show git log,identifier,git.git_log,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,299,1.0,1,600,1.0,git.git_log git.git_show git.git_diff_unstaged git.git_diff_staged git.git_status git.git_diff git.git_branch git.git_checkout git.git_commit git.git_add +q145,List postgres schemas,verb_product,postgres.list_schemas,1.0,1.0,1.0,1.0000,1.0000,1,1,160,1.0,1,277,1.0,1,515,1.0,postgres.list_schemas postgres.list_tables postgres.explain_query postgres.query postgres.describe_table clickup.clickup_list_tasks trello.trello_list_cards aws_s3.s3_list_buckets linear.linear_list_teams jira.jira_list_projects +q146,Open nodes in the knowledge graph,verb_product,memory.open_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,163,1.0,1,260,1.0,1,531,1.0,memory.open_nodes memory.search_nodes memory.read_graph memory.create_entities memory.delete_relations memory.create_relations memory.add_observations memory.delete_entities figma.figma_get_file_nodes playwright.browser_tab_list +q147,Get file metadata from the filesystem,paraphrase,filesystem.read_file,0.0,1.0,1.0,0.5000,0.6309,2,1,220,1.0,1,334,1.0,1,766,1.0,filesystem.read_file filesystem.get_file_info filesystem.read_text_file filesystem.read_media_file google_drive.gdrive_get_file figma.figma_get_file_nodes gitlab.gitlab_get_file figma.figma_get_images github.get_file_contents filesystem.write_file +q148,Write a new local file,paraphrase,filesystem.write_file,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,384,1.0,1,791,1.0,filesystem.write_file filesystem.read_file filesystem.read_text_file filesystem.edit_file filesystem.read_media_file playwright.browser_tab_new brave_search.brave_local_search github.issue_write figma.figma_get_file google_drive.gdrive_update_file +q149,Search GitHub issues,verb_product,github.search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,233,1.0,1,385,1.0,1,683,1.0,github.search_issues linear.linear_search_issues jira.jira_search_issues github.list_issues github.search_users github.search_orgs github.search_commits github.search_pull_requests gitlab.gitlab_list_issues github.search_code +q150,Get the authenticated GitHub user,paraphrase,github.get_teams,0.0,1.0,1.0,0.5000,0.6309,2,1,179,1.0,1,304,1.0,1,622,1.0,github.get_teams github.get_me github.list_notifications okta.okta_get_user github.get_latest_release slack.slack_get_user_profile linear.get_user github.get_repository_tree github.list_gists github.list_starred_repositories diff --git a/experiments/mcp-tool-search/results/bm25_ident_split.json b/experiments/mcp-tool-search/results/bm25_ident_split.json new file mode 100644 index 00000000..c786fb07 --- /dev/null +++ b/experiments/mcp-tool-search/results/bm25_ident_split.json @@ -0,0 +1,6969 @@ +{ + "summary": { + "n": 150, + "index_s": 0.0009321830000317277, + "rss_mb": 0.03515625, + "p50_ms": 0.14297149999720205, + "p95_ms": 0.2555590999463675, + "recall@1": 0.9066666666666666, + "recall@1_lo": 0.86, + "recall@1_hi": 0.9533333333333334, + "recall@3": 0.98, + "recall@3_lo": 0.9533333333333334, + "recall@3_hi": 1.0, + "recall@5": 0.9866666666666667, + "recall@5_lo": 0.9666666666666667, + "recall@5_hi": 1.0, + "mrr@10": 0.9424074074074075, + "mrr@10_lo": 0.91, + "mrr@10_hi": 0.971111111111111, + "ndcg@10": 0.9520969102241539, + "ndcg@10_lo": 0.9253719508213323, + "ndcg@10_hi": 0.9751765482010183 + }, + "meta": { + "name": "bm25_ident_split", + "needs_model": false, + "needs_api": false, + "notes": "BM25 + identifier splitting" + }, + "rows": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "github.issue_write", + "gitlab.gitlab_create_issue", + "github.issue_read", + "github.add_issue_comment", + "github.assign_copilot_to_issue", + "bitbucket.bitbucket_create_issue", + "github.update_issue_comment", + "gitlab.gitlab_get_issue", + "github.create_gist" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 380, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 584, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1035, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "kind": "verb_product", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_create_issue", + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "jira.create_board", + "jira.create_sprint", + "jira.create_filter", + "jira.create_webhook", + "jira.create_attachment", + "jira.create_version" + ], + "hit1": "jira.jira_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 281, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 466, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 951, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "kind": "verb_product", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "slack.slack_post_message", + "twilio.twilio_send_sms", + "slack.slack_add_reaction", + "gmail.gmail_send_email", + "discord.discord_add_reaction", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_search_messages", + "slack.slack_create_channel" + ], + "hit1": "discord.discord_send_message", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 760, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "kind": "resource", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "ranking": [ + "github.list_pull_requests", + "github.search_pull_requests", + "bitbucket.bitbucket_list_prs", + "gitlab.gitlab_list_merge_requests", + "github.create_pull_request", + "playwright.browser_tab_list", + "memory.open_nodes", + "github.pull_request_read", + "github.merge_pull_request", + "github.update_pull_request_branch" + ], + "hit1": "github.list_pull_requests", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 460, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 836, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "ranking": [ + "github.create_pull_request", + "github.request_copilot_review", + "github.pull_request_read", + "github.merge_pull_request", + "github.pull_request_review_write", + "github.update_pull_request", + "github.update_pull_request_branch", + "github.search_pull_requests", + "github.add_comment_to_pending_review", + "github.list_pull_requests" + ], + "hit1": "github.create_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 364, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 612, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1144, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "kind": "verb_product", + "gold": [ + "postgres.query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "mysql.mysql_query", + "sqlite.sqlite_query", + "postgres.explain_query", + "salesforce.salesforce_query", + "postgres.list_schemas", + "postgres.list_tables", + "postgres.describe_table", + "docker.docker_run", + "github.get_check_run" + ], + "hit1": "postgres.query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 592, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "kind": "paraphrase", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "ranking": [ + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "figma.figma_get_file_nodes", + "gitlab.gitlab_get_file", + "figma.figma_get_images", + "github.delete_file", + "memory.read_graph", + "brave_search.brave_local_search", + "github.get_file_contents" + ], + "hit1": "filesystem.read_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 809, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "kind": "paraphrase", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "ranking": [ + "google_drive.gdrive_list_folder", + "slack.list_files", + "github.push_files", + "filesystem.search_files", + "google_drive.gdrive_search", + "github.create_gist", + "cloudflare.cf_purge_cache", + "github.update_gist", + "filesystem.list_directory", + "filesystem.move_file" + ], + "hit1": "google_drive.gdrive_list_folder", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.1111111111111111, + "ndcg@10": 0.1922943146971571, + "hit_rank": 9, + "k3_calls": 3, + "k3_tokens": 717, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 756, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 734, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "ranking": [ + "memory.read_graph", + "memory.create_entities", + "memory.delete_relations", + "memory.create_relations", + "memory.add_observations", + "memory.open_nodes", + "memory.search_nodes", + "memory.delete_entities", + "postgres.list_schemas", + "time.get_current_time" + ], + "hit1": "memory.read_graph", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.5338931479009518, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 154, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 261, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 507, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "kind": "verb_product", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "ranking": [ + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "jira.jira_search_issues", + "memory.search_nodes", + "github.update_pull_request_branch", + "filesystem.list_directory_with_sizes", + "confluence.confluence_search", + "github.search_users", + "github.search_orgs", + "grafana.grafana_search_dashboards" + ], + "hit1": "brave_search.brave_web_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 410, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "kind": "verb_product", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "ranking": [ + "fetch.fetch", + "filesystem.read_file", + "playwright.browser_navigate", + "puppeteer.puppeteer_navigate", + "git.git_add", + "playwright.browser_snapshot", + "github.get_file_contents", + "git.git_status", + "docker.docker_logs", + "slack.slack_set_channel_topic" + ], + "hit1": "fetch.fetch", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "kind": "identifier", + "gold": [ + "git.git_status" + ], + "alts": [], + "ranking": [ + "git.git_status", + "git.git_show", + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_log", + "git.git_diff", + "sentry.sentry_update_issue", + "git.git_branch", + "git.git_checkout", + "zendesk.zendesk_list_tickets" + ], + "hit1": "git.git_status", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 613, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "kind": "paraphrase", + "gold": [ + "git.git_commit" + ], + "alts": [], + "ranking": [ + "git.git_commit", + "git.git_reset", + "git.git_diff_staged", + "github.get_commit", + "git.git_show", + "git.git_log", + "github.push_files" + ], + "hit1": "git.git_commit", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_comment", + "linear.linear_create_issue", + "linear.linear_update_issue", + "linear.linear_get_issue", + "linear.linear_assign_issue", + "linear.linear_create_project", + "bitbucket.bitbucket_create_issue", + "gitlab.gitlab_create_issue", + "linear.create_cycle", + "linear.create_label" + ], + "hit1": "linear.linear_create_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 917, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "kind": "paraphrase", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_project", + "asana.asana_get_task", + "clickup.clickup_get_task", + "asana.asana_update_task", + "clickup.clickup_update_task", + "asana.asana_create_task", + "clickup.clickup_create_task", + "mysql.mysql_list_tables", + "mysql.mysql_query", + "linear.get_cycle" + ], + "hit1": "linear.linear_create_project", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 146, + "k3_calls": 49, + "k3_tokens": 12414, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 30, + "k5_tokens": 12154, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11794, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "github.update_issue_comment", + "github.add_issue_comment", + "github.assign_copilot_to_issue", + "linear.linear_create_comment", + "jira.jira_add_comment", + "trello.trello_add_comment", + "github.issue_read", + "slack.slack_post_message", + "github.issue_write" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 307, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 478, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 922, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "kind": "ambiguous", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "ranking": [ + "github.merge_pull_request", + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_get_merge_request", + "github.update_pull_request_branch", + "github.pull_request_read", + "github.request_copilot_review", + "gitlab.gitlab_create_merge_request", + "github.pull_request_review_write", + "github.update_pull_request", + "github.create_pull_request" + ], + "hit1": "github.merge_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 468, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1121, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "kind": "verb_product", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "ranking": [ + "github.actions_run_trigger", + "github.actions_get", + "github.actions_list", + "github.actions_get_job_logs", + "circleci.circleci_trigger_pipeline", + "github.list_tags", + "github.search_users", + "github.search_orgs", + "github.get_tag", + "github.search_commits" + ], + "hit1": "github.actions_run_trigger", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 768, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "kind": "resource", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "ranking": [ + "github.list_dependabot_alerts", + "github.get_dependabot_alert", + "github.list_secret_scanning_alerts", + "github.list_code_scanning_alerts", + "redis.redis_get", + "kubernetes.k8s_get_pod", + "shopify.shopify_get_product", + "shopify.shopify_get_order", + "vercel.vercel_get_deployment", + "grafana.grafana_get_dashboard" + ], + "hit1": "github.list_dependabot_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 715, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "kind": "verb_product", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_issues", + "sentry.sentry_list_projects", + "sentry.sentry_list_events", + "gitlab.gitlab_list_issues", + "linear.linear_list_issues", + "sentry.sentry_get_issue", + "github.list_issues", + "sentry.sentry_update_issue", + "github.search_issues", + "linear.linear_search_issues" + ], + "hit1": "sentry.sentry_list_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 395, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_customer", + "stripe.stripe_create_invoice", + "stripe.stripe_get_customer", + "stripe.stripe_create_payment_intent", + "stripe.stripe_list_invoices", + "stripe.stripe_list_subscriptions", + "stripe.stripe_create_product", + "stripe.stripe_create_refund", + "stripe.stripe_create_price", + "stripe.stripe_list_customers" + ], + "hit1": "stripe.stripe_create_customer", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "kind": "paraphrase", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_refund", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_list_customers", + "stripe.stripe_create_product", + "stripe.stripe_list_subscriptions" + ], + "hit1": "stripe.stripe_create_refund", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "kind": "verb_product", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "ranking": [ + "gmail.gmail_send_email", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "twilio.twilio_send_sms", + "gmail.gmail_modify_email", + "gmail.gmail_list_labels", + "puppeteer.puppeteer_click", + "puppeteer.puppeteer_fill", + "gmail.gmail_search_emails", + "gmail.gmail_create_draft" + ], + "hit1": "gmail.gmail_send_email", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "kind": "verb_product", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "ranking": [ + "gmail.gmail_search_emails", + "gmail.gmail_list_labels", + "gmail.gmail_get_email", + "gmail.gmail_create_draft", + "gmail.gmail_trash_email", + "gmail.gmail_send_email", + "gmail.gmail_modify_email", + "mysql.mysql_list_tables", + "mysql.mysql_query", + "brave_search.brave_web_search" + ], + "hit1": "gmail.gmail_search_emails", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 691, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "kind": "paraphrase", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_update_file", + "google_drive.gdrive_share_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_create_file", + "slack.slack_upload_file", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "aws_s3.s3_put_object", + "google_calendar.gcal_list_calendars", + "filesystem.edit_file" + ], + "hit1": "google_drive.gdrive_update_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 523, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 446, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_create_event", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "github.pull_request_review_write", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project", + "trello.trello_create_board", + "stripe.stripe_create_customer" + ], + "hit1": "google_calendar.gcal_create_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 801, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_list_pods", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_apply", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_delete", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_pod_logs", + "clickup.clickup_list_tasks" + ], + "hit1": "kubernetes.k8s_list_pods", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_get_pod", + "docker.docker_logs", + "datadog.datadog_list_logs", + "github.actions_get_job_logs", + "aws_s3.s3_get_object", + "elasticsearch.es_get_doc", + "github.get_tag", + "slack.slack_get_channel_history", + "figma.figma_get_file_nodes" + ], + "hit1": "kubernetes.k8s_pod_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 830, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "ranking": [ + "docker.docker_run", + "docker.docker_exec", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_images", + "docker.docker_ps", + "docker.docker_build", + "github.get_check_run", + "github.delete_check_run" + ], + "hit1": "docker.docker_run", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "kind": "verb_product", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "ranking": [ + "docker.docker_logs", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_exec", + "docker.docker_run", + "kubernetes.k8s_pod_logs", + "git.git_log", + "docker.docker_images", + "docker.docker_ps", + "docker.docker_build" + ], + "hit1": "docker.docker_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 649, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "kind": "ambiguous", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "ranking": [ + "playwright.browser_navigate", + "puppeteer.puppeteer_navigate", + "playwright.browser_tab_new", + "fetch.fetch", + "twilio.twilio_make_call", + "playwright.browser_evaluate", + "playwright.browser_press_key", + "playwright.browser_snapshot", + "playwright.browser_click", + "playwright.browser_wait_for" + ], + "hit1": "playwright.browser_navigate", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 346, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 617, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "kind": "ambiguous", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "ranking": [ + "playwright.browser_take_screenshot", + "puppeteer.puppeteer_screenshot", + "playwright.browser_snapshot", + "filesystem.read_file", + "playwright.browser_evaluate", + "playwright.browser_press_key", + "playwright.browser_click", + "notion.notion_get_page", + "notion.notion_update_page", + "git.git_status" + ], + "hit1": "playwright.browser_take_screenshot", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 608, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_get_merge_request", + "gitlab.gitlab_create_merge_request", + "github.merge_pull_request", + "gitlab.gitlab_list_merge_requests", + "gitlab.gitlab_create_issue", + "gitlab.gitlab_create_pipeline", + "github.create_pull_request", + "github.request_copilot_review", + "github.pull_request_review_write" + ], + "hit1": "gitlab.gitlab_merge_merge_request", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1031, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "kind": "verb_product", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "ranking": [ + "notion.notion_search", + "notion.notion_list_users", + "notion.notion_get_page", + "notion.notion_get_block", + "brave_search.brave_web_search", + "notion.notion_create_database", + "notion.notion_update_page", + "notion.notion_query_database", + "notion.notion_get_comments", + "confluence.confluence_search" + ], + "hit1": "notion.notion_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 671, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "kind": "verb_product", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "ranking": [ + "notion.notion_create_page", + "notion.notion_get_page", + "notion.notion_update_page", + "notion.notion_get_comments", + "notion.notion_create_database", + "confluence.confluence_create_page", + "notion.create_comment", + "notion.create_users_token", + "notion.create_data_source", + "confluence.confluence_get_page" + ], + "hit1": "notion.notion_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 822, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "kind": "verb_product", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "ranking": [ + "notion.notion_query_database", + "notion.notion_create_database", + "notion.notion_search", + "notion.notion_create_page", + "notion.notion_list_users", + "notion.notion_get_page", + "notion.notion_get_block", + "notion.notion_update_page", + "notion.notion_get_comments", + "prometheus.prom_query" + ], + "hit1": "notion.notion_query_database", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 668, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_search", + "jira.jira_create_issue", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project", + "trello.trello_create_board", + "stripe.stripe_create_customer" + ], + "hit1": "zendesk.zendesk_create_ticket", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_create_deal", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_search_contacts", + "github.create_gist", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.create_or_update_file" + ], + "hit1": "hubspot.hubspot_create_contact", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 899, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "kind": "verb_product", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_convert_lead", + "salesforce.salesforce_create_lead", + "salesforce.salesforce_create_record", + "salesforce.salesforce_list_opportunities", + "salesforce.salesforce_query", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_update_record", + "salesforce.salesforce_get_record", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project" + ], + "hit1": "salesforce.salesforce_convert_lead", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 710, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "kind": "identifier", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "postgres.query", + "docker.docker_run", + "github.get_check_run", + "github.delete_check_run", + "github.update_check_run", + "prometheus.prom_query", + "github.create_check_run" + ], + "hit1": "salesforce.salesforce_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 332, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "kind": "verb_product", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "ranking": [ + "shopify.shopify_list_orders", + "shopify.shopify_list_customers", + "shopify.shopify_list_products", + "shopify.shopify_get_product", + "shopify.shopify_get_order", + "shopify.shopify_update_product", + "shopify.shopify_create_order", + "shopify.shopify_create_product", + "clickup.clickup_list_tasks", + "trello.trello_list_cards" + ], + "hit1": "shopify.shopify_list_orders", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project", + "trello.trello_create_board", + "stripe.stripe_create_customer" + ], + "hit1": "pagerduty.pagerduty_create_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 696, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "kind": "paraphrase", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_list_oncalls", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_create_incident", + "twilio.twilio_make_call", + "slack.slack_create_channel", + "filesystem.list_allowed_directories", + "memory.search_nodes" + ], + "hit1": "pagerduty.pagerduty_list_oncalls", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "kind": "verb_product", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "ranking": [ + "datadog.datadog_query_metrics", + "datadog.datadog_list_monitors", + "datadog.datadog_create_monitor", + "datadog.datadog_list_logs", + "datadog.datadog_get_monitor", + "prometheus.prom_list_metrics", + "prometheus.prom_query", + "prometheus.prom_query_range", + "postgres.explain_query", + "salesforce.salesforce_query" + ], + "hit1": "datadog.datadog_query_metrics", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "kind": "verb_product", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "ranking": [ + "datadog.datadog_list_logs", + "datadog.datadog_list_monitors", + "datadog.datadog_get_monitor", + "datadog.datadog_create_monitor", + "datadog.datadog_query_metrics", + "docker.docker_logs", + "kubernetes.k8s_pod_logs", + "github.actions_get_job_logs", + "brave_search.brave_web_search", + "confluence.confluence_search" + ], + "hit1": "datadog.datadog_list_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 855, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_list_buckets", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "linear.linear_list_teams", + "jira.jira_list_projects", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "gmail.gmail_list_labels", + "cloudflare.cf_list_zones", + "vercel.vercel_list_projects" + ], + "hit1": "aws_s3.s3_list_buckets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 258, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 474, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "kind": "paraphrase", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_put_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_get_object", + "slack.slack_upload_file", + "github.assign_copilot_to_issue", + "twilio.twilio_send_sms", + "gmail.gmail_send_email", + "slack.slack_add_reaction", + "slack.slack_reply_to_thread", + "puppeteer.puppeteer_click" + ], + "hit1": "aws_s3.s3_put_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 850, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "cloudflare.cf_list_dns", + "cloudflare.cf_create_dns" + ], + "hit1": "cloudflare.cf_purge_cache", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "salesforce.salesforce_create_record", + "cloudflare.cf_list_zones", + "cloudflare.cf_purge_cache", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_update_record", + "salesforce.salesforce_get_record", + "git.git_commit", + "stripe.stripe_create_invoice" + ], + "hit1": "cloudflare.cf_create_dns", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 708, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_deployments", + "vercel.vercel_list_projects", + "kubernetes.k8s_list_deployments", + "vercel.vercel_get_deployment", + "vercel.vercel_create_project", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "aws_s3.s3_list_buckets", + "linear.linear_list_teams", + "jira.jira_list_projects" + ], + "hit1": "vercel.vercel_list_deployments", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 173, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 529, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "ranking": [ + "supabase.supabase_insert", + "mongodb.mongo_insert", + "supabase.supabase_list_tables", + "supabase.supabase_select", + "supabase.supabase_invoke_fn", + "supabase.supabase_update", + "playwright.browser_type" + ], + "hit1": "supabase.supabase_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 203, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 347, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "kind": "paraphrase", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "ranking": [ + "figma.figma_get_images", + "figma.figma_get_file_nodes", + "docker.docker_images", + "figma.figma_get_file", + "figma.figma_get_comments", + "figma.figma_post_comment", + "memory.delete_observations", + "aws_s3.s3_get_object", + "memory.delete_relations", + "docker.docker_logs" + ], + "hit1": "figma.figma_get_images", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 637, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "kind": "verb_product", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "ranking": [ + "twilio.twilio_send_sms", + "twilio.twilio_list_messages", + "gmail.gmail_send_email", + "twilio.twilio_make_call", + "puppeteer.puppeteer_click", + "puppeteer.puppeteer_fill", + "discord.discord_send_message", + "filesystem.list_directory_with_sizes", + "puppeteer.puppeteer_navigate", + "puppeteer.puppeteer_screenshot" + ], + "hit1": "twilio.twilio_send_sms", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "kind": "verb_product", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "ranking": [ + "okta.okta_create_user", + "okta.okta_get_user", + "okta.okta_deactivate_user", + "okta.okta_list_users", + "linear.create_user", + "slack.slack_get_user_profile", + "asana.asana_create_project", + "linear.get_user", + "linear.delete_user", + "linear.update_user" + ], + "hit1": "okta.okta_create_user", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 622, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query", + "prometheus.prom_query_range", + "prometheus.prom_list_metrics", + "datadog.datadog_query_metrics", + "postgres.explain_query", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "postgres.query", + "notion.notion_query_database" + ], + "hit1": "prometheus.prom_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "kind": "verb_product", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "ranking": [ + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "grafana.grafana_create_annotation", + "brave_search.brave_web_search", + "confluence.confluence_search", + "github.search_users", + "github.search_orgs", + "brave_search.brave_local_search", + "github.search_commits", + "slack.slack_search_messages" + ], + "hit1": "grafana.grafana_search_dashboards", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "kind": "verb_product", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "ranking": [ + "circleci.circleci_trigger_pipeline", + "circleci.circleci_get_pipeline", + "gitlab.gitlab_create_pipeline", + "circleci.circleci_list_pipelines", + "github.actions_run_trigger", + "gitlab.get_pipeline", + "gitlab.delete_pipeline", + "gitlab.update_pipeline", + "gitlab.create_pipeline", + "hubspot.hubspot_create_deal" + ], + "hit1": "circleci.circleci_trigger_pipeline", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "kind": "identifier", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "ranking": [ + "terraform.tf_plan", + "terraform.tf_apply", + "terraform.tf_state_list", + "postgres.explain_query", + "docker.docker_run", + "github.get_check_run", + "github.delete_check_run", + "github.update_check_run", + "github.create_check_run", + "salesforce.salesforce_query" + ], + "hit1": "terraform.tf_plan", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "kind": "verb_product", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "ranking": [ + "asana.asana_create_task", + "asana.asana_get_task", + "asana.asana_update_task", + "asana.asana_create_project", + "clickup.clickup_create_task", + "asana.asana_list_tasks", + "clickup.clickup_get_task", + "clickup.clickup_update_task", + "okta.okta_create_user", + "github.create_issue" + ], + "hit1": "asana.asana_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "ranking": [ + "trello.trello_create_card", + "trello.trello_add_comment", + "trello.trello_create_board", + "trello.trello_list_cards", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project", + "stripe.stripe_create_customer", + "stripe.stripe_create_product", + "salesforce.salesforce_create_record", + "okta.okta_create_user" + ], + "hit1": "trello.trello_create_card", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 679, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_create_task", + "clickup.clickup_get_task", + "clickup.clickup_update_task", + "clickup.clickup_list_tasks", + "asana.asana_create_task", + "asana.asana_get_task", + "asana.asana_update_task", + "puppeteer.puppeteer_click", + "playwright.browser_click", + "stripe.stripe_create_invoice" + ], + "hit1": "clickup.clickup_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 415, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "kind": "verb_product", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "ranking": [ + "confluence.confluence_create_page", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "confluence.confluence_search", + "notion.notion_create_page", + "notion.notion_get_page", + "notion.notion_update_page", + "playwright.browser_take_screenshot", + "playwright.browser_evaluate", + "playwright.browser_press_key" + ], + "hit1": "confluence.confluence_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "kind": "verb_product", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "discord.discord_add_reaction", + "discord.discord_get_channel", + "discord.discord_list_channels", + "discord.discord_list_members", + "twilio.twilio_send_sms", + "gmail.gmail_send_email", + "slack.slack_post_message", + "gmail.gmail_get_email", + "gmail.gmail_trash_email" + ], + "hit1": "discord.discord_send_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_search", + "elasticsearch.es_get_doc", + "elasticsearch.es_index_doc", + "brave_search.brave_web_search", + "confluence.confluence_search", + "github.search_users", + "github.search_orgs", + "grafana.grafana_search_dashboards", + "brave_search.brave_local_search", + "github.search_commits" + ], + "hit1": "elasticsearch.es_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 672, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "redis.redis_set", + "redis.redis_delete", + "redis.redis_keys", + "jira.jira_get_issue", + "aws_s3.s3_get_object", + "figma.figma_get_file", + "playwright.browser_press_key", + "figma.figma_get_comments", + "figma.figma_get_file_nodes" + ], + "hit1": "redis.redis_get", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 589, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_find", + "mongodb.mongo_insert", + "mongodb.mongo_delete", + "mongodb.mongo_update", + "mongodb.mongo_list_collections" + ], + "hit1": "mongodb.mongo_find", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "kind": "verb_product", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "ranking": [ + "sqlite.sqlite_list_tables", + "sqlite.sqlite_describe_table", + "sqlite.sqlite_query", + "mysql.mysql_list_tables", + "supabase.supabase_list_tables", + "postgres.list_tables", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "aws_s3.s3_list_buckets", + "linear.linear_list_teams" + ], + "hit1": "sqlite.sqlite_list_tables", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 156, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 243, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 496, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "kind": "verb_product", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "ranking": [ + "postgres.describe_table", + "sqlite.sqlite_describe_table", + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update", + "postgres.explain_query", + "postgres.list_schemas", + "postgres.query", + "postgres.list_tables" + ], + "hit1": "postgres.describe_table", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "bitbucket.bitbucket_create_issue", + "github.issue_write", + "gitlab.gitlab_create_issue", + "github.issue_read", + "github.add_issue_comment", + "linear.linear_create_issue", + "jira.jira_create_issue", + "linear.linear_create_comment", + "github.assign_copilot_to_issue" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 378, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 568, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1091, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "ranking": [ + "bitbucket.bitbucket_create_issue", + "gitlab.gitlab_create_issue", + "github.create_issue", + "linear.linear_create_issue", + "jira.jira_create_issue", + "linear.linear_create_comment", + "github.issue_write", + "sentry.sentry_get_issue", + "jira.jira_get_issue", + "jira.jira_assign_issue" + ], + "hit1": "bitbucket.bitbucket_create_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5805078709398304, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 332, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 579, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 971, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "kind": "identifier", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "ranking": [ + "filesystem.create_directory", + "filesystem.list_directory", + "filesystem.list_directory_with_sizes", + "filesystem.get_file_info", + "filesystem.directory_tree", + "github.get_file_contents", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "aws_s3.s3_list_buckets", + "linear.linear_list_teams" + ], + "hit1": "filesystem.create_directory", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 607, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "kind": "paraphrase", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "ranking": [ + "sequential_thinking.sequentialthinking" + ], + "hit1": "sequential_thinking.sequentialthinking", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "kind": "paraphrase", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "ranking": [ + "time.get_current_time", + "time.convert_time", + "playwright.browser_wait_for", + "google_calendar.gcal_list_events", + "slack.slack_create_channel", + "filesystem.list_allowed_directories", + "grafana.grafana_create_annotation", + "jira.jira_add_worklog", + "github.get_milestone", + "github.delete_milestone" + ], + "hit1": "time.get_current_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "kind": "verb_product", + "gold": [ + "time.convert_time" + ], + "alts": [], + "ranking": [ + "time.convert_time", + "salesforce.salesforce_convert_lead", + "time.get_current_time", + "playwright.browser_wait_for", + "google_calendar.gcal_list_events", + "memory.create_relations", + "git.git_diff", + "grafana.grafana_create_annotation", + "jira.jira_add_worklog" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "kind": "verb_product", + "gold": [ + "github.star_repository" + ], + "alts": [], + "ranking": [ + "github.star_repository", + "github.list_tags", + "github.delete_repository", + "github.unstar_repository", + "github.get_tag", + "github.fork_repository", + "github.search_issues", + "github.actions_list", + "github.get_label", + "github.list_branches" + ], + "hit1": "github.star_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 225, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 771, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "kind": "resource", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "ranking": [ + "github.fork_repository", + "github.delete_repository", + "github.star_repository", + "github.unstar_repository", + "github.create_repository", + "github.list_repository_collaborators", + "github.get_repository_tree", + "github.list_branches", + "github.list_releases", + "github.list_dependabot_alerts" + ], + "hit1": "github.fork_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 792, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "kind": "verb_product", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "ranking": [ + "github.get_latest_release", + "github.get_release_by_tag", + "gitlab.get_release", + "github.get_teams", + "github.get_repository_tree", + "github.get_me", + "gitlab.delete_release", + "gitlab.update_release", + "github.get_tag", + "gitlab.create_release" + ], + "hit1": "github.get_latest_release", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 383, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "kind": "resource", + "gold": [ + "github.create_gist" + ], + "alts": [], + "ranking": [ + "github.create_gist", + "github.get_gist", + "github.update_gist", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project", + "trello.trello_create_board", + "stripe.stripe_create_customer", + "stripe.stripe_create_product", + "salesforce.salesforce_create_record", + "okta.okta_create_user" + ], + "hit1": "github.create_gist", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 649, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "kind": "verb_product", + "gold": [ + "github.search_code" + ], + "alts": [], + "ranking": [ + "github.search_code", + "github.list_code_scanning_alerts", + "github.get_code_scanning_alert", + "github.search_users", + "github.search_orgs", + "github.search_commits", + "github.search_issues", + "github.search_pull_requests", + "github.search_repositories", + "hubspot.hubspot_search_contacts" + ], + "hit1": "github.search_code", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "kind": "verb_product", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "ranking": [ + "github.list_notifications", + "github.mark_all_notifications_read", + "github.list_tags", + "github.actions_list", + "github.list_gists", + "github.projects_list", + "github.list_branches", + "github.list_releases", + "github.list_discussions", + "github.list_labels" + ], + "hit1": "github.list_notifications", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 311, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "kind": "verb_product", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "ranking": [ + "github.assign_copilot_to_issue", + "linear.linear_assign_issue", + "jira.jira_assign_issue", + "jira.jira_update_issue", + "jira.jira_transition_issue", + "github.add_issue_comment", + "linear.linear_create_comment", + "jira.jira_add_comment", + "github.request_copilot_review", + "jira.jira_add_worklog" + ], + "hit1": "github.assign_copilot_to_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 833, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "kind": "resource", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "ranking": [ + "github.list_secret_scanning_alerts", + "github.get_secret_scanning_alert", + "github.list_code_scanning_alerts", + "github.get_code_scanning_alert", + "github.list_dependabot_alerts", + "redis.redis_get", + "kubernetes.k8s_get_pod", + "shopify.shopify_get_product", + "shopify.shopify_get_order", + "vercel.vercel_get_deployment" + ], + "hit1": "github.list_secret_scanning_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "ranking": [ + "github.create_or_update_file", + "github.projects_write", + "github.label_write", + "github.get_file_contents", + "filesystem.write_file", + "github.update_issue_comment", + "github.issue_write", + "slack.update_file", + "google_drive.gdrive_update_file", + "figma.figma_get_comments" + ], + "hit1": "github.create_or_update_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 369, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 546, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1001, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "kind": "resource", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "ranking": [ + "github.list_repository_collaborators", + "github.list_commits", + "trello.trello_list_cards", + "pagerduty.pagerduty_list_oncalls", + "github.get_discussion_comments", + "github.update_issue_comment", + "github.request_copilot_review", + "github.actions_list", + "bitbucket.bitbucket_list_prs", + "github.create_pull_request" + ], + "hit1": "github.list_repository_collaborators", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 278, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 433, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 932, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "kind": "verb_product", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "ranking": [ + "slack.slack_reply_to_thread", + "slack.slack_get_thread_replies", + "github.get_notification_details", + "slack.get_reminder", + "slack.delete_reminder", + "slack.get_canvase", + "slack.delete_canvase", + "slack.get_bookmark", + "slack.delete_bookmark", + "slack.get_usergroup" + ], + "hit1": "slack.slack_reply_to_thread", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "kind": "verb_product", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "ranking": [ + "slack.slack_add_reaction", + "discord.discord_add_reaction", + "trello.trello_add_comment", + "gmail.gmail_modify_email", + "playwright.browser_click", + "jira.jira_add_comment", + "memory.add_observations", + "git.git_add", + "jira.jira_add_worklog", + "github.assign_copilot_to_issue" + ], + "hit1": "slack.slack_add_reaction", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 271, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 811, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "kind": "verb_product", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "ranking": [ + "slack.slack_search_messages", + "gmail.gmail_search_emails", + "slack.slack_get_channel_history", + "twilio.twilio_list_messages", + "brave_search.brave_web_search", + "confluence.confluence_search", + "github.search_users", + "github.search_orgs", + "grafana.grafana_search_dashboards", + "brave_search.brave_local_search" + ], + "hit1": "slack.slack_search_messages", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 664, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "kind": "verb_product", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "ranking": [ + "slack.slack_create_channel", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_get_channel_history", + "slack.slack_set_channel_topic", + "slack.slack_post_message", + "slack.slack_upload_file", + "slack.slack_reply_to_thread", + "slack.slack_get_thread_replies", + "slack.slack_add_reaction" + ], + "hit1": "slack.slack_create_channel", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "kind": "verb_product", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_transition_issue", + "jira.jira_get_issue", + "jira.jira_assign_issue", + "jira.jira_update_issue", + "jira.jira_add_comment", + "jira.jira_add_worklog", + "jira.jira_create_issue", + "sentry.sentry_get_issue", + "gitlab.gitlab_get_issue", + "sentry.sentry_update_issue" + ], + "hit1": "jira.jira_transition_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 778, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "kind": "identifier", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "ranking": [ + "jira.jira_search_issues", + "filesystem.list_directory_with_sizes", + "brave_search.brave_web_search", + "confluence.confluence_search", + "github.search_users", + "github.search_orgs", + "grafana.grafana_search_dashboards", + "jira.jira_list_projects", + "brave_search.brave_local_search", + "github.search_commits" + ], + "hit1": "jira.jira_search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 241, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 619, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "kind": "verb_product", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "ranking": [ + "jira.jira_add_worklog", + "jira.jira_add_comment", + "memory.add_observations", + "git.git_add", + "discord.discord_add_reaction", + "github.add_issue_comment", + "slack.slack_add_reaction", + "github.add_comment_to_pending_review", + "jira.jira_transition_issue", + "linear.linear_create_comment" + ], + "hit1": "jira.jira_add_worklog", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 236, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 874, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "kind": "verb_product", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "ranking": [ + "linear.linear_list_teams", + "github.get_teams", + "linear.linear_list_projects", + "linear.linear_list_issues", + "linear.list_cycles", + "linear.list_labels", + "linear.list_users", + "linear.list_comments", + "linear.list_attachments", + "linear.linear_search_issues" + ], + "hit1": "linear.linear_list_teams", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 137, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 646, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "ranking": [ + "linear.linear_create_comment", + "github.update_issue_comment", + "linear.get_comment", + "linear.delete_comment", + "linear.update_comment", + "github.add_issue_comment", + "jira.jira_add_comment", + "linear.linear_update_issue", + "linear.linear_get_issue", + "linear.linear_assign_issue" + ], + "hit1": "linear.linear_create_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "kind": "paraphrase", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "ranking": [ + "filesystem.move_file", + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "brave_search.brave_local_search", + "gmail.gmail_trash_email", + "figma.figma_get_file", + "google_drive.gdrive_update_file", + "google_drive.gdrive_share_file", + "slack.get_file" + ], + "hit1": "filesystem.move_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "kind": "paraphrase", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "ranking": [ + "filesystem.search_files", + "linear.linear_search_issues", + "google_drive.gdrive_search", + "redis.redis_keys", + "github.search_repositories", + "slack.list_files", + "brave_search.brave_web_search", + "github.push_files", + "confluence.confluence_search", + "github.search_users" + ], + "hit1": "filesystem.search_files", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "kind": "paraphrase", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "ranking": [ + "filesystem.edit_file", + "filesystem.list_directory_with_sizes", + "filesystem.write_file", + "filesystem.move_file", + "playwright.browser_tab_new", + "slack.get_file", + "slack.delete_file", + "twilio.twilio_make_call", + "slack.update_file", + "figma.figma_get_file" + ], + "hit1": "filesystem.edit_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "ranking": [ + "memory.read_graph", + "memory.delete_relations", + "memory.create_entities", + "memory.create_relations", + "memory.add_observations", + "memory.open_nodes", + "memory.delete_entities", + "memory.search_nodes", + "filesystem.read_file", + "filesystem.read_text_file" + ], + "hit1": "memory.read_graph", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 259, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 561, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "kind": "verb_product", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "memory.open_nodes", + "figma.figma_get_file_nodes", + "brave_search.brave_web_search", + "confluence.confluence_search", + "github.search_users", + "github.search_orgs", + "grafana.grafana_search_dashboards", + "brave_search.brave_local_search", + "github.search_commits" + ], + "hit1": "memory.search_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 635, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "kind": "verb_product", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "ranking": [ + "git.git_diff_unstaged", + "git.git_diff", + "git.git_diff_staged", + "git.git_show", + "git.git_log", + "git.git_status", + "git.git_branch", + "git.git_checkout", + "git.git_commit", + "git.git_add" + ], + "hit1": "git.git_diff_unstaged", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 313, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 602, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "kind": "identifier", + "gold": [ + "git.git_add" + ], + "alts": [], + "ranking": [ + "git.git_add", + "filesystem.list_directory_with_sizes", + "salesforce.salesforce_list_opportunities", + "github.push_files", + "github.add_issue_comment", + "github.create_gist", + "slack.list_files", + "github.update_gist", + "jira.jira_add_comment", + "github.add_comment_to_pending_review" + ], + "hit1": "git.git_add", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 197, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 872, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "kind": "verb_product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "ranking": [ + "shopify.shopify_create_product", + "shopify.shopify_get_product", + "shopify.shopify_update_product", + "shopify.shopify_create_order", + "stripe.stripe_create_product", + "shopify.shopify_get_order", + "shopify.shopify_list_customers", + "shopify.shopify_list_products", + "shopify.shopify_list_orders", + "stripe.stripe_create_price" + ], + "hit1": "shopify.shopify_create_product", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "kind": "verb_product", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "ranking": [ + "stripe.stripe_cancel_subscription", + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_list_customers", + "stripe.stripe_create_product", + "stripe.stripe_list_subscriptions", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_refund" + ], + "hit1": "stripe.stripe_cancel_subscription", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 167, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 285, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 610, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "kind": "resource", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_refund", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project", + "trello.trello_create_board", + "stripe.stripe_create_customer", + "stripe.stripe_create_product", + "salesforce.salesforce_create_record", + "okta.okta_create_user" + ], + "hit1": "stripe.stripe_create_payment_intent", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 654, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "kind": "verb_product", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "ranking": [ + "datadog.datadog_create_monitor", + "datadog.datadog_get_monitor", + "datadog.datadog_list_monitors", + "datadog.datadog_list_logs", + "datadog.datadog_query_metrics", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project", + "trello.trello_create_board", + "stripe.stripe_create_customer", + "stripe.stripe_create_product" + ], + "hit1": "datadog.datadog_create_monitor", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "kind": "paraphrase", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "ranking": [ + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "sentry.sentry_list_events", + "pagerduty.pagerduty_resolve_incident", + "sentry.sentry_list_projects", + "sentry.sentry_list_issues", + "jira.jira_get_issue", + "jira.jira_assign_issue", + "jira.jira_update_issue", + "gitlab.gitlab_get_issue" + ], + "hit1": "sentry.sentry_get_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_scale_deployment", + "vercel.vercel_get_deployment", + "kubernetes.k8s_apply", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_delete", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_pod_logs" + ], + "hit1": "kubernetes.k8s_scale_deployment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_apply", + "terraform.tf_apply", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_delete", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_pod_logs" + ], + "hit1": "kubernetes.k8s_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 308, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "ranking": [ + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_exec", + "docker.docker_run", + "docker.docker_images", + "docker.docker_ps", + "docker.docker_build", + "kubernetes.k8s_pod_logs" + ], + "hit1": "docker.docker_stop", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "kind": "verb_product", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "ranking": [ + "docker.docker_build", + "docker.docker_run", + "docker.docker_images", + "docker.docker_stop", + "docker.docker_ps", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_exec", + "filesystem.read_media_file" + ], + "hit1": "docker.docker_build", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "kind": "verb_product", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "ranking": [ + "playwright.browser_click", + "puppeteer.puppeteer_click", + "playwright.browser_type", + "playwright.browser_take_screenshot", + "playwright.browser_snapshot", + "clickup.clickup_list_tasks", + "clickup.clickup_get_task", + "playwright.browser_tab_list", + "playwright.browser_tab_close", + "clickup.clickup_create_task" + ], + "hit1": "playwright.browser_click", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 612, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "kind": "paraphrase", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "ranking": [ + "playwright.browser_type", + "playwright.browser_wait_for", + "puppeteer.puppeteer_fill", + "playwright.browser_tab_list", + "playwright.browser_tab_close", + "playwright.browser_navigate", + "playwright.browser_tab_new", + "filesystem.read_text_file", + "supabase.supabase_insert", + "mongodb.mongo_insert" + ], + "hit1": "playwright.browser_type", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 648, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_list_projects", + "github.projects_list", + "jira.jira_list_projects", + "vercel.vercel_list_projects", + "linear.linear_list_projects", + "sentry.sentry_list_projects", + "gitlab.gitlab_list_issues", + "gitlab.gitlab_list_merge_requests", + "github.projects_get", + "gitlab.gitlab_create_issue" + ], + "hit1": "gitlab.gitlab_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 255, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 609, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "kind": "verb_product", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "ranking": [ + "bitbucket.bitbucket_create_pr", + "bitbucket.bitbucket_list_prs", + "bitbucket.bitbucket_create_issue", + "github.pull_request_review_write", + "github.pull_request_read", + "github.merge_pull_request", + "github.request_copilot_review", + "github.create_pull_request", + "github.update_pull_request_branch", + "github.update_pull_request" + ], + "hit1": "bitbucket.bitbucket_create_pr", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 546, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1139, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "kind": "verb_product", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "ranking": [ + "gmail.gmail_create_draft", + "gmail.gmail_list_labels", + "gmail.gmail_search_emails", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "gmail.gmail_send_email", + "gmail.gmail_modify_email", + "github.create_pull_request", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project" + ], + "hit1": "gmail.gmail_create_draft", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "kind": "verb_product", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_share_file", + "google_drive.gdrive_update_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "google_calendar.gcal_list_calendars", + "figma.figma_get_file", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event" + ], + "hit1": "google_drive.gdrive_share_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "kind": "verb_product", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "kubernetes.k8s_get_events", + "sentry.sentry_list_events", + "google_drive.gdrive_list_folder", + "google_drive.gdrive_search", + "google_drive.gdrive_update_file" + ], + "hit1": "google_calendar.gcal_list_events", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "redis.redis_delete", + "kubernetes.k8s_delete", + "aws_s3.s3_delete_object", + "github.delete_milestone", + "github.delete_hook" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 735, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_projects", + "vercel.vercel_list_deployments", + "jira.jira_list_projects", + "linear.linear_list_projects", + "sentry.sentry_list_projects", + "vercel.vercel_get_deployment", + "vercel.vercel_create_project", + "gitlab.gitlab_list_projects", + "github.projects_list", + "github.projects_get" + ], + "hit1": "vercel.vercel_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 248, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "ranking": [ + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update", + "supabase.supabase_list_tables", + "supabase.supabase_invoke_fn", + "memory.delete_observations", + "aws_s3.s3_get_object", + "memory.delete_relations", + "docker.docker_logs", + "elasticsearch.es_get_doc" + ], + "hit1": "supabase.supabase_select", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "kind": "verb_product", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "figma.figma_get_file", + "figma.figma_get_comments", + "slack.slack_post_message", + "figma.figma_get_images", + "figma.figma_get_file_nodes", + "linear.get_comment", + "linear.delete_comment", + "notion.get_comment", + "notion.delete_comment" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 646, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "kind": "verb_product", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "ranking": [ + "okta.okta_list_users", + "okta.okta_create_user", + "okta.okta_get_user", + "okta.okta_deactivate_user", + "notion.notion_list_users", + "slack.slack_list_users", + "linear.list_users", + "notion.list_users_tokens", + "slack.slack_get_users", + "github.search_users" + ], + "hit1": "okta.okta_list_users", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 271, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query_range", + "prometheus.prom_query", + "prometheus.prom_list_metrics", + "datadog.datadog_query_metrics", + "postgres.explain_query", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "postgres.query", + "notion.notion_query_database" + ], + "hit1": "prometheus.prom_query_range", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "kind": "verb_product", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "ranking": [ + "grafana.grafana_create_annotation", + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project", + "trello.trello_create_board", + "stripe.stripe_create_customer", + "stripe.stripe_create_product", + "salesforce.salesforce_create_record", + "okta.okta_create_user" + ], + "hit1": "grafana.grafana_create_annotation", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 638, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "kind": "identifier", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "ranking": [ + "terraform.tf_apply", + "kubernetes.k8s_apply", + "terraform.tf_plan", + "terraform.tf_state_list" + ], + "hit1": "terraform.tf_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "kind": "verb_product", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "ranking": [ + "asana.asana_list_tasks", + "clickup.clickup_list_tasks", + "asana.asana_get_task", + "asana.asana_create_project", + "asana.asana_create_task", + "asana.asana_update_task", + "trello.trello_list_cards", + "aws_s3.s3_list_buckets", + "linear.linear_list_teams", + "jira.jira_list_projects" + ], + "hit1": "asana.asana_list_tasks", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 642, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "ranking": [ + "trello.trello_add_comment", + "trello.trello_create_card", + "trello.trello_list_cards", + "trello.trello_create_board", + "github.update_issue_comment", + "figma.figma_post_comment", + "linear.get_comment", + "linear.delete_comment", + "notion.get_comment", + "notion.delete_comment" + ], + "hit1": "trello.trello_add_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 422, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_update_task", + "clickup.clickup_get_task", + "clickup.clickup_create_task", + "asana.asana_update_task", + "clickup.clickup_list_tasks", + "asana.asana_get_task", + "puppeteer.puppeteer_click", + "playwright.browser_click", + "asana.asana_create_task", + "mongodb.mongo_update" + ], + "hit1": "clickup.clickup_update_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 761, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "kind": "verb_product", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "ranking": [ + "confluence.confluence_search", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "confluence.confluence_create_page", + "brave_search.brave_web_search", + "github.search_users", + "github.search_orgs", + "grafana.grafana_search_dashboards", + "brave_search.brave_local_search", + "github.search_commits" + ], + "hit1": "confluence.confluence_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 691, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_deal", + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_search_contacts", + "github.create_gist", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.create_or_update_file" + ], + "hit1": "hubspot.hubspot_create_deal", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 899, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_search", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_create_ticket", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "aws_s3.s3_list_buckets", + "linear.linear_list_teams", + "jira.jira_list_projects" + ], + "hit1": "zendesk.zendesk_list_tickets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 605, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "kind": "verb_product", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "ranking": [ + "discord.discord_list_channels", + "discord.discord_list_members", + "slack.slack_list_channels", + "discord.discord_get_channel", + "discord.discord_send_message", + "discord.discord_add_reaction", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "aws_s3.s3_list_buckets", + "linear.linear_list_teams" + ], + "hit1": "discord.discord_list_channels", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 592, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "ranking": [ + "redis.redis_set", + "redis.redis_get", + "redis.redis_delete", + "redis.redis_keys", + "slack.slack_set_channel_topic", + "playwright.browser_press_key", + "jira.jira_get_issue", + "aws_s3.s3_delete_object", + "aws_s3.s3_get_object", + "figma.figma_get_file" + ], + "hit1": "redis.redis_set", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert", + "mongodb.mongo_delete", + "mongodb.mongo_find", + "mongodb.mongo_update", + "mongodb.mongo_list_collections", + "supabase.supabase_insert" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "kind": "verb_product", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "ranking": [ + "mysql.mysql_query", + "postgres.query", + "sqlite.sqlite_query", + "postgres.explain_query", + "mysql.mysql_list_tables", + "salesforce.salesforce_query", + "docker.docker_run", + "github.get_check_run", + "github.delete_check_run", + "github.update_check_run" + ], + "hit1": "mysql.mysql_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 611, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "elasticsearch.es_search", + "playwright.browser_tab_close", + "github.get_milestone", + "github.delete_milestone", + "github.get_hook", + "github.delete_hook", + "github.get_environment", + "github.delete_environment" + ], + "hit1": "elasticsearch.es_index_doc", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 346, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 605, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_put_object", + "okta.okta_get_user", + "asana.asana_get_task", + "puppeteer.puppeteer_click", + "elasticsearch.es_search", + "okta.okta_create_user", + "puppeteer.puppeteer_fill", + "aws_s3.s3_list_objects" + ], + "hit1": "aws_s3.s3_get_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 668, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones", + "cloudflare.cf_list_dns", + "cloudflare.cf_purge_cache", + "cloudflare.cf_create_dns", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "aws_s3.s3_list_buckets", + "linear.linear_list_teams", + "jira.jira_list_projects", + "kubernetes.k8s_list_deployments" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 579, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_create_customer", + "stripe.stripe_create_product", + "stripe.stripe_create_refund", + "stripe.stripe_create_price", + "stripe.stripe_create_payment_intent", + "stripe.stripe_list_invoices", + "stripe.stripe_get_customer", + "stripe.stripe_list_customers", + "stripe.stripe_list_subscriptions" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "kind": "verb_product", + "gold": [ + "github.actions_list" + ], + "alts": [], + "ranking": [ + "github.actions_list", + "github.actions_get", + "github.actions_run_trigger", + "github.actions_get_job_logs", + "github.list_tags", + "github.list_gists", + "github.projects_list", + "github.list_branches", + "github.list_releases", + "github.list_discussions" + ], + "hit1": "github.actions_list", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 457, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 790, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "kind": "verb_product", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "ranking": [ + "github.actions_get_job_logs", + "github.actions_get", + "github.actions_list", + "gitlab.get_job", + "github.actions_run_trigger", + "kubernetes.k8s_pod_logs", + "git.git_log", + "gitlab.delete_job", + "gitlab.update_job", + "docker.docker_logs" + ], + "hit1": "github.actions_get_job_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 442, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 786, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "kind": "verb_product", + "gold": [ + "github.create_repository" + ], + "alts": [], + "ranking": [ + "github.create_repository", + "github.delete_repository", + "github.star_repository", + "github.unstar_repository", + "github.create_branch", + "github.fork_repository", + "github.create_issue", + "github.list_repository_collaborators", + "github.get_repository_tree", + "github.list_tags" + ], + "hit1": "github.create_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 406, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 874, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "kind": "verb_product", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "ranking": [ + "git.git_checkout", + "git.git_create_branch", + "github.create_branch", + "git.git_branch", + "gitlab.gitlab_create_merge_request", + "github.update_pull_request_branch", + "github.list_commits", + "github.delete_file", + "github.push_files", + "github.create_or_update_file" + ], + "hit1": "git.git_checkout", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 266, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1001, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "kind": "identifier", + "gold": [ + "git.git_log" + ], + "alts": [], + "ranking": [ + "git.git_log", + "git.git_show", + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_status", + "git.git_diff", + "git.git_branch", + "git.git_checkout", + "git.git_commit", + "git.git_add" + ], + "hit1": "git.git_log", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "kind": "verb_product", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "ranking": [ + "postgres.list_schemas", + "postgres.list_tables", + "postgres.explain_query", + "postgres.query", + "postgres.describe_table", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "aws_s3.s3_list_buckets", + "linear.linear_list_teams", + "jira.jira_list_projects" + ], + "hit1": "postgres.list_schemas", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 160, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 515, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "kind": "verb_product", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "ranking": [ + "memory.open_nodes", + "memory.search_nodes", + "memory.read_graph", + "memory.create_entities", + "memory.delete_relations", + "memory.create_relations", + "memory.add_observations", + "memory.delete_entities", + "figma.figma_get_file_nodes", + "playwright.browser_tab_list" + ], + "hit1": "memory.open_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 260, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 531, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "kind": "paraphrase", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "ranking": [ + "filesystem.read_file", + "filesystem.get_file_info", + "filesystem.read_text_file", + "filesystem.read_media_file", + "google_drive.gdrive_get_file", + "figma.figma_get_file_nodes", + "gitlab.gitlab_get_file", + "figma.figma_get_images", + "github.get_file_contents", + "filesystem.write_file" + ], + "hit1": "filesystem.read_file", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 220, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 766, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "kind": "paraphrase", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "ranking": [ + "filesystem.write_file", + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.edit_file", + "filesystem.read_media_file", + "playwright.browser_tab_new", + "brave_search.brave_local_search", + "github.issue_write", + "figma.figma_get_file", + "google_drive.gdrive_update_file" + ], + "hit1": "filesystem.write_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 791, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "kind": "verb_product", + "gold": [ + "github.search_issues" + ], + "alts": [], + "ranking": [ + "github.search_issues", + "linear.linear_search_issues", + "jira.jira_search_issues", + "github.list_issues", + "github.search_users", + "github.search_orgs", + "github.search_commits", + "github.search_pull_requests", + "gitlab.gitlab_list_issues", + "github.search_code" + ], + "hit1": "github.search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 233, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 683, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "kind": "paraphrase", + "gold": [ + "github.get_me" + ], + "alts": [], + "ranking": [ + "github.get_teams", + "github.get_me", + "github.list_notifications", + "okta.okta_get_user", + "github.get_latest_release", + "slack.slack_get_user_profile", + "linear.get_user", + "github.get_repository_tree", + "github.list_gists", + "github.list_starred_repositories" + ], + "hit1": "github.get_teams", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 304, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 622, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "cost": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.34, + "p95_calls": 1.0, + "success_within_1": 0.98, + "success_within_2": 0.9866666666666667, + "success_within_3": 0.9933333333333333, + "mean_tokens": 315.88, + "p95_tokens": 332.54999999999995, + "total_tokens": 47382, + "tokens_per_successful_query": 315.88, + "total_tokens_successful": 47382 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.2, + "p95_calls": 1.0, + "success_within_1": 0.9866666666666667, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 452.35333333333335, + "p95_tokens": 515.3999999999988, + "total_tokens": 67853, + "tokens_per_successful_query": 452.35333333333335, + "total_tokens_successful": 67853 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.0933333333333333, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 795.3866666666667, + "p95_tokens": 1001.0, + "total_tokens": 119308, + "tokens_per_successful_query": 795.3866666666667, + "total_tokens_successful": 119308 + } + } +} diff --git a/experiments/mcp-tool-search/results/bm25_name_only.csv b/experiments/mcp-tool-search/results/bm25_name_only.csv new file mode 100644 index 00000000..a92000f2 --- /dev/null +++ b/experiments/mcp-tool-search/results/bm25_name_only.csv @@ -0,0 +1,151 @@ +id,query,kind,hit1,recall@1,recall@3,recall@5,mrr@10,ndcg@10,hit_rank,k3_calls,k3_tokens,k3_success1,k5_calls,k5_tokens,k5_success1,k10_calls,k10_tokens,k10_success1,ranking +q001,Create a GitHub issue,verb_product,github.create_issue,1.0,1.0,1.0,1.0000,0.7602,1,1,369,1.0,1,575,1.0,1,911,1.0,github.create_issue git.git_create_branch linear.linear_create_issue jira.jira_create_issue gitlab.gitlab_create_issue bitbucket.bitbucket_create_issue git.git_status git.git_diff git.git_commit git.git_add +q002,Create a Jira ticket,verb_product,zendesk.zendesk_create_ticket,0.0,1.0,1.0,0.5000,0.6309,2,1,333,1.0,1,481,1.0,1,808,1.0,zendesk.zendesk_create_ticket jira.jira_create_issue jira.jira_create_sprint zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket jira.jira_update_issue jira.jira_get_issue jira.jira_search_issues jira.jira_add_comment jira.jira_transition_issue +q003,Send a Slack message,verb_product,discord.discord_send_message,0.0,1.0,1.0,0.5000,0.6309,2,1,250,1.0,1,427,1.0,1,737,1.0,discord.discord_send_message slack.slack_post_message gmail.gmail_send_email twilio.twilio_send_sms slack.slack_add_reaction slack.slack_list_channels slack.slack_get_users slack.slack_join_channel slack.slack_leave_channel slack.slack_create_channel +q004,List open pull requests,resource,github.list_pull_requests,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,434,1.0,1,1009,1.0,github.list_pull_requests github.search_pull_requests memory.open_nodes gitlab.gitlab_list_merge_requests github.create_pull_request github.pull_request_read github.update_pull_request github.merge_pull_request github.update_pull_request_branch github.pull_request_review_write +q005,Open a pull request on GitHub,verb_product,github.create_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,405,1.0,1,560,1.0,1,978,1.0,github.create_pull_request github.pull_request_read github.update_pull_request github.merge_pull_request memory.open_nodes github.update_pull_request_branch github.pull_request_review_write git.git_status git.git_diff git.git_commit +q006,Run a SQL query against postgres,verb_product,postgres.query,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,335,1.0,1,714,1.0,postgres.query docker.docker_run postgres.explain_query salesforce.salesforce_query sqlite.sqlite_query mysql.mysql_query prometheus.prom_query github.actions_run_trigger github.create_check_run github.get_check_run +q007,Read a local file from disk,paraphrase,filesystem.read_file,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,398,1.0,1,798,1.0,filesystem.read_file filesystem.read_text_file filesystem.read_media_file brave_search.brave_local_search github.issue_read memory.read_graph github.pull_request_read github.delete_file filesystem.write_file filesystem.edit_file +q008,What files are in this folder,paraphrase,github.push_files,0.0,0.0,0.0,0.0000,0.0000,85,29,7932,0.0,17,7554,0.0,9,7648,0.0,github.push_files filesystem.search_files slack.list_files google_drive.gdrive_list_folder +q009,Remember this fact in the knowledge graph,paraphrase,memory.read_graph,0.0,0.0,0.0,0.0000,0.0000,90,30,8091,0.0,18,7815,0.0,9,7608,0.0,memory.read_graph +q010,Search the web with Brave,verb_product,brave_search.brave_web_search,1.0,1.0,1.0,1.0000,1.0000,1,1,245,1.0,1,404,1.0,1,713,1.0,brave_search.brave_web_search brave_search.brave_local_search filesystem.list_directory_with_sizes github.search_repositories github.search_code github.search_commits github.search_issues github.search_users github.search_orgs filesystem.search_files +q011,Fetch the contents of a URL,verb_product,fetch.fetch,1.0,1.0,1.0,1.0000,1.0000,1,1,261,1.0,1,366,1.0,1,765,1.0,fetch.fetch github.get_file_contents +q012,Show git status,identifier,git.git_status,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,325,1.0,1,597,1.0,git.git_status git.git_show git.git_diff git.git_commit git.git_add git.git_reset git.git_log git.git_checkout git.git_branch git.git_diff_unstaged +q013,Commit staged changes,paraphrase,github.get_commit,0.0,1.0,1.0,0.5000,0.6309,2,1,218,1.0,1,310,1.0,1,746,1.0,github.get_commit git.git_commit git.git_diff_staged +q014,Create a Linear issue,verb_product,linear.linear_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,322,1.0,1,528,1.0,1,974,1.0,linear.linear_create_issue linear.linear_update_issue linear.linear_get_issue linear.linear_assign_issue github.create_issue linear.linear_create_comment linear.linear_create_project jira.jira_create_issue gitlab.gitlab_create_issue bitbucket.bitbucket_create_issue +q015,Make a task in my project tracker Linear,paraphrase,linear.linear_create_project,0.0,0.0,0.0,0.0000,0.0000,146,49,12480,0.0,30,12220,0.0,15,11860,0.0,linear.linear_create_project twilio.twilio_make_call vercel.vercel_create_project asana.asana_create_project asana.asana_create_task asana.asana_get_task asana.asana_update_task clickup.clickup_create_task clickup.clickup_get_task clickup.clickup_update_task +q016,Post a comment on a GitHub issue,verb_product,figma.figma_post_comment,0.0,1.0,1.0,0.5000,0.6309,2,1,307,1.0,1,429,1.0,1,732,1.0,figma.figma_post_comment github.add_issue_comment github.update_issue_comment slack.slack_post_message git.git_status git.git_diff git.git_commit git.git_add git.git_reset git.git_log +q017,Merge the pull request,ambiguous,github.merge_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,354,1.0,1,586,1.0,1,1108,1.0,github.merge_pull_request gitlab.gitlab_merge_merge_request github.create_pull_request github.pull_request_read github.update_pull_request gitlab.gitlab_create_merge_request gitlab.gitlab_get_merge_request github.update_pull_request_branch github.pull_request_review_write github.list_pull_requests +q018,Trigger a GitHub Actions workflow,verb_product,github.actions_run_trigger,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,416,1.0,1,719,1.0,github.actions_run_trigger github.actions_list github.actions_get circleci.circleci_trigger_pipeline git.git_status git.git_diff git.git_commit git.git_add git.git_reset git.git_log +q019,Get Dependabot alerts,resource,github.list_dependabot_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,287,1.0,1,421,1.0,1,801,1.0,github.list_dependabot_alerts github.get_dependabot_alert github.list_code_scanning_alerts github.list_secret_scanning_alerts github.get_me github.get_teams github.get_commit github.get_tag github.get_discussion github.actions_get +q020,List Sentry issues,verb_product,sentry.sentry_list_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,289,1.0,1,425,1.0,1,796,1.0,sentry.sentry_list_issues github.list_issues sentry.sentry_list_events sentry.sentry_list_projects linear.linear_list_issues gitlab.gitlab_list_issues github.search_issues sentry.sentry_get_issue sentry.sentry_update_issue linear.linear_search_issues +q021,Create a Stripe customer,verb_product,stripe.stripe_create_customer,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,334,1.0,1,669,1.0,stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_create_invoice stripe.stripe_create_product stripe.stripe_create_price stripe.stripe_create_refund stripe.stripe_create_payment_intent stripe.stripe_list_customers stripe.stripe_list_invoices stripe.stripe_list_subscriptions +q022,Refund a Stripe payment,paraphrase,stripe.stripe_create_refund,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,352,1.0,1,662,1.0,stripe.stripe_create_refund stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_list_customers stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_create_product stripe.stripe_create_price +q023,Send an email with Gmail,verb_product,gmail.gmail_send_email,1.0,1.0,1.0,1.0000,1.0000,1,1,262,1.0,1,377,1.0,1,716,1.0,gmail.gmail_send_email gmail.gmail_get_email gmail.gmail_modify_email gmail.gmail_trash_email filesystem.list_directory_with_sizes twilio.twilio_send_sms discord.discord_send_message gmail.gmail_search_emails gmail.gmail_create_draft gmail.gmail_list_labels +q024,Search my Gmail inbox,verb_product,gmail.gmail_search_emails,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,352,1.0,1,714,1.0,gmail.gmail_search_emails gmail.gmail_send_email gmail.gmail_get_email gmail.gmail_create_draft gmail.gmail_list_labels gmail.gmail_modify_email gmail.gmail_trash_email github.search_repositories github.search_code github.search_commits +q025,Upload a file to Google Drive,paraphrase,slack.slack_upload_file,0.0,0.0,0.0,0.0000,0.0000,254,85,20407,0.0,51,19659,0.0,26,19548,0.0,slack.slack_upload_file github.assign_copilot_to_issue slack.slack_reply_to_thread github.add_comment_to_pending_review github.delete_file filesystem.read_file filesystem.write_file filesystem.edit_file filesystem.move_file slack.create_file +q026,Create a calendar event,resource,google_calendar.gcal_create_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,480,1.0,1,844,1.0,google_calendar.gcal_create_event google_calendar.gcal_update_event google_calendar.gcal_delete_event github.create_repository github.create_branch github.create_issue github.create_gist filesystem.create_directory memory.create_entities memory.create_relations +q027,List Kubernetes pods,verb_product,kubernetes.k8s_list_pods,1.0,1.0,1.0,1.0000,1.0000,1,1,251,1.0,1,393,1.0,1,750,1.0,kubernetes.k8s_list_pods github.list_commits github.list_branches github.list_tags github.list_releases github.list_issues github.list_notifications github.list_discussions github.actions_list github.list_gists +q028,Get logs from a k8s pod,verb_product,kubernetes.k8s_pod_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,262,1.0,1,455,1.0,1,803,1.0,kubernetes.k8s_pod_logs kubernetes.k8s_get_pod docker.docker_logs github.actions_get_job_logs datadog.datadog_list_logs github.get_me github.get_teams github.get_commit github.get_tag github.get_discussion +q029,Run a docker container,verb_product,docker.docker_run,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,327,1.0,1,709,1.0,docker.docker_run docker.docker_ps docker.docker_logs docker.docker_stop docker.docker_rm docker.docker_images docker.docker_build docker.docker_exec github.actions_run_trigger github.create_check_run +q030,Show docker container logs,verb_product,docker.docker_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,241,1.0,1,380,1.0,1,648,1.0,docker.docker_logs git.git_show kubernetes.k8s_pod_logs docker.docker_ps docker.docker_run docker.docker_stop docker.docker_rm docker.docker_images docker.docker_build docker.docker_exec +q031,Navigate the browser to a URL,ambiguous,playwright.browser_navigate,1.0,1.0,1.0,1.0000,1.0000,1,1,169,1.0,1,309,1.0,1,756,1.0,playwright.browser_navigate puppeteer.puppeteer_navigate playwright.browser_snapshot playwright.browser_click playwright.browser_type playwright.browser_evaluate github.assign_copilot_to_issue slack.slack_reply_to_thread github.add_comment_to_pending_review playwright.browser_fill_form +q032,Take a screenshot of the page,ambiguous,playwright.browser_take_screenshot,1.0,1.0,1.0,1.0000,1.0000,1,1,226,1.0,1,382,1.0,1,713,1.0,playwright.browser_take_screenshot puppeteer.puppeteer_screenshot notion.notion_get_page notion.notion_create_page notion.notion_update_page confluence.confluence_get_page confluence.confluence_create_page confluence.confluence_update_page +q033,Create a GitLab merge request,verb_product,gitlab.gitlab_merge_merge_request,0.0,1.0,1.0,0.5000,0.6309,2,1,323,1.0,1,552,1.0,1,881,1.0,gitlab.gitlab_merge_merge_request gitlab.gitlab_create_merge_request github.merge_pull_request gitlab.gitlab_get_merge_request github.create_pull_request git.git_create_branch git.git_status git.git_diff git.git_commit git.git_add +q034,Search Notion pages,verb_product,notion.notion_search,1.0,1.0,1.0,1.0000,1.0000,1,1,229,1.0,1,351,1.0,1,686,1.0,notion.notion_search notion.notion_get_page notion.notion_create_page notion.notion_update_page notion.notion_get_block notion.notion_append_block notion.notion_query_database notion.notion_create_database notion.notion_list_users notion.notion_get_comments +q035,Create a Notion page,verb_product,notion.notion_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,230,1.0,1,407,1.0,1,748,1.0,notion.notion_create_page notion.notion_get_page notion.notion_update_page confluence.confluence_create_page notion.notion_create_database notion.notion_search confluence.confluence_get_page confluence.confluence_update_page notion.notion_get_block notion.notion_append_block +q036,Query a Notion database,verb_product,notion.notion_query_database,1.0,1.0,1.0,1.0000,1.0000,1,1,245,1.0,1,360,1.0,1,687,1.0,notion.notion_query_database notion.notion_create_database postgres.query postgres.explain_query salesforce.salesforce_query sqlite.sqlite_query mysql.mysql_query prometheus.prom_query notion.notion_search datadog.datadog_query_metrics +q037,Create a Zendesk ticket,verb_product,zendesk.zendesk_create_ticket,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,368,1.0,1,813,1.0,zendesk.zendesk_create_ticket zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_search zendesk.zendesk_list_tickets github.create_repository github.create_branch github.create_issue github.create_gist filesystem.create_directory +q038,Create a HubSpot contact,verb_product,hubspot.hubspot_create_contact,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,476,1.0,1,813,1.0,hubspot.hubspot_create_contact hubspot.hubspot_get_contact github.create_repository github.create_branch github.create_issue github.create_gist filesystem.create_directory memory.create_entities memory.create_relations github.create_milestone +q039,Create a Salesforce lead,verb_product,salesforce.salesforce_create_lead,1.0,1.0,1.0,1.0000,1.0000,1,1,252,1.0,1,402,1.0,1,784,1.0,salesforce.salesforce_create_lead salesforce.salesforce_convert_lead salesforce.salesforce_create_record salesforce.salesforce_query salesforce.salesforce_update_record salesforce.salesforce_get_record salesforce.salesforce_delete_record salesforce.salesforce_list_opportunities github.create_repository github.create_branch +q040,Run a SOQL query,identifier,postgres.query,0.0,0.0,1.0,0.2500,0.4307,4,2,408,0.0,1,334,1.0,1,713,1.0,postgres.query docker.docker_run postgres.explain_query salesforce.salesforce_query sqlite.sqlite_query mysql.mysql_query prometheus.prom_query github.actions_run_trigger github.create_check_run github.get_check_run +q041,List Shopify orders,verb_product,shopify.shopify_list_orders,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,354,1.0,1,728,1.0,shopify.shopify_list_orders shopify.shopify_list_products shopify.shopify_list_customers shopify.shopify_get_product shopify.shopify_create_product shopify.shopify_update_product shopify.shopify_get_order shopify.shopify_create_order github.list_commits github.list_branches +q042,Create a PagerDuty incident,verb_product,pagerduty.pagerduty_create_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,405,1.0,1,818,1.0,pagerduty.pagerduty_create_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident github.create_repository github.create_branch github.create_issue github.create_gist filesystem.create_directory memory.create_entities +q043,Who is on call in PagerDuty,paraphrase,twilio.twilio_make_call,0.0,0.0,0.0,0.0000,0.0000,244,82,19786,0.0,49,18956,0.0,25,18737,0.0,twilio.twilio_make_call +q044,Query Datadog metrics,verb_product,datadog.datadog_query_metrics,1.0,1.0,1.0,1.0000,1.0000,1,1,225,1.0,1,340,1.0,1,619,1.0,datadog.datadog_query_metrics postgres.query prometheus.prom_list_metrics postgres.explain_query salesforce.salesforce_query sqlite.sqlite_query mysql.mysql_query prometheus.prom_query datadog.datadog_list_monitors datadog.datadog_get_monitor +q045,Search Datadog logs,verb_product,datadog.datadog_list_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,280,1.0,1,444,1.0,1,871,1.0,datadog.datadog_list_logs docker.docker_logs kubernetes.k8s_pod_logs datadog.datadog_query_metrics datadog.datadog_list_monitors datadog.datadog_get_monitor datadog.datadog_create_monitor github.actions_get_job_logs github.search_repositories github.search_code +q046,List S3 buckets,verb_product,aws_s3.s3_list_buckets,1.0,1.0,1.0,1.0000,1.0000,1,1,236,1.0,1,378,1.0,1,735,1.0,aws_s3.s3_list_buckets github.list_commits github.list_branches github.list_tags github.list_releases github.list_issues github.list_notifications github.list_discussions github.actions_list github.list_gists +q047,Upload an object to S3,paraphrase,aws_s3.s3_get_object,0.0,1.0,1.0,0.5000,0.6309,2,1,250,1.0,1,437,1.0,1,839,1.0,aws_s3.s3_get_object aws_s3.s3_put_object aws_s3.s3_delete_object slack.slack_upload_file github.assign_copilot_to_issue slack.slack_reply_to_thread github.add_comment_to_pending_review +q048,Purge Cloudflare cache,verb_product,cloudflare.cf_purge_cache,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,341,1.0,1,756,1.0,cloudflare.cf_purge_cache +q049,Create a Cloudflare DNS record,verb_product,cloudflare.cf_create_dns,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,415,1.0,1,879,1.0,cloudflare.cf_create_dns salesforce.salesforce_create_record cloudflare.cf_list_dns salesforce.salesforce_update_record salesforce.salesforce_get_record salesforce.salesforce_delete_record github.create_repository github.create_branch github.create_issue github.create_gist +q050,List Vercel deployments,verb_product,vercel.vercel_list_deployments,1.0,1.0,1.0,1.0000,1.0000,1,1,173,1.0,1,291,1.0,1,710,1.0,vercel.vercel_list_deployments kubernetes.k8s_list_deployments vercel.vercel_list_projects vercel.vercel_get_deployment vercel.vercel_create_project github.list_commits github.list_branches github.list_tags github.list_releases github.list_issues +q051,Insert a row into Supabase,verb_product,supabase.supabase_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,230,1.0,1,352,1.0,1,679,1.0,supabase.supabase_insert mongodb.mongo_insert supabase.supabase_select supabase.supabase_update supabase.supabase_list_tables supabase.supabase_invoke_fn +q052,Export images from Figma,paraphrase,figma.figma_get_images,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,325,1.0,1,651,1.0,figma.figma_get_images docker.docker_images figma.figma_get_file figma.figma_get_comments figma.figma_post_comment figma.figma_get_file_nodes +q053,Send an SMS with Twilio,verb_product,twilio.twilio_send_sms,1.0,1.0,1.0,1.0000,1.0000,1,1,261,1.0,1,401,1.0,1,718,1.0,twilio.twilio_send_sms filesystem.list_directory_with_sizes gmail.gmail_send_email twilio.twilio_list_messages twilio.twilio_make_call discord.discord_send_message +q054,Create an Okta user,verb_product,okta.okta_create_user,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,329,1.0,1,631,1.0,okta.okta_create_user okta.okta_get_user okta.okta_deactivate_user linear.create_user linear.get_user linear.update_user linear.delete_user okta.okta_list_users slack.slack_get_user_profile github.create_repository +q055,Instant query Prometheus,verb_product,postgres.query,0.0,0.0,0.0,0.1667,0.3562,6,2,365,0.0,2,704,0.0,1,684,1.0,postgres.query postgres.explain_query salesforce.salesforce_query sqlite.sqlite_query mysql.mysql_query prometheus.prom_query notion.notion_query_database datadog.datadog_query_metrics prometheus.prom_query_range +q056,Search Grafana dashboards,verb_product,grafana.grafana_search_dashboards,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,376,1.0,1,685,1.0,grafana.grafana_search_dashboards grafana.grafana_get_dashboard grafana.grafana_create_annotation github.search_repositories github.search_code github.search_commits github.search_issues github.search_users github.search_orgs filesystem.search_files +q057,Trigger a CircleCI pipeline,verb_product,circleci.circleci_trigger_pipeline,1.0,1.0,1.0,1.0000,1.0000,1,1,308,1.0,1,430,1.0,1,701,1.0,circleci.circleci_trigger_pipeline github.actions_run_trigger gitlab.create_pipeline gitlab.get_pipeline gitlab.update_pipeline gitlab.delete_pipeline gitlab.gitlab_create_pipeline circleci.circleci_get_pipeline +q058,Run terraform plan,identifier,terraform.tf_plan,1.0,1.0,1.0,1.0000,1.0000,1,1,275,1.0,1,430,1.0,1,712,1.0,terraform.tf_plan docker.docker_run github.actions_run_trigger github.create_check_run github.get_check_run github.update_check_run github.delete_check_run +q059,Create an Asana task,verb_product,asana.asana_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,429,1.0,1,821,1.0,asana.asana_create_task asana.asana_get_task asana.asana_update_task asana.asana_create_project clickup.clickup_create_task asana.asana_list_tasks clickup.clickup_get_task clickup.clickup_update_task github.create_repository github.create_branch +q060,Create a Trello card,verb_product,trello.trello_create_card,1.0,1.0,1.0,1.0000,1.0000,1,1,245,1.0,1,396,1.0,1,809,1.0,trello.trello_create_card trello.trello_create_board trello.trello_list_cards trello.trello_add_comment github.create_repository github.create_branch github.create_issue github.create_gist filesystem.create_directory memory.create_entities +q061,Create a ClickUp task,verb_product,playwright.browser_click,0.0,0.0,1.0,0.2500,0.4307,4,2,496,0.0,1,386,1.0,1,800,1.0,playwright.browser_click puppeteer.puppeteer_click asana.asana_create_task clickup.clickup_create_task asana.asana_get_task asana.asana_update_task clickup.clickup_get_task clickup.clickup_update_task github.create_repository github.create_branch +q062,Create a Confluence page,verb_product,confluence.confluence_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,401,1.0,1,835,1.0,confluence.confluence_create_page confluence.confluence_get_page confluence.confluence_update_page notion.notion_create_page confluence.confluence_search notion.notion_get_page notion.notion_update_page github.create_repository github.create_branch github.create_issue +q063,Send a Discord message,verb_product,discord.discord_send_message,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,386,1.0,1,664,1.0,discord.discord_send_message slack.slack_post_message gmail.gmail_send_email twilio.twilio_send_sms discord.discord_list_channels discord.discord_get_channel discord.discord_add_reaction discord.discord_list_members +q064,Search Elasticsearch,verb_product,github.search_repositories,0.0,0.0,0.0,0.0000,0.0000,290,97,22834,0.0,58,22047,0.0,29,21525,0.0,github.search_repositories github.search_code github.search_commits github.search_issues github.search_users github.search_orgs filesystem.search_files memory.search_nodes notion.notion_search google_drive.gdrive_search +q065,Get a Redis key,verb_product,redis.redis_get,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,280,1.0,1,628,1.0,redis.redis_get playwright.browser_press_key redis.redis_set redis.redis_delete redis.redis_keys github.get_me github.get_teams github.get_commit github.get_tag github.get_discussion +q066,Find MongoDB documents,verb_product,mongodb.mongo_find,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,381,1.0,1,738,1.0,mongodb.mongo_find mongodb.mongo_insert mongodb.mongo_update mongodb.mongo_delete mongodb.mongo_list_collections +q067,List SQLite tables,verb_product,postgres.list_tables,0.0,1.0,1.0,0.5000,0.6309,2,1,151,1.0,1,294,1.0,1,675,1.0,postgres.list_tables sqlite.sqlite_list_tables mysql.mysql_list_tables supabase.supabase_list_tables github.list_commits github.list_branches github.list_tags github.list_releases github.list_issues github.list_notifications +q068,Describe a postgres table,verb_product,postgres.describe_table,1.0,1.0,1.0,1.0000,1.0000,1,1,187,1.0,1,292,1.0,1,714,1.0,postgres.describe_table sqlite.sqlite_describe_table +q069,github_create_issue,identifier,github.create_issue,1.0,1.0,1.0,1.0000,0.8956,1,1,398,1.0,1,580,1.0,1,1139,1.0,github.create_issue linear.linear_create_issue jira.jira_create_issue gitlab.gitlab_create_issue bitbucket.bitbucket_create_issue github.issue_write github.issue_read github.add_issue_comment github.update_issue_comment linear.linear_update_issue +q070,createIssue,identifier,github.create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,397,1.0,1,579,1.0,1,1138,1.0,github.create_issue linear.linear_create_issue jira.jira_create_issue gitlab.gitlab_create_issue bitbucket.bitbucket_create_issue github.issue_write github.issue_read github.add_issue_comment github.update_issue_comment linear.linear_update_issue +q071,list-directory,identifier,filesystem.list_directory,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,344,1.0,1,725,1.0,filesystem.list_directory filesystem.create_directory filesystem.directory_tree filesystem.list_directory_with_sizes github.list_commits github.list_branches github.list_tags github.list_releases github.list_issues github.list_notifications +q072,sequential thinking next thought,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,116,39,9799,0.0,24,9727,0.0,12,9475,0.0, +q073,What time is it in Tokyo,paraphrase,time.convert_time,1.0,1.0,1.0,1.0000,0.8597,1,1,204,1.0,1,309,1.0,1,731,1.0,time.convert_time time.get_current_time +q074,Convert time between timezones,verb_product,time.convert_time,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,296,1.0,1,732,1.0,time.convert_time time.get_current_time salesforce.salesforce_convert_lead +q075,Star a GitHub repo,verb_product,github.star_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,333,1.0,1,612,1.0,github.star_repository git.git_status git.git_diff git.git_commit git.git_add git.git_reset git.git_log git.git_checkout git.git_show git.git_branch +q076,Fork a repository,resource,github.fork_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,248,1.0,1,386,1.0,1,729,1.0,github.fork_repository github.create_repository github.delete_repository github.star_repository github.unstar_repository github.list_repository_collaborators github.get_repository_tree +q077,Get the latest GitHub release,verb_product,github.get_latest_release,1.0,1.0,1.0,1.0000,1.0000,1,1,245,1.0,1,367,1.0,1,704,1.0,github.get_latest_release gitlab.get_release gitlab.create_release gitlab.update_release gitlab.delete_release github.get_release_by_tag git.git_status git.git_diff git.git_commit git.git_add +q078,Create a gist,resource,github.create_gist,1.0,1.0,1.0,1.0000,1.0000,1,1,229,1.0,1,409,1.0,1,800,1.0,github.create_gist github.get_gist github.update_gist github.create_repository github.create_branch github.create_issue filesystem.create_directory memory.create_entities memory.create_relations github.create_milestone +q079,Search GitHub code,verb_product,github.search_code,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,333,1.0,1,612,1.0,github.search_code git.git_status git.git_diff git.git_commit git.git_add git.git_reset git.git_log git.git_checkout git.git_show git.git_branch +q080,List GitHub notifications,verb_product,github.list_notifications,1.0,1.0,1.0,1.0000,1.0000,1,1,175,1.0,1,303,1.0,1,601,1.0,github.list_notifications github.mark_all_notifications_read git.git_status git.git_diff git.git_commit git.git_add git.git_reset git.git_log git.git_checkout git.git_show +q081,Assign Copilot to an issue,verb_product,github.assign_copilot_to_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,457,1.0,1,1083,1.0,github.assign_copilot_to_issue linear.linear_assign_issue jira.jira_assign_issue github.request_copilot_review slack.slack_reply_to_thread github.create_issue github.issue_write github.issue_read github.add_comment_to_pending_review github.add_issue_comment +q082,Get secret scanning alerts,resource,github.list_secret_scanning_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,295,1.0,1,470,1.0,1,818,1.0,github.list_secret_scanning_alerts github.get_secret_scanning_alert github.list_code_scanning_alerts github.get_code_scanning_alert github.list_dependabot_alerts github.get_me github.get_teams github.get_commit github.get_tag github.get_discussion +q083,Create or update a file on GitHub,verb_product,github.create_or_update_file,1.0,1.0,1.0,1.0000,0.7602,1,1,333,1.0,1,508,1.0,1,861,1.0,github.create_or_update_file slack.update_file slack.create_file git.git_create_branch google_drive.gdrive_update_file google_drive.gdrive_create_file git.git_status git.git_diff git.git_commit git.git_add +q084,List collaborators on a repo,resource,github.list_repository_collaborators,1.0,1.0,1.0,1.0000,1.0000,1,1,274,1.0,1,416,1.0,1,773,1.0,github.list_repository_collaborators github.list_commits github.list_branches github.list_tags github.list_releases github.list_issues github.list_notifications github.list_discussions github.actions_list github.list_gists +q085,Reply in a Slack thread,verb_product,slack.slack_reply_to_thread,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,414,1.0,1,740,1.0,slack.slack_reply_to_thread slack.slack_get_thread_replies slack.slack_post_message slack.slack_add_reaction slack.slack_list_channels slack.slack_get_users slack.slack_join_channel slack.slack_leave_channel slack.slack_create_channel slack.slack_upload_file +q086,Add an emoji reaction on Slack,verb_product,slack.slack_add_reaction,1.0,1.0,1.0,1.0000,1.0000,1,1,256,1.0,1,426,1.0,1,792,1.0,slack.slack_add_reaction discord.discord_add_reaction memory.add_observations git.git_add github.add_issue_comment jira.jira_add_comment jira.jira_add_worklog trello.trello_add_comment slack.slack_post_message slack.slack_list_channels +q087,Search Slack messages,verb_product,slack.slack_search_messages,1.0,1.0,1.0,1.0000,1.0000,1,1,231,1.0,1,352,1.0,1,665,1.0,slack.slack_search_messages twilio.twilio_list_messages github.search_repositories github.search_code github.search_commits github.search_issues github.search_users github.search_orgs filesystem.search_files memory.search_nodes +q088,Create a Slack channel,verb_product,slack.slack_create_channel,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,342,1.0,1,685,1.0,slack.slack_create_channel slack.slack_join_channel slack.slack_leave_channel slack.slack_get_channel_history slack.slack_set_channel_topic discord.discord_get_channel slack.slack_post_message slack.slack_add_reaction slack.slack_list_channels slack.slack_get_users +q089,Transition a Jira issue,verb_product,jira.jira_transition_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,270,1.0,1,389,1.0,1,763,1.0,jira.jira_transition_issue jira.jira_create_issue jira.jira_update_issue jira.jira_get_issue jira.jira_assign_issue jira.jira_search_issues jira.jira_add_comment jira.jira_list_projects jira.jira_create_sprint jira.jira_add_worklog +q090,Search Jira with JQL,identifier,jira.jira_search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,385,1.0,1,733,1.0,jira.jira_search_issues filesystem.list_directory_with_sizes jira.jira_create_issue jira.jira_update_issue jira.jira_get_issue jira.jira_add_comment jira.jira_transition_issue jira.jira_assign_issue jira.jira_list_projects jira.jira_create_sprint +q091,Add a worklog to Jira,verb_product,jira.jira_add_worklog,1.0,1.0,1.0,1.0000,1.0000,1,1,334,1.0,1,455,1.0,1,914,1.0,jira.jira_add_worklog jira.jira_add_comment github.add_comment_to_pending_review memory.add_observations git.git_add github.assign_copilot_to_issue slack.slack_reply_to_thread github.add_issue_comment slack.slack_add_reaction trello.trello_add_comment +q092,List Linear teams,verb_product,linear.linear_list_teams,1.0,1.0,1.0,1.0000,1.0000,1,1,176,1.0,1,354,1.0,1,728,1.0,linear.linear_list_teams github.get_teams linear.linear_list_issues linear.linear_list_projects linear.linear_create_issue linear.linear_update_issue linear.linear_get_issue linear.linear_search_issues linear.linear_create_comment linear.linear_create_project +q093,Comment on a Linear issue,verb_product,linear.linear_create_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,334,1.0,1,464,1.0,1,885,1.0,linear.linear_create_comment linear.linear_create_issue linear.linear_update_issue linear.linear_get_issue linear.linear_assign_issue github.add_issue_comment github.update_issue_comment linear.create_comment linear.get_comment linear.update_comment +q094,Move a local file,paraphrase,filesystem.move_file,1.0,1.0,1.0,1.0000,1.0000,1,1,271,1.0,1,398,1.0,1,760,1.0,filesystem.move_file brave_search.brave_local_search github.delete_file filesystem.read_file filesystem.write_file filesystem.edit_file slack.create_file slack.get_file slack.update_file slack.delete_file +q095,Recursively search files by pattern,paraphrase,filesystem.search_files,1.0,1.0,1.0,1.0000,1.0000,1,1,280,1.0,1,463,1.0,1,772,1.0,filesystem.search_files github.push_files slack.list_files github.get_release_by_tag github.search_repositories github.search_code github.search_commits github.search_issues github.search_users github.search_orgs +q096,Edit a file in place with old and new string,paraphrase,filesystem.edit_file,1.0,1.0,1.0,1.0000,1.0000,1,1,227,1.0,1,398,1.0,1,746,1.0,filesystem.edit_file playwright.browser_tab_new filesystem.list_directory_with_sizes github.delete_file filesystem.read_file filesystem.write_file filesystem.move_file slack.create_file slack.get_file slack.update_file +q097,Read the whole knowledge graph,paraphrase,memory.read_graph,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,387,1.0,1,642,1.0,memory.read_graph github.issue_read filesystem.read_file github.pull_request_read filesystem.read_text_file filesystem.read_media_file github.mark_all_notifications_read +q098,Search memory nodes,verb_product,memory.search_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,202,1.0,1,361,1.0,1,670,1.0,memory.search_nodes memory.open_nodes figma.figma_get_file_nodes github.search_repositories github.search_code github.search_commits github.search_issues github.search_users github.search_orgs filesystem.search_files +q099,Show unstaged git diff,verb_product,git.git_diff_unstaged,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,303,1.0,1,604,1.0,git.git_diff_unstaged git.git_show git.git_diff git.git_diff_staged git.git_status git.git_commit git.git_add git.git_reset git.git_log git.git_checkout +q100,Stage files with git add,identifier,git.git_add,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,403,1.0,1,687,1.0,git.git_add github.push_files filesystem.search_files slack.list_files filesystem.list_directory_with_sizes memory.add_observations git.git_status git.git_diff git.git_commit git.git_reset +q101,Create a Shopify product,verb_product,shopify.shopify_create_product,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,377,1.0,1,702,1.0,shopify.shopify_create_product shopify.shopify_get_product shopify.shopify_update_product stripe.stripe_create_product shopify.shopify_create_order shopify.shopify_list_products shopify.shopify_list_orders shopify.shopify_get_order shopify.shopify_list_customers github.create_repository +q102,Cancel a Stripe subscription,verb_product,stripe.stripe_cancel_subscription,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,301,1.0,1,632,1.0,stripe.stripe_cancel_subscription stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_list_customers stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_create_product stripe.stripe_create_price stripe.stripe_list_subscriptions stripe.stripe_create_refund +q103,Create a payment intent,resource,stripe.stripe_create_payment_intent,1.0,1.0,1.0,1.0000,1.0000,1,1,245,1.0,1,477,1.0,1,814,1.0,stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent github.create_repository github.create_branch github.create_issue github.create_gist filesystem.create_directory memory.create_entities memory.create_relations github.create_milestone +q104,Acknowledge a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,261,1.0,1,381,1.0,1,787,1.0,pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_resolve_incident +q105,Create a Datadog monitor,verb_product,datadog.datadog_create_monitor,1.0,1.0,1.0,1.0000,1.0000,1,1,297,1.0,1,459,1.0,1,904,1.0,datadog.datadog_create_monitor datadog.datadog_get_monitor datadog.datadog_query_metrics datadog.datadog_list_monitors datadog.datadog_list_logs github.create_repository github.create_branch github.create_issue github.create_gist filesystem.create_directory +q106,Resolve a Sentry issue,paraphrase,sentry.sentry_get_issue,0.0,1.0,1.0,0.5000,0.6309,2,1,245,1.0,1,412,1.0,1,935,1.0,sentry.sentry_get_issue sentry.sentry_update_issue pagerduty.pagerduty_resolve_incident sentry.sentry_list_issues sentry.sentry_list_events sentry.sentry_list_projects github.create_issue github.issue_write github.issue_read github.add_issue_comment +q107,Scale a Kubernetes deployment,verb_product,kubernetes.k8s_scale_deployment,1.0,1.0,1.0,1.0000,1.0000,1,1,232,1.0,1,337,1.0,1,759,1.0,kubernetes.k8s_scale_deployment vercel.vercel_get_deployment +q108,Apply a Kubernetes manifest,verb_product,kubernetes.k8s_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,202,1.0,1,307,1.0,1,729,1.0,kubernetes.k8s_apply terraform.tf_apply +q109,Stop a docker container,verb_product,docker.docker_stop,1.0,1.0,1.0,1.0000,1.0000,1,1,174,1.0,1,327,1.0,1,581,1.0,docker.docker_stop docker.docker_ps docker.docker_logs docker.docker_run docker.docker_rm docker.docker_images docker.docker_build docker.docker_exec +q110,Build a docker image,verb_product,docker.docker_build,1.0,1.0,1.0,1.0000,1.0000,1,1,193,1.0,1,332,1.0,1,581,1.0,docker.docker_build docker.docker_ps docker.docker_logs docker.docker_run docker.docker_stop docker.docker_rm docker.docker_images docker.docker_exec +q111,Click an element in Playwright,verb_product,playwright.browser_click,1.0,1.0,1.0,1.0000,1.0000,1,1,193,1.0,1,298,1.0,1,720,1.0,playwright.browser_click puppeteer.puppeteer_click +q112,Type text into a browser input,paraphrase,playwright.browser_type,1.0,1.0,1.0,1.0000,0.7602,1,1,236,1.0,1,338,1.0,1,598,1.0,playwright.browser_type filesystem.read_text_file playwright.browser_navigate playwright.browser_snapshot playwright.browser_click playwright.browser_evaluate playwright.browser_fill_form playwright.browser_take_screenshot playwright.browser_tab_list playwright.browser_tab_new +q113,List GitLab projects,verb_product,github.projects_list,0.0,0.0,1.0,0.2000,0.3869,5,2,326,0.0,1,268,1.0,1,573,1.0,github.projects_list linear.linear_list_projects jira.jira_list_projects sentry.sentry_list_projects gitlab.gitlab_list_projects vercel.vercel_list_projects github.projects_get github.projects_write git.git_status git.git_diff +q114,Create a Bitbucket pull request,verb_product,github.create_pull_request,0.0,0.0,0.0,0.1429,0.3333,7,3,1123,0.0,2,1177,0.0,1,1155,1.0,github.create_pull_request github.pull_request_read github.update_pull_request github.merge_pull_request github.update_pull_request_branch github.pull_request_review_write bitbucket.bitbucket_create_pr bitbucket.bitbucket_create_issue gitlab.gitlab_create_merge_request bitbucket.bitbucket_list_prs +q115,Create a Gmail draft,verb_product,gmail.gmail_create_draft,1.0,1.0,1.0,1.0000,1.0000,1,1,265,1.0,1,352,1.0,1,814,1.0,gmail.gmail_create_draft gmail.gmail_send_email gmail.gmail_search_emails gmail.gmail_get_email gmail.gmail_list_labels gmail.gmail_modify_email gmail.gmail_trash_email github.create_repository github.create_branch github.create_issue +q116,Share a Google Drive file,verb_product,google_drive.gdrive_share_file,1.0,1.0,1.0,1.0000,1.0000,1,1,278,1.0,1,440,1.0,1,771,1.0,google_drive.gdrive_share_file github.delete_file filesystem.read_file filesystem.write_file filesystem.edit_file filesystem.move_file slack.create_file slack.get_file slack.update_file slack.delete_file +q117,List Google Calendar events,verb_product,sentry.sentry_list_events,0.0,1.0,1.0,0.5000,0.6309,2,1,233,1.0,1,408,1.0,1,789,1.0,sentry.sentry_list_events google_calendar.gcal_list_events kubernetes.k8s_get_events github.list_commits github.list_branches github.list_tags github.list_releases github.list_issues github.list_notifications github.list_discussions +q118,Delete a calendar event,resource,google_calendar.gcal_delete_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,455,1.0,1,798,1.0,google_calendar.gcal_delete_event google_calendar.gcal_create_event google_calendar.gcal_update_event kubernetes.k8s_delete github.delete_repository github.delete_file memory.delete_entities memory.delete_observations memory.delete_relations mongodb.mongo_delete +q119,List Vercel projects,verb_product,vercel.vercel_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,263,1.0,1,576,1.0,vercel.vercel_list_projects github.projects_list vercel.vercel_list_deployments linear.linear_list_projects jira.jira_list_projects sentry.sentry_list_projects gitlab.gitlab_list_projects github.projects_get github.projects_write vercel.vercel_get_deployment +q120,Select rows from Supabase,verb_product,supabase.supabase_select,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,354,1.0,1,711,1.0,supabase.supabase_select supabase.supabase_insert supabase.supabase_update supabase.supabase_list_tables supabase.supabase_invoke_fn +q121,Post a Figma comment,verb_product,figma.figma_post_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,362,1.0,1,721,1.0,figma.figma_post_comment slack.slack_post_message figma.figma_get_file figma.figma_get_comments figma.figma_get_images linear.create_comment linear.get_comment linear.update_comment linear.delete_comment notion.create_comment +q122,List Okta users,verb_product,okta.okta_list_users,1.0,1.0,1.0,1.0000,1.0000,1,1,209,1.0,1,327,1.0,1,606,1.0,okta.okta_list_users linear.list_users slack.slack_list_users notion.notion_list_users notion.list_users_tokens okta.okta_get_user okta.okta_create_user okta.okta_deactivate_user github.search_users slack.slack_get_users +q123,Range query Prometheus,verb_product,prometheus.prom_query_range,1.0,1.0,1.0,1.0000,1.0000,1,1,227,1.0,1,339,1.0,1,684,1.0,prometheus.prom_query_range postgres.query postgres.explain_query salesforce.salesforce_query sqlite.sqlite_query mysql.mysql_query prometheus.prom_query notion.notion_query_database datadog.datadog_query_metrics +q124,Create a Grafana annotation,verb_product,grafana.grafana_create_annotation,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,398,1.0,1,762,1.0,grafana.grafana_create_annotation grafana.grafana_search_dashboards grafana.grafana_get_dashboard github.create_repository github.create_branch github.create_issue github.create_gist filesystem.create_directory memory.create_entities memory.create_relations +q125,Apply terraform,identifier,kubernetes.k8s_apply,0.0,1.0,1.0,0.5000,0.6309,2,1,200,1.0,1,305,1.0,1,727,1.0,kubernetes.k8s_apply terraform.tf_apply +q126,List Asana tasks,verb_product,asana.asana_list_tasks,1.0,1.0,1.0,1.0000,1.0000,1,1,245,1.0,1,386,1.0,1,774,1.0,asana.asana_list_tasks clickup.clickup_list_tasks asana.asana_create_task asana.asana_get_task asana.asana_update_task asana.asana_create_project github.list_commits github.list_branches github.list_tags github.list_releases +q127,Comment on a Trello card,verb_product,trello.trello_create_card,0.0,1.0,1.0,0.5000,0.6309,2,1,260,1.0,1,412,1.0,1,729,1.0,trello.trello_create_card trello.trello_add_comment trello.trello_list_cards trello.trello_create_board linear.create_comment linear.get_comment linear.update_comment linear.delete_comment notion.create_comment notion.get_comment +q128,Update a ClickUp task,verb_product,asana.asana_update_task,0.0,1.0,1.0,0.5000,0.6309,2,1,264,1.0,1,423,1.0,1,788,1.0,asana.asana_update_task clickup.clickup_update_task playwright.browser_click puppeteer.puppeteer_click asana.asana_create_task asana.asana_get_task clickup.clickup_create_task clickup.clickup_get_task github.update_gist mongodb.mongo_update +q129,Search Confluence,verb_product,confluence.confluence_search,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,403,1.0,1,712,1.0,confluence.confluence_search confluence.confluence_get_page confluence.confluence_create_page confluence.confluence_update_page github.search_repositories github.search_code github.search_commits github.search_issues github.search_users github.search_orgs +q130,Create a HubSpot deal,verb_product,hubspot.hubspot_create_deal,1.0,1.0,1.0,1.0000,1.0000,1,1,292,1.0,1,502,1.0,1,856,1.0,hubspot.hubspot_create_deal github.create_repository github.create_branch github.create_issue github.create_gist filesystem.create_directory memory.create_entities memory.create_relations github.create_milestone github.create_hook +q131,List Zendesk tickets,verb_product,zendesk.zendesk_list_tickets,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,367,1.0,1,786,1.0,zendesk.zendesk_list_tickets zendesk.zendesk_search zendesk.zendesk_create_ticket zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket github.list_commits github.list_branches github.list_tags github.list_releases github.list_issues +q132,List Discord channels,verb_product,discord.discord_list_channels,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,300,1.0,1,705,1.0,discord.discord_list_channels slack.slack_list_channels discord.discord_list_members discord.discord_send_message discord.discord_get_channel discord.discord_add_reaction github.list_commits github.list_branches github.list_tags github.list_releases +q133,Set a Redis key,verb_product,redis.redis_set,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,280,1.0,1,603,1.0,redis.redis_set playwright.browser_press_key redis.redis_get redis.redis_delete redis.redis_keys slack.slack_set_channel_topic +q134,Insert MongoDB documents,verb_product,mongodb.mongo_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,399,1.0,1,698,1.0,mongodb.mongo_insert supabase.supabase_insert mongodb.mongo_find mongodb.mongo_update mongodb.mongo_delete mongodb.mongo_list_collections +q135,Run a MySQL query,verb_product,postgres.query,0.0,0.0,0.0,0.1667,0.3562,6,2,408,0.0,2,734,0.0,1,713,1.0,postgres.query docker.docker_run postgres.explain_query salesforce.salesforce_query sqlite.sqlite_query mysql.mysql_query prometheus.prom_query github.actions_run_trigger github.create_check_run github.get_check_run +q136,Index a document in Elasticsearch,verb_product,elasticsearch.es_index_doc,1.0,1.0,1.0,1.0000,1.0000,1,1,198,1.0,1,356,1.0,1,771,1.0,elasticsearch.es_index_doc +q137,Get an S3 object,verb_product,aws_s3.s3_get_object,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,341,1.0,1,735,1.0,aws_s3.s3_get_object aws_s3.s3_put_object aws_s3.s3_delete_object github.get_me github.get_teams github.get_commit github.get_tag github.get_discussion github.actions_get github.get_gist +q138,List Cloudflare zones,verb_product,cloudflare.cf_list_zones,1.0,1.0,1.0,1.0000,1.0000,1,1,232,1.0,1,374,1.0,1,731,1.0,cloudflare.cf_list_zones github.list_commits github.list_branches github.list_tags github.list_releases github.list_issues github.list_notifications github.list_discussions github.actions_list github.list_gists +q139,Create a Stripe invoice,verb_product,stripe.stripe_create_invoice,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,353,1.0,1,669,1.0,stripe.stripe_create_invoice stripe.stripe_create_customer stripe.stripe_create_product stripe.stripe_create_price stripe.stripe_create_refund stripe.stripe_create_payment_intent stripe.stripe_get_customer stripe.stripe_list_customers stripe.stripe_list_invoices stripe.stripe_list_subscriptions +q140,List GitHub Actions workflows,verb_product,github.actions_list,1.0,1.0,1.0,1.0000,1.0000,1,1,293,1.0,1,409,1.0,1,710,1.0,github.actions_list github.actions_get github.actions_run_trigger git.git_status git.git_diff git.git_commit git.git_add git.git_reset git.git_log git.git_checkout +q141,Get GitHub Actions job logs,verb_product,github.actions_get_job_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,256,1.0,1,385,1.0,1,822,1.0,github.actions_get_job_logs github.actions_get gitlab.get_job github.actions_list docker.docker_logs kubernetes.k8s_pod_logs gitlab.create_job gitlab.update_job gitlab.delete_job github.actions_run_trigger +q142,Create a GitHub repository,verb_product,github.create_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,252,1.0,1,401,1.0,1,717,1.0,github.create_repository git.git_create_branch github.delete_repository github.fork_repository github.star_repository github.unstar_repository git.git_status git.git_diff git.git_commit git.git_add +q143,Checkout a git branch,verb_product,git.git_checkout,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,364,1.0,1,667,1.0,git.git_checkout git.git_branch git.git_create_branch github.create_branch git.git_status git.git_diff git.git_commit git.git_add git.git_reset git.git_log +q144,Show git log,identifier,git.git_log,1.0,1.0,1.0,1.0000,1.0000,1,1,194,1.0,1,322,1.0,1,597,1.0,git.git_log git.git_show git.git_status git.git_diff git.git_commit git.git_add git.git_reset git.git_checkout git.git_branch git.git_diff_unstaged +q145,List postgres schemas,verb_product,postgres.list_schemas,1.0,1.0,1.0,1.0000,1.0000,1,1,231,1.0,1,373,1.0,1,730,1.0,postgres.list_schemas github.list_commits github.list_branches github.list_tags github.list_releases github.list_issues github.list_notifications github.list_discussions github.actions_list github.list_gists +q146,Open nodes in the knowledge graph,verb_product,memory.open_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,163,1.0,1,293,1.0,1,699,1.0,memory.open_nodes memory.read_graph memory.search_nodes figma.figma_get_file_nodes +q147,Get file metadata from the filesystem,paraphrase,slack.get_file,0.0,0.0,1.0,0.2500,0.4307,4,2,503,0.0,1,390,1.0,1,782,1.0,slack.get_file github.get_file_contents github.get_file_blame filesystem.get_file_info google_drive.gdrive_get_file gitlab.gitlab_get_file figma.figma_get_file figma.figma_get_file_nodes github.delete_file filesystem.read_file +q148,Write a new local file,paraphrase,filesystem.write_file,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,437,1.0,1,959,1.0,filesystem.write_file brave_search.brave_local_search playwright.browser_tab_new github.issue_write github.projects_write github.label_write github.pull_request_review_write github.delete_file filesystem.read_file filesystem.edit_file +q149,Search GitHub issues,verb_product,github.search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,233,1.0,1,387,1.0,1,690,1.0,github.search_issues linear.linear_search_issues jira.jira_search_issues github.list_issues git.git_status git.git_diff git.git_commit git.git_add git.git_reset git.git_log +q150,Get the authenticated GitHub user,paraphrase,linear.get_user,0.0,0.0,0.0,0.0000,0.0000,11,4,796,0.0,3,1039,0.0,2,1422,0.0,linear.get_user okta.okta_get_user slack.slack_get_user_profile linear.create_user linear.update_user linear.delete_user git.git_status git.git_diff git.git_commit git.git_add diff --git a/experiments/mcp-tool-search/results/bm25_name_only.json b/experiments/mcp-tool-search/results/bm25_name_only.json new file mode 100644 index 00000000..eb77dd94 --- /dev/null +++ b/experiments/mcp-tool-search/results/bm25_name_only.json @@ -0,0 +1,6824 @@ +{ + "summary": { + "n": 150, + "index_s": 0.0006164310000258411, + "rss_mb": 0.03125, + "p50_ms": 0.10278799999241528, + "p95_ms": 0.14317309993998595, + "recall@1": 0.82, + "recall@1_lo": 0.7533333333333333, + "recall@1_hi": 0.88, + "recall@3": 0.9, + "recall@3_lo": 0.8533333333333334, + "recall@3_hi": 0.9466666666666667, + "recall@5": 0.9266666666666666, + "recall@5_lo": 0.8866666666666667, + "recall@5_hi": 0.9666666666666667, + "mrr@10": 0.8695079365079365, + "mrr@10_lo": 0.8217301587301588, + "mrr@10_hi": 0.9156666666666666, + "ndcg@10": 0.8822109879794696, + "ndcg@10_lo": 0.8381211455296853, + "ndcg@10_hi": 0.9216378856737126 + }, + "meta": { + "name": "bm25_name_only", + "needs_model": false, + "needs_api": false, + "notes": "BM25 over tool name field" + }, + "rows": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "git.git_create_branch", + "linear.linear_create_issue", + "jira.jira_create_issue", + "gitlab.gitlab_create_issue", + "bitbucket.bitbucket_create_issue", + "git.git_status", + "git.git_diff", + "git.git_commit", + "git.git_add" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 369, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 575, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 911, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "kind": "verb_product", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_create_ticket", + "jira.jira_create_issue", + "jira.jira_create_sprint", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "jira.jira_update_issue", + "jira.jira_get_issue", + "jira.jira_search_issues", + "jira.jira_add_comment", + "jira.jira_transition_issue" + ], + "hit1": "zendesk.zendesk_create_ticket", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 481, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 808, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "kind": "verb_product", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "slack.slack_post_message", + "gmail.gmail_send_email", + "twilio.twilio_send_sms", + "slack.slack_add_reaction", + "slack.slack_list_channels", + "slack.slack_get_users", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_create_channel" + ], + "hit1": "discord.discord_send_message", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "kind": "resource", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "ranking": [ + "github.list_pull_requests", + "github.search_pull_requests", + "memory.open_nodes", + "gitlab.gitlab_list_merge_requests", + "github.create_pull_request", + "github.pull_request_read", + "github.update_pull_request", + "github.merge_pull_request", + "github.update_pull_request_branch", + "github.pull_request_review_write" + ], + "hit1": "github.list_pull_requests", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 434, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1009, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "ranking": [ + "github.create_pull_request", + "github.pull_request_read", + "github.update_pull_request", + "github.merge_pull_request", + "memory.open_nodes", + "github.update_pull_request_branch", + "github.pull_request_review_write", + "git.git_status", + "git.git_diff", + "git.git_commit" + ], + "hit1": "github.create_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 405, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 560, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 978, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "kind": "verb_product", + "gold": [ + "postgres.query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "docker.docker_run", + "postgres.explain_query", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "prometheus.prom_query", + "github.actions_run_trigger", + "github.create_check_run", + "github.get_check_run" + ], + "hit1": "postgres.query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "kind": "paraphrase", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "ranking": [ + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "brave_search.brave_local_search", + "github.issue_read", + "memory.read_graph", + "github.pull_request_read", + "github.delete_file", + "filesystem.write_file", + "filesystem.edit_file" + ], + "hit1": "filesystem.read_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 398, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 798, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "kind": "paraphrase", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "ranking": [ + "github.push_files", + "filesystem.search_files", + "slack.list_files", + "google_drive.gdrive_list_folder" + ], + "hit1": "github.push_files", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 85, + "k3_calls": 29, + "k3_tokens": 7932, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 17, + "k5_tokens": 7554, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7648, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "ranking": [ + "memory.read_graph" + ], + "hit1": "memory.read_graph", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 90, + "k3_calls": 30, + "k3_tokens": 8091, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7815, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7608, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "kind": "verb_product", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "ranking": [ + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "filesystem.list_directory_with_sizes", + "github.search_repositories", + "github.search_code", + "github.search_commits", + "github.search_issues", + "github.search_users", + "github.search_orgs", + "filesystem.search_files" + ], + "hit1": "brave_search.brave_web_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "kind": "verb_product", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "ranking": [ + "fetch.fetch", + "github.get_file_contents" + ], + "hit1": "fetch.fetch", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "kind": "identifier", + "gold": [ + "git.git_status" + ], + "alts": [], + "ranking": [ + "git.git_status", + "git.git_show", + "git.git_diff", + "git.git_commit", + "git.git_add", + "git.git_reset", + "git.git_log", + "git.git_checkout", + "git.git_branch", + "git.git_diff_unstaged" + ], + "hit1": "git.git_status", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 325, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "kind": "paraphrase", + "gold": [ + "git.git_commit" + ], + "alts": [], + "ranking": [ + "github.get_commit", + "git.git_commit", + "git.git_diff_staged" + ], + "hit1": "github.get_commit", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 310, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_issue", + "linear.linear_update_issue", + "linear.linear_get_issue", + "linear.linear_assign_issue", + "github.create_issue", + "linear.linear_create_comment", + "linear.linear_create_project", + "jira.jira_create_issue", + "gitlab.gitlab_create_issue", + "bitbucket.bitbucket_create_issue" + ], + "hit1": "linear.linear_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 322, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 528, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 974, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "kind": "paraphrase", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_project", + "twilio.twilio_make_call", + "vercel.vercel_create_project", + "asana.asana_create_project", + "asana.asana_create_task", + "asana.asana_get_task", + "asana.asana_update_task", + "clickup.clickup_create_task", + "clickup.clickup_get_task", + "clickup.clickup_update_task" + ], + "hit1": "linear.linear_create_project", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 146, + "k3_calls": 49, + "k3_tokens": 12480, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 30, + "k5_tokens": 12220, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11860, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "github.add_issue_comment", + "github.update_issue_comment", + "slack.slack_post_message", + "git.git_status", + "git.git_diff", + "git.git_commit", + "git.git_add", + "git.git_reset", + "git.git_log" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 307, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 732, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "kind": "ambiguous", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "ranking": [ + "github.merge_pull_request", + "gitlab.gitlab_merge_merge_request", + "github.create_pull_request", + "github.pull_request_read", + "github.update_pull_request", + "gitlab.gitlab_create_merge_request", + "gitlab.gitlab_get_merge_request", + "github.update_pull_request_branch", + "github.pull_request_review_write", + "github.list_pull_requests" + ], + "hit1": "github.merge_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 354, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 586, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1108, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "kind": "verb_product", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "ranking": [ + "github.actions_run_trigger", + "github.actions_list", + "github.actions_get", + "circleci.circleci_trigger_pipeline", + "git.git_status", + "git.git_diff", + "git.git_commit", + "git.git_add", + "git.git_reset", + "git.git_log" + ], + "hit1": "github.actions_run_trigger", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 719, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "kind": "resource", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "ranking": [ + "github.list_dependabot_alerts", + "github.get_dependabot_alert", + "github.list_code_scanning_alerts", + "github.list_secret_scanning_alerts", + "github.get_me", + "github.get_teams", + "github.get_commit", + "github.get_tag", + "github.get_discussion", + "github.actions_get" + ], + "hit1": "github.list_dependabot_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 287, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 421, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 801, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "kind": "verb_product", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_issues", + "github.list_issues", + "sentry.sentry_list_events", + "sentry.sentry_list_projects", + "linear.linear_list_issues", + "gitlab.gitlab_list_issues", + "github.search_issues", + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "linear.linear_search_issues" + ], + "hit1": "sentry.sentry_list_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 289, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 425, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_create_invoice", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "stripe.stripe_create_refund", + "stripe.stripe_create_payment_intent", + "stripe.stripe_list_customers", + "stripe.stripe_list_invoices", + "stripe.stripe_list_subscriptions" + ], + "hit1": "stripe.stripe_create_customer", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "kind": "paraphrase", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_refund", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_list_customers", + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_create_product", + "stripe.stripe_create_price" + ], + "hit1": "stripe.stripe_create_refund", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "kind": "verb_product", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "ranking": [ + "gmail.gmail_send_email", + "gmail.gmail_get_email", + "gmail.gmail_modify_email", + "gmail.gmail_trash_email", + "filesystem.list_directory_with_sizes", + "twilio.twilio_send_sms", + "discord.discord_send_message", + "gmail.gmail_search_emails", + "gmail.gmail_create_draft", + "gmail.gmail_list_labels" + ], + "hit1": "gmail.gmail_send_email", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 716, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "kind": "verb_product", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "ranking": [ + "gmail.gmail_search_emails", + "gmail.gmail_send_email", + "gmail.gmail_get_email", + "gmail.gmail_create_draft", + "gmail.gmail_list_labels", + "gmail.gmail_modify_email", + "gmail.gmail_trash_email", + "github.search_repositories", + "github.search_code", + "github.search_commits" + ], + "hit1": "gmail.gmail_search_emails", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "kind": "paraphrase", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "ranking": [ + "slack.slack_upload_file", + "github.assign_copilot_to_issue", + "slack.slack_reply_to_thread", + "github.add_comment_to_pending_review", + "github.delete_file", + "filesystem.read_file", + "filesystem.write_file", + "filesystem.edit_file", + "filesystem.move_file", + "slack.create_file" + ], + "hit1": "slack.slack_upload_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 254, + "k3_calls": 85, + "k3_tokens": 20407, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 51, + "k5_tokens": 19659, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 26, + "k10_tokens": 19548, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_create_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_delete_event", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.create_gist", + "filesystem.create_directory", + "memory.create_entities", + "memory.create_relations" + ], + "hit1": "google_calendar.gcal_create_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 480, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 844, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_list_pods", + "github.list_commits", + "github.list_branches", + "github.list_tags", + "github.list_releases", + "github.list_issues", + "github.list_notifications", + "github.list_discussions", + "github.actions_list", + "github.list_gists" + ], + "hit1": "kubernetes.k8s_list_pods", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 393, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 750, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_get_pod", + "docker.docker_logs", + "github.actions_get_job_logs", + "datadog.datadog_list_logs", + "github.get_me", + "github.get_teams", + "github.get_commit", + "github.get_tag", + "github.get_discussion" + ], + "hit1": "kubernetes.k8s_pod_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 803, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "ranking": [ + "docker.docker_run", + "docker.docker_ps", + "docker.docker_logs", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_images", + "docker.docker_build", + "docker.docker_exec", + "github.actions_run_trigger", + "github.create_check_run" + ], + "hit1": "docker.docker_run", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "kind": "verb_product", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "ranking": [ + "docker.docker_logs", + "git.git_show", + "kubernetes.k8s_pod_logs", + "docker.docker_ps", + "docker.docker_run", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_images", + "docker.docker_build", + "docker.docker_exec" + ], + "hit1": "docker.docker_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 241, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 648, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "kind": "ambiguous", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "ranking": [ + "playwright.browser_navigate", + "puppeteer.puppeteer_navigate", + "playwright.browser_snapshot", + "playwright.browser_click", + "playwright.browser_type", + "playwright.browser_evaluate", + "github.assign_copilot_to_issue", + "slack.slack_reply_to_thread", + "github.add_comment_to_pending_review", + "playwright.browser_fill_form" + ], + "hit1": "playwright.browser_navigate", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 169, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 309, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 756, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "kind": "ambiguous", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "ranking": [ + "playwright.browser_take_screenshot", + "puppeteer.puppeteer_screenshot", + "notion.notion_get_page", + "notion.notion_create_page", + "notion.notion_update_page", + "confluence.confluence_get_page", + "confluence.confluence_create_page", + "confluence.confluence_update_page" + ], + "hit1": "playwright.browser_take_screenshot", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_create_merge_request", + "github.merge_pull_request", + "gitlab.gitlab_get_merge_request", + "github.create_pull_request", + "git.git_create_branch", + "git.git_status", + "git.git_diff", + "git.git_commit", + "git.git_add" + ], + "hit1": "gitlab.gitlab_merge_merge_request", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 323, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 552, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 881, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "kind": "verb_product", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "ranking": [ + "notion.notion_search", + "notion.notion_get_page", + "notion.notion_create_page", + "notion.notion_update_page", + "notion.notion_get_block", + "notion.notion_append_block", + "notion.notion_query_database", + "notion.notion_create_database", + "notion.notion_list_users", + "notion.notion_get_comments" + ], + "hit1": "notion.notion_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 351, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "kind": "verb_product", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "ranking": [ + "notion.notion_create_page", + "notion.notion_get_page", + "notion.notion_update_page", + "confluence.confluence_create_page", + "notion.notion_create_database", + "notion.notion_search", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "notion.notion_get_block", + "notion.notion_append_block" + ], + "hit1": "notion.notion_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 748, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "kind": "verb_product", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "ranking": [ + "notion.notion_query_database", + "notion.notion_create_database", + "postgres.query", + "postgres.explain_query", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "prometheus.prom_query", + "notion.notion_search", + "datadog.datadog_query_metrics" + ], + "hit1": "notion.notion_query_database", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_search", + "zendesk.zendesk_list_tickets", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.create_gist", + "filesystem.create_directory" + ], + "hit1": "zendesk.zendesk_create_ticket", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 813, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.create_gist", + "filesystem.create_directory", + "memory.create_entities", + "memory.create_relations", + "github.create_milestone" + ], + "hit1": "hubspot.hubspot_create_contact", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 476, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 813, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "kind": "verb_product", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_create_lead", + "salesforce.salesforce_convert_lead", + "salesforce.salesforce_create_record", + "salesforce.salesforce_query", + "salesforce.salesforce_update_record", + "salesforce.salesforce_get_record", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_list_opportunities", + "github.create_repository", + "github.create_branch" + ], + "hit1": "salesforce.salesforce_create_lead", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 784, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "kind": "identifier", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "docker.docker_run", + "postgres.explain_query", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "prometheus.prom_query", + "github.actions_run_trigger", + "github.create_check_run", + "github.get_check_run" + ], + "hit1": "postgres.query", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 408, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "kind": "verb_product", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "ranking": [ + "shopify.shopify_list_orders", + "shopify.shopify_list_products", + "shopify.shopify_list_customers", + "shopify.shopify_get_product", + "shopify.shopify_create_product", + "shopify.shopify_update_product", + "shopify.shopify_get_order", + "shopify.shopify_create_order", + "github.list_commits", + "github.list_branches" + ], + "hit1": "shopify.shopify_list_orders", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.create_gist", + "filesystem.create_directory", + "memory.create_entities" + ], + "hit1": "pagerduty.pagerduty_create_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 818, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "kind": "paraphrase", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "ranking": [ + "twilio.twilio_make_call" + ], + "hit1": "twilio.twilio_make_call", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 244, + "k3_calls": 82, + "k3_tokens": 19786, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 49, + "k5_tokens": 18956, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 25, + "k10_tokens": 18737, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "kind": "verb_product", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "ranking": [ + "datadog.datadog_query_metrics", + "postgres.query", + "prometheus.prom_list_metrics", + "postgres.explain_query", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "prometheus.prom_query", + "datadog.datadog_list_monitors", + "datadog.datadog_get_monitor" + ], + "hit1": "datadog.datadog_query_metrics", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 225, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 619, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "kind": "verb_product", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "ranking": [ + "datadog.datadog_list_logs", + "docker.docker_logs", + "kubernetes.k8s_pod_logs", + "datadog.datadog_query_metrics", + "datadog.datadog_list_monitors", + "datadog.datadog_get_monitor", + "datadog.datadog_create_monitor", + "github.actions_get_job_logs", + "github.search_repositories", + "github.search_code" + ], + "hit1": "datadog.datadog_list_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 280, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 444, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 871, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_list_buckets", + "github.list_commits", + "github.list_branches", + "github.list_tags", + "github.list_releases", + "github.list_issues", + "github.list_notifications", + "github.list_discussions", + "github.actions_list", + "github.list_gists" + ], + "hit1": "aws_s3.s3_list_buckets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 236, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 735, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "kind": "paraphrase", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_get_object", + "aws_s3.s3_put_object", + "aws_s3.s3_delete_object", + "slack.slack_upload_file", + "github.assign_copilot_to_issue", + "slack.slack_reply_to_thread", + "github.add_comment_to_pending_review" + ], + "hit1": "aws_s3.s3_get_object", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 839, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_purge_cache" + ], + "hit1": "cloudflare.cf_purge_cache", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 756, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_create_dns", + "salesforce.salesforce_create_record", + "cloudflare.cf_list_dns", + "salesforce.salesforce_update_record", + "salesforce.salesforce_get_record", + "salesforce.salesforce_delete_record", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.create_gist" + ], + "hit1": "cloudflare.cf_create_dns", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 415, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 879, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_deployments", + "kubernetes.k8s_list_deployments", + "vercel.vercel_list_projects", + "vercel.vercel_get_deployment", + "vercel.vercel_create_project", + "github.list_commits", + "github.list_branches", + "github.list_tags", + "github.list_releases", + "github.list_issues" + ], + "hit1": "vercel.vercel_list_deployments", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 173, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 710, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "ranking": [ + "supabase.supabase_insert", + "mongodb.mongo_insert", + "supabase.supabase_select", + "supabase.supabase_update", + "supabase.supabase_list_tables", + "supabase.supabase_invoke_fn" + ], + "hit1": "supabase.supabase_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 679, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "kind": "paraphrase", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "ranking": [ + "figma.figma_get_images", + "docker.docker_images", + "figma.figma_get_file", + "figma.figma_get_comments", + "figma.figma_post_comment", + "figma.figma_get_file_nodes" + ], + "hit1": "figma.figma_get_images", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 325, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 651, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "kind": "verb_product", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "ranking": [ + "twilio.twilio_send_sms", + "filesystem.list_directory_with_sizes", + "gmail.gmail_send_email", + "twilio.twilio_list_messages", + "twilio.twilio_make_call", + "discord.discord_send_message" + ], + "hit1": "twilio.twilio_send_sms", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "kind": "verb_product", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "ranking": [ + "okta.okta_create_user", + "okta.okta_get_user", + "okta.okta_deactivate_user", + "linear.create_user", + "linear.get_user", + "linear.update_user", + "linear.delete_user", + "okta.okta_list_users", + "slack.slack_get_user_profile", + "github.create_repository" + ], + "hit1": "okta.okta_create_user", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 631, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "postgres.explain_query", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "prometheus.prom_query", + "notion.notion_query_database", + "datadog.datadog_query_metrics", + "prometheus.prom_query_range" + ], + "hit1": "postgres.query", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.16666666666666666, + "ndcg@10": 0.3562071871080222, + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 365, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 704, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "kind": "verb_product", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "ranking": [ + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "grafana.grafana_create_annotation", + "github.search_repositories", + "github.search_code", + "github.search_commits", + "github.search_issues", + "github.search_users", + "github.search_orgs", + "filesystem.search_files" + ], + "hit1": "grafana.grafana_search_dashboards", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 376, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 685, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "kind": "verb_product", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "ranking": [ + "circleci.circleci_trigger_pipeline", + "github.actions_run_trigger", + "gitlab.create_pipeline", + "gitlab.get_pipeline", + "gitlab.update_pipeline", + "gitlab.delete_pipeline", + "gitlab.gitlab_create_pipeline", + "circleci.circleci_get_pipeline" + ], + "hit1": "circleci.circleci_trigger_pipeline", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 308, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 430, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 701, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "kind": "identifier", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "ranking": [ + "terraform.tf_plan", + "docker.docker_run", + "github.actions_run_trigger", + "github.create_check_run", + "github.get_check_run", + "github.update_check_run", + "github.delete_check_run" + ], + "hit1": "terraform.tf_plan", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 275, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 430, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 712, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "kind": "verb_product", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "ranking": [ + "asana.asana_create_task", + "asana.asana_get_task", + "asana.asana_update_task", + "asana.asana_create_project", + "clickup.clickup_create_task", + "asana.asana_list_tasks", + "clickup.clickup_get_task", + "clickup.clickup_update_task", + "github.create_repository", + "github.create_branch" + ], + "hit1": "asana.asana_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 821, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "ranking": [ + "trello.trello_create_card", + "trello.trello_create_board", + "trello.trello_list_cards", + "trello.trello_add_comment", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.create_gist", + "filesystem.create_directory", + "memory.create_entities" + ], + "hit1": "trello.trello_create_card", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 809, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "ranking": [ + "playwright.browser_click", + "puppeteer.puppeteer_click", + "asana.asana_create_task", + "clickup.clickup_create_task", + "asana.asana_get_task", + "asana.asana_update_task", + "clickup.clickup_get_task", + "clickup.clickup_update_task", + "github.create_repository", + "github.create_branch" + ], + "hit1": "playwright.browser_click", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 496, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 800, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "kind": "verb_product", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "ranking": [ + "confluence.confluence_create_page", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "notion.notion_create_page", + "confluence.confluence_search", + "notion.notion_get_page", + "notion.notion_update_page", + "github.create_repository", + "github.create_branch", + "github.create_issue" + ], + "hit1": "confluence.confluence_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 835, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "kind": "verb_product", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "slack.slack_post_message", + "gmail.gmail_send_email", + "twilio.twilio_send_sms", + "discord.discord_list_channels", + "discord.discord_get_channel", + "discord.discord_add_reaction", + "discord.discord_list_members" + ], + "hit1": "discord.discord_send_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 664, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "ranking": [ + "github.search_repositories", + "github.search_code", + "github.search_commits", + "github.search_issues", + "github.search_users", + "github.search_orgs", + "filesystem.search_files", + "memory.search_nodes", + "notion.notion_search", + "google_drive.gdrive_search" + ], + "hit1": "github.search_repositories", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 290, + "k3_calls": 97, + "k3_tokens": 22834, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 58, + "k5_tokens": 22047, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 29, + "k10_tokens": 21525, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "playwright.browser_press_key", + "redis.redis_set", + "redis.redis_delete", + "redis.redis_keys", + "github.get_me", + "github.get_teams", + "github.get_commit", + "github.get_tag", + "github.get_discussion" + ], + "hit1": "redis.redis_get", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 628, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_find", + "mongodb.mongo_insert", + "mongodb.mongo_update", + "mongodb.mongo_delete", + "mongodb.mongo_list_collections" + ], + "hit1": "mongodb.mongo_find", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "kind": "verb_product", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "ranking": [ + "postgres.list_tables", + "sqlite.sqlite_list_tables", + "mysql.mysql_list_tables", + "supabase.supabase_list_tables", + "github.list_commits", + "github.list_branches", + "github.list_tags", + "github.list_releases", + "github.list_issues", + "github.list_notifications" + ], + "hit1": "postgres.list_tables", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "kind": "verb_product", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "ranking": [ + "postgres.describe_table", + "sqlite.sqlite_describe_table" + ], + "hit1": "postgres.describe_table", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 187, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "linear.linear_create_issue", + "jira.jira_create_issue", + "gitlab.gitlab_create_issue", + "bitbucket.bitbucket_create_issue", + "github.issue_write", + "github.issue_read", + "github.add_issue_comment", + "github.update_issue_comment", + "linear.linear_update_issue" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8955796649110442, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 398, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 580, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1139, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "ranking": [ + "github.create_issue", + "linear.linear_create_issue", + "jira.jira_create_issue", + "gitlab.gitlab_create_issue", + "bitbucket.bitbucket_create_issue", + "github.issue_write", + "github.issue_read", + "github.add_issue_comment", + "github.update_issue_comment", + "linear.linear_update_issue" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 397, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 579, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1138, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "kind": "identifier", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "ranking": [ + "filesystem.list_directory", + "filesystem.create_directory", + "filesystem.directory_tree", + "filesystem.list_directory_with_sizes", + "github.list_commits", + "github.list_branches", + "github.list_tags", + "github.list_releases", + "github.list_issues", + "github.list_notifications" + ], + "hit1": "filesystem.list_directory", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "kind": "paraphrase", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 116, + "k3_calls": 39, + "k3_tokens": 9799, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 24, + "k5_tokens": 9727, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9475, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "kind": "paraphrase", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "ranking": [ + "time.convert_time", + "time.get_current_time" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8597186998521972, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 309, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 731, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "kind": "verb_product", + "gold": [ + "time.convert_time" + ], + "alts": [], + "ranking": [ + "time.convert_time", + "time.get_current_time", + "salesforce.salesforce_convert_lead" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 732, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "kind": "verb_product", + "gold": [ + "github.star_repository" + ], + "alts": [], + "ranking": [ + "github.star_repository", + "git.git_status", + "git.git_diff", + "git.git_commit", + "git.git_add", + "git.git_reset", + "git.git_log", + "git.git_checkout", + "git.git_show", + "git.git_branch" + ], + "hit1": "github.star_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 612, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "kind": "resource", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "ranking": [ + "github.fork_repository", + "github.create_repository", + "github.delete_repository", + "github.star_repository", + "github.unstar_repository", + "github.list_repository_collaborators", + "github.get_repository_tree" + ], + "hit1": "github.fork_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 248, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "kind": "verb_product", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "ranking": [ + "github.get_latest_release", + "gitlab.get_release", + "gitlab.create_release", + "gitlab.update_release", + "gitlab.delete_release", + "github.get_release_by_tag", + "git.git_status", + "git.git_diff", + "git.git_commit", + "git.git_add" + ], + "hit1": "github.get_latest_release", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "kind": "resource", + "gold": [ + "github.create_gist" + ], + "alts": [], + "ranking": [ + "github.create_gist", + "github.get_gist", + "github.update_gist", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "filesystem.create_directory", + "memory.create_entities", + "memory.create_relations", + "github.create_milestone" + ], + "hit1": "github.create_gist", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 800, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "kind": "verb_product", + "gold": [ + "github.search_code" + ], + "alts": [], + "ranking": [ + "github.search_code", + "git.git_status", + "git.git_diff", + "git.git_commit", + "git.git_add", + "git.git_reset", + "git.git_log", + "git.git_checkout", + "git.git_show", + "git.git_branch" + ], + "hit1": "github.search_code", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 612, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "kind": "verb_product", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "ranking": [ + "github.list_notifications", + "github.mark_all_notifications_read", + "git.git_status", + "git.git_diff", + "git.git_commit", + "git.git_add", + "git.git_reset", + "git.git_log", + "git.git_checkout", + "git.git_show" + ], + "hit1": "github.list_notifications", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 175, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "kind": "verb_product", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "ranking": [ + "github.assign_copilot_to_issue", + "linear.linear_assign_issue", + "jira.jira_assign_issue", + "github.request_copilot_review", + "slack.slack_reply_to_thread", + "github.create_issue", + "github.issue_write", + "github.issue_read", + "github.add_comment_to_pending_review", + "github.add_issue_comment" + ], + "hit1": "github.assign_copilot_to_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 457, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1083, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "kind": "resource", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "ranking": [ + "github.list_secret_scanning_alerts", + "github.get_secret_scanning_alert", + "github.list_code_scanning_alerts", + "github.get_code_scanning_alert", + "github.list_dependabot_alerts", + "github.get_me", + "github.get_teams", + "github.get_commit", + "github.get_tag", + "github.get_discussion" + ], + "hit1": "github.list_secret_scanning_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 818, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "ranking": [ + "github.create_or_update_file", + "slack.update_file", + "slack.create_file", + "git.git_create_branch", + "google_drive.gdrive_update_file", + "google_drive.gdrive_create_file", + "git.git_status", + "git.git_diff", + "git.git_commit", + "git.git_add" + ], + "hit1": "github.create_or_update_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 508, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 861, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "kind": "resource", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "ranking": [ + "github.list_repository_collaborators", + "github.list_commits", + "github.list_branches", + "github.list_tags", + "github.list_releases", + "github.list_issues", + "github.list_notifications", + "github.list_discussions", + "github.actions_list", + "github.list_gists" + ], + "hit1": "github.list_repository_collaborators", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 274, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 773, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "kind": "verb_product", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "ranking": [ + "slack.slack_reply_to_thread", + "slack.slack_get_thread_replies", + "slack.slack_post_message", + "slack.slack_add_reaction", + "slack.slack_list_channels", + "slack.slack_get_users", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_create_channel", + "slack.slack_upload_file" + ], + "hit1": "slack.slack_reply_to_thread", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 414, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "kind": "verb_product", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "ranking": [ + "slack.slack_add_reaction", + "discord.discord_add_reaction", + "memory.add_observations", + "git.git_add", + "github.add_issue_comment", + "jira.jira_add_comment", + "jira.jira_add_worklog", + "trello.trello_add_comment", + "slack.slack_post_message", + "slack.slack_list_channels" + ], + "hit1": "slack.slack_add_reaction", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 426, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 792, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "kind": "verb_product", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "ranking": [ + "slack.slack_search_messages", + "twilio.twilio_list_messages", + "github.search_repositories", + "github.search_code", + "github.search_commits", + "github.search_issues", + "github.search_users", + "github.search_orgs", + "filesystem.search_files", + "memory.search_nodes" + ], + "hit1": "slack.slack_search_messages", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "kind": "verb_product", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "ranking": [ + "slack.slack_create_channel", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_get_channel_history", + "slack.slack_set_channel_topic", + "discord.discord_get_channel", + "slack.slack_post_message", + "slack.slack_add_reaction", + "slack.slack_list_channels", + "slack.slack_get_users" + ], + "hit1": "slack.slack_create_channel", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 685, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "kind": "verb_product", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_transition_issue", + "jira.jira_create_issue", + "jira.jira_update_issue", + "jira.jira_get_issue", + "jira.jira_assign_issue", + "jira.jira_search_issues", + "jira.jira_add_comment", + "jira.jira_list_projects", + "jira.jira_create_sprint", + "jira.jira_add_worklog" + ], + "hit1": "jira.jira_transition_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 270, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "kind": "identifier", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "ranking": [ + "jira.jira_search_issues", + "filesystem.list_directory_with_sizes", + "jira.jira_create_issue", + "jira.jira_update_issue", + "jira.jira_get_issue", + "jira.jira_add_comment", + "jira.jira_transition_issue", + "jira.jira_assign_issue", + "jira.jira_list_projects", + "jira.jira_create_sprint" + ], + "hit1": "jira.jira_search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "kind": "verb_product", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "ranking": [ + "jira.jira_add_worklog", + "jira.jira_add_comment", + "github.add_comment_to_pending_review", + "memory.add_observations", + "git.git_add", + "github.assign_copilot_to_issue", + "slack.slack_reply_to_thread", + "github.add_issue_comment", + "slack.slack_add_reaction", + "trello.trello_add_comment" + ], + "hit1": "jira.jira_add_worklog", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 334, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 914, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "kind": "verb_product", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "ranking": [ + "linear.linear_list_teams", + "github.get_teams", + "linear.linear_list_issues", + "linear.linear_list_projects", + "linear.linear_create_issue", + "linear.linear_update_issue", + "linear.linear_get_issue", + "linear.linear_search_issues", + "linear.linear_create_comment", + "linear.linear_create_project" + ], + "hit1": "linear.linear_list_teams", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 176, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "ranking": [ + "linear.linear_create_comment", + "linear.linear_create_issue", + "linear.linear_update_issue", + "linear.linear_get_issue", + "linear.linear_assign_issue", + "github.add_issue_comment", + "github.update_issue_comment", + "linear.create_comment", + "linear.get_comment", + "linear.update_comment" + ], + "hit1": "linear.linear_create_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 334, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 464, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 885, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "kind": "paraphrase", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "ranking": [ + "filesystem.move_file", + "brave_search.brave_local_search", + "github.delete_file", + "filesystem.read_file", + "filesystem.write_file", + "filesystem.edit_file", + "slack.create_file", + "slack.get_file", + "slack.update_file", + "slack.delete_file" + ], + "hit1": "filesystem.move_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 271, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 398, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 760, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "kind": "paraphrase", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "ranking": [ + "filesystem.search_files", + "github.push_files", + "slack.list_files", + "github.get_release_by_tag", + "github.search_repositories", + "github.search_code", + "github.search_commits", + "github.search_issues", + "github.search_users", + "github.search_orgs" + ], + "hit1": "filesystem.search_files", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 280, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 772, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "kind": "paraphrase", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "ranking": [ + "filesystem.edit_file", + "playwright.browser_tab_new", + "filesystem.list_directory_with_sizes", + "github.delete_file", + "filesystem.read_file", + "filesystem.write_file", + "filesystem.move_file", + "slack.create_file", + "slack.get_file", + "slack.update_file" + ], + "hit1": "filesystem.edit_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 398, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "ranking": [ + "memory.read_graph", + "github.issue_read", + "filesystem.read_file", + "github.pull_request_read", + "filesystem.read_text_file", + "filesystem.read_media_file", + "github.mark_all_notifications_read" + ], + "hit1": "memory.read_graph", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 387, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 642, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "kind": "verb_product", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "memory.open_nodes", + "figma.figma_get_file_nodes", + "github.search_repositories", + "github.search_code", + "github.search_commits", + "github.search_issues", + "github.search_users", + "github.search_orgs", + "filesystem.search_files" + ], + "hit1": "memory.search_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 361, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 670, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "kind": "verb_product", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "ranking": [ + "git.git_diff_unstaged", + "git.git_show", + "git.git_diff", + "git.git_diff_staged", + "git.git_status", + "git.git_commit", + "git.git_add", + "git.git_reset", + "git.git_log", + "git.git_checkout" + ], + "hit1": "git.git_diff_unstaged", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 604, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "kind": "identifier", + "gold": [ + "git.git_add" + ], + "alts": [], + "ranking": [ + "git.git_add", + "github.push_files", + "filesystem.search_files", + "slack.list_files", + "filesystem.list_directory_with_sizes", + "memory.add_observations", + "git.git_status", + "git.git_diff", + "git.git_commit", + "git.git_reset" + ], + "hit1": "git.git_add", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "kind": "verb_product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "ranking": [ + "shopify.shopify_create_product", + "shopify.shopify_get_product", + "shopify.shopify_update_product", + "stripe.stripe_create_product", + "shopify.shopify_create_order", + "shopify.shopify_list_products", + "shopify.shopify_list_orders", + "shopify.shopify_get_order", + "shopify.shopify_list_customers", + "github.create_repository" + ], + "hit1": "shopify.shopify_create_product", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "kind": "verb_product", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "ranking": [ + "stripe.stripe_cancel_subscription", + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_list_customers", + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "stripe.stripe_list_subscriptions", + "stripe.stripe_create_refund" + ], + "hit1": "stripe.stripe_cancel_subscription", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "kind": "resource", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.create_gist", + "filesystem.create_directory", + "memory.create_entities", + "memory.create_relations", + "github.create_milestone" + ], + "hit1": "stripe.stripe_create_payment_intent", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 477, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 814, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_resolve_incident" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 787, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "kind": "verb_product", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "ranking": [ + "datadog.datadog_create_monitor", + "datadog.datadog_get_monitor", + "datadog.datadog_query_metrics", + "datadog.datadog_list_monitors", + "datadog.datadog_list_logs", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.create_gist", + "filesystem.create_directory" + ], + "hit1": "datadog.datadog_create_monitor", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 904, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "kind": "paraphrase", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "ranking": [ + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "pagerduty.pagerduty_resolve_incident", + "sentry.sentry_list_issues", + "sentry.sentry_list_events", + "sentry.sentry_list_projects", + "github.create_issue", + "github.issue_write", + "github.issue_read", + "github.add_issue_comment" + ], + "hit1": "sentry.sentry_get_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 935, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_scale_deployment", + "vercel.vercel_get_deployment" + ], + "hit1": "kubernetes.k8s_scale_deployment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_apply", + "terraform.tf_apply" + ], + "hit1": "kubernetes.k8s_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 307, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "ranking": [ + "docker.docker_stop", + "docker.docker_ps", + "docker.docker_logs", + "docker.docker_run", + "docker.docker_rm", + "docker.docker_images", + "docker.docker_build", + "docker.docker_exec" + ], + "hit1": "docker.docker_stop", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 174, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 581, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "kind": "verb_product", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "ranking": [ + "docker.docker_build", + "docker.docker_ps", + "docker.docker_logs", + "docker.docker_run", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_images", + "docker.docker_exec" + ], + "hit1": "docker.docker_build", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 332, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 581, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "kind": "verb_product", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "ranking": [ + "playwright.browser_click", + "puppeteer.puppeteer_click" + ], + "hit1": "playwright.browser_click", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 298, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 720, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "kind": "paraphrase", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "ranking": [ + "playwright.browser_type", + "filesystem.read_text_file", + "playwright.browser_navigate", + "playwright.browser_snapshot", + "playwright.browser_click", + "playwright.browser_evaluate", + "playwright.browser_fill_form", + "playwright.browser_take_screenshot", + "playwright.browser_tab_list", + "playwright.browser_tab_new" + ], + "hit1": "playwright.browser_type", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 236, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 598, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "ranking": [ + "github.projects_list", + "linear.linear_list_projects", + "jira.jira_list_projects", + "sentry.sentry_list_projects", + "gitlab.gitlab_list_projects", + "vercel.vercel_list_projects", + "github.projects_get", + "github.projects_write", + "git.git_status", + "git.git_diff" + ], + "hit1": "github.projects_list", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.2, + "ndcg@10": 0.38685280723454163, + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 326, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 268, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 573, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "kind": "verb_product", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "ranking": [ + "github.create_pull_request", + "github.pull_request_read", + "github.update_pull_request", + "github.merge_pull_request", + "github.update_pull_request_branch", + "github.pull_request_review_write", + "bitbucket.bitbucket_create_pr", + "bitbucket.bitbucket_create_issue", + "gitlab.gitlab_create_merge_request", + "bitbucket.bitbucket_list_prs" + ], + "hit1": "github.create_pull_request", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.14285714285714285, + "ndcg@10": 0.3333333333333333, + "hit_rank": 7, + "k3_calls": 3, + "k3_tokens": 1123, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 1177, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1155, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "kind": "verb_product", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "ranking": [ + "gmail.gmail_create_draft", + "gmail.gmail_send_email", + "gmail.gmail_search_emails", + "gmail.gmail_get_email", + "gmail.gmail_list_labels", + "gmail.gmail_modify_email", + "gmail.gmail_trash_email", + "github.create_repository", + "github.create_branch", + "github.create_issue" + ], + "hit1": "gmail.gmail_create_draft", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 265, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 814, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "kind": "verb_product", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_share_file", + "github.delete_file", + "filesystem.read_file", + "filesystem.write_file", + "filesystem.edit_file", + "filesystem.move_file", + "slack.create_file", + "slack.get_file", + "slack.update_file", + "slack.delete_file" + ], + "hit1": "google_drive.gdrive_share_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 278, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 440, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 771, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "kind": "verb_product", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_events", + "google_calendar.gcal_list_events", + "kubernetes.k8s_get_events", + "github.list_commits", + "github.list_branches", + "github.list_tags", + "github.list_releases", + "github.list_issues", + "github.list_notifications", + "github.list_discussions" + ], + "hit1": "sentry.sentry_list_events", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 233, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 408, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 789, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_update_event", + "kubernetes.k8s_delete", + "github.delete_repository", + "github.delete_file", + "memory.delete_entities", + "memory.delete_observations", + "memory.delete_relations", + "mongodb.mongo_delete" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 798, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_projects", + "github.projects_list", + "vercel.vercel_list_deployments", + "linear.linear_list_projects", + "jira.jira_list_projects", + "sentry.sentry_list_projects", + "gitlab.gitlab_list_projects", + "github.projects_get", + "github.projects_write", + "vercel.vercel_get_deployment" + ], + "hit1": "vercel.vercel_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 263, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 576, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "ranking": [ + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update", + "supabase.supabase_list_tables", + "supabase.supabase_invoke_fn" + ], + "hit1": "supabase.supabase_select", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 711, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "kind": "verb_product", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "slack.slack_post_message", + "figma.figma_get_file", + "figma.figma_get_comments", + "figma.figma_get_images", + "linear.create_comment", + "linear.get_comment", + "linear.update_comment", + "linear.delete_comment", + "notion.create_comment" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "kind": "verb_product", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "ranking": [ + "okta.okta_list_users", + "linear.list_users", + "slack.slack_list_users", + "notion.notion_list_users", + "notion.list_users_tokens", + "okta.okta_get_user", + "okta.okta_create_user", + "okta.okta_deactivate_user", + "github.search_users", + "slack.slack_get_users" + ], + "hit1": "okta.okta_list_users", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 209, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query_range", + "postgres.query", + "postgres.explain_query", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "prometheus.prom_query", + "notion.notion_query_database", + "datadog.datadog_query_metrics" + ], + "hit1": "prometheus.prom_query_range", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "kind": "verb_product", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "ranking": [ + "grafana.grafana_create_annotation", + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.create_gist", + "filesystem.create_directory", + "memory.create_entities", + "memory.create_relations" + ], + "hit1": "grafana.grafana_create_annotation", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 398, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 762, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "kind": "identifier", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_apply", + "terraform.tf_apply" + ], + "hit1": "kubernetes.k8s_apply", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 305, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "kind": "verb_product", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "ranking": [ + "asana.asana_list_tasks", + "clickup.clickup_list_tasks", + "asana.asana_create_task", + "asana.asana_get_task", + "asana.asana_update_task", + "asana.asana_create_project", + "github.list_commits", + "github.list_branches", + "github.list_tags", + "github.list_releases" + ], + "hit1": "asana.asana_list_tasks", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "ranking": [ + "trello.trello_create_card", + "trello.trello_add_comment", + "trello.trello_list_cards", + "trello.trello_create_board", + "linear.create_comment", + "linear.get_comment", + "linear.update_comment", + "linear.delete_comment", + "notion.create_comment", + "notion.get_comment" + ], + "hit1": "trello.trello_create_card", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "ranking": [ + "asana.asana_update_task", + "clickup.clickup_update_task", + "playwright.browser_click", + "puppeteer.puppeteer_click", + "asana.asana_create_task", + "asana.asana_get_task", + "clickup.clickup_create_task", + "clickup.clickup_get_task", + "github.update_gist", + "mongodb.mongo_update" + ], + "hit1": "asana.asana_update_task", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 264, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 423, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 788, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "kind": "verb_product", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "ranking": [ + "confluence.confluence_search", + "confluence.confluence_get_page", + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "github.search_repositories", + "github.search_code", + "github.search_commits", + "github.search_issues", + "github.search_users", + "github.search_orgs" + ], + "hit1": "confluence.confluence_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 712, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_deal", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.create_gist", + "filesystem.create_directory", + "memory.create_entities", + "memory.create_relations", + "github.create_milestone", + "github.create_hook" + ], + "hit1": "hubspot.hubspot_create_deal", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 292, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 502, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 856, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_search", + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "github.list_commits", + "github.list_branches", + "github.list_tags", + "github.list_releases", + "github.list_issues" + ], + "hit1": "zendesk.zendesk_list_tickets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 786, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "kind": "verb_product", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "ranking": [ + "discord.discord_list_channels", + "slack.slack_list_channels", + "discord.discord_list_members", + "discord.discord_send_message", + "discord.discord_get_channel", + "discord.discord_add_reaction", + "github.list_commits", + "github.list_branches", + "github.list_tags", + "github.list_releases" + ], + "hit1": "discord.discord_list_channels", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "ranking": [ + "redis.redis_set", + "playwright.browser_press_key", + "redis.redis_get", + "redis.redis_delete", + "redis.redis_keys", + "slack.slack_set_channel_topic" + ], + "hit1": "redis.redis_set", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 603, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert", + "supabase.supabase_insert", + "mongodb.mongo_find", + "mongodb.mongo_update", + "mongodb.mongo_delete", + "mongodb.mongo_list_collections" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "kind": "verb_product", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "docker.docker_run", + "postgres.explain_query", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "prometheus.prom_query", + "github.actions_run_trigger", + "github.create_check_run", + "github.get_check_run" + ], + "hit1": "postgres.query", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.16666666666666666, + "ndcg@10": 0.3562071871080222, + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 408, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 734, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_index_doc" + ], + "hit1": "elasticsearch.es_index_doc", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 198, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 771, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_get_object", + "aws_s3.s3_put_object", + "aws_s3.s3_delete_object", + "github.get_me", + "github.get_teams", + "github.get_commit", + "github.get_tag", + "github.get_discussion", + "github.actions_get", + "github.get_gist" + ], + "hit1": "aws_s3.s3_get_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 735, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones", + "github.list_commits", + "github.list_branches", + "github.list_tags", + "github.list_releases", + "github.list_issues", + "github.list_notifications", + "github.list_discussions", + "github.actions_list", + "github.list_gists" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 374, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 731, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_create_customer", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "stripe.stripe_create_refund", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_customer", + "stripe.stripe_list_customers", + "stripe.stripe_list_invoices", + "stripe.stripe_list_subscriptions" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "kind": "verb_product", + "gold": [ + "github.actions_list" + ], + "alts": [], + "ranking": [ + "github.actions_list", + "github.actions_get", + "github.actions_run_trigger", + "git.git_status", + "git.git_diff", + "git.git_commit", + "git.git_add", + "git.git_reset", + "git.git_log", + "git.git_checkout" + ], + "hit1": "github.actions_list", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 710, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "kind": "verb_product", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "ranking": [ + "github.actions_get_job_logs", + "github.actions_get", + "gitlab.get_job", + "github.actions_list", + "docker.docker_logs", + "kubernetes.k8s_pod_logs", + "gitlab.create_job", + "gitlab.update_job", + "gitlab.delete_job", + "github.actions_run_trigger" + ], + "hit1": "github.actions_get_job_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 822, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "kind": "verb_product", + "gold": [ + "github.create_repository" + ], + "alts": [], + "ranking": [ + "github.create_repository", + "git.git_create_branch", + "github.delete_repository", + "github.fork_repository", + "github.star_repository", + "github.unstar_repository", + "git.git_status", + "git.git_diff", + "git.git_commit", + "git.git_add" + ], + "hit1": "github.create_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "kind": "verb_product", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "ranking": [ + "git.git_checkout", + "git.git_branch", + "git.git_create_branch", + "github.create_branch", + "git.git_status", + "git.git_diff", + "git.git_commit", + "git.git_add", + "git.git_reset", + "git.git_log" + ], + "hit1": "git.git_checkout", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "kind": "identifier", + "gold": [ + "git.git_log" + ], + "alts": [], + "ranking": [ + "git.git_log", + "git.git_show", + "git.git_status", + "git.git_diff", + "git.git_commit", + "git.git_add", + "git.git_reset", + "git.git_checkout", + "git.git_branch", + "git.git_diff_unstaged" + ], + "hit1": "git.git_log", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "kind": "verb_product", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "ranking": [ + "postgres.list_schemas", + "github.list_commits", + "github.list_branches", + "github.list_tags", + "github.list_releases", + "github.list_issues", + "github.list_notifications", + "github.list_discussions", + "github.actions_list", + "github.list_gists" + ], + "hit1": "postgres.list_schemas", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 730, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "kind": "verb_product", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "ranking": [ + "memory.open_nodes", + "memory.read_graph", + "memory.search_nodes", + "figma.figma_get_file_nodes" + ], + "hit1": "memory.open_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 293, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "kind": "paraphrase", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "ranking": [ + "slack.get_file", + "github.get_file_contents", + "github.get_file_blame", + "filesystem.get_file_info", + "google_drive.gdrive_get_file", + "gitlab.gitlab_get_file", + "figma.figma_get_file", + "figma.figma_get_file_nodes", + "github.delete_file", + "filesystem.read_file" + ], + "hit1": "slack.get_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 503, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 782, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "kind": "paraphrase", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "ranking": [ + "filesystem.write_file", + "brave_search.brave_local_search", + "playwright.browser_tab_new", + "github.issue_write", + "github.projects_write", + "github.label_write", + "github.pull_request_review_write", + "github.delete_file", + "filesystem.read_file", + "filesystem.edit_file" + ], + "hit1": "filesystem.write_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 959, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "kind": "verb_product", + "gold": [ + "github.search_issues" + ], + "alts": [], + "ranking": [ + "github.search_issues", + "linear.linear_search_issues", + "jira.jira_search_issues", + "github.list_issues", + "git.git_status", + "git.git_diff", + "git.git_commit", + "git.git_add", + "git.git_reset", + "git.git_log" + ], + "hit1": "github.search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 233, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 387, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 690, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "kind": "paraphrase", + "gold": [ + "github.get_me" + ], + "alts": [], + "ranking": [ + "linear.get_user", + "okta.okta_get_user", + "slack.slack_get_user_profile", + "linear.create_user", + "linear.update_user", + "linear.delete_user", + "git.git_status", + "git.git_diff", + "git.git_commit", + "git.git_add" + ], + "hit1": "linear.get_user", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 11, + "k3_calls": 4, + "k3_tokens": 796, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 3, + "k5_tokens": 1039, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 1.0, + "k10_calls": 2, + "k10_tokens": 1422, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "cost": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 3.7666666666666666, + "p95_calls": 3.549999999999983, + "success_within_1": 0.9, + "success_within_2": 0.94, + "success_within_3": 0.9466666666666667, + "mean_tokens": 924.0666666666667, + "p95_tokens": 975.8499999999945, + "total_tokens": 138610, + "tokens_per_successful_query": 924.0666666666667, + "total_tokens_successful": 138610 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 2.6333333333333333, + "p95_calls": 2.549999999999983, + "success_within_1": 0.9266666666666666, + "success_within_2": 0.9466666666666667, + "success_within_3": 0.9533333333333334, + "mean_tokens": 1035.96, + "p95_tokens": 1114.8999999999976, + "total_tokens": 155394, + "tokens_per_successful_query": 1035.96, + "total_tokens_successful": 155394 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.7933333333333332, + "p95_calls": 1.549999999999983, + "success_within_1": 0.9466666666666667, + "success_within_2": 0.9533333333333334, + "success_within_3": 0.9533333333333334, + "mean_tokens": 1365.6933333333334, + "p95_tokens": 1301.8499999999954, + "total_tokens": 204854, + "tokens_per_successful_query": 1365.6933333333334, + "total_tokens_successful": 204854 + } + } +} diff --git a/experiments/mcp-tool-search/results/bm25_plain.csv b/experiments/mcp-tool-search/results/bm25_plain.csv new file mode 100644 index 00000000..96955718 --- /dev/null +++ b/experiments/mcp-tool-search/results/bm25_plain.csv @@ -0,0 +1,151 @@ +id,query,kind,hit1,recall@1,recall@3,recall@5,mrr@10,ndcg@10,hit_rank,k3_calls,k3_tokens,k3_success1,k5_calls,k5_tokens,k5_success1,k10_calls,k10_tokens,k10_success1,ranking +q001,Create a GitHub issue,verb_product,github.create_issue,1.0,1.0,1.0,1.0000,0.9502,1,1,380,1.0,1,570,1.0,1,1093,1.0,github.create_issue bitbucket.bitbucket_create_issue github.issue_write gitlab.gitlab_create_issue github.issue_read github.add_issue_comment linear.linear_create_issue jira.jira_create_issue linear.linear_create_comment github.assign_copilot_to_issue +q002,Create a Jira ticket,verb_product,jira.jira_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,281,1.0,1,466,1.0,1,951,1.0,jira.jira_create_issue zendesk.zendesk_create_ticket zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket jira.create_board jira.create_sprint jira.create_filter jira.create_webhook jira.create_attachment jira.create_version +q003,Send a Slack message,verb_product,discord.discord_send_message,0.0,1.0,1.0,0.5000,0.6309,2,1,244,1.0,1,427,1.0,1,760,1.0,discord.discord_send_message slack.slack_post_message twilio.twilio_send_sms slack.slack_add_reaction gmail.gmail_send_email discord.discord_add_reaction slack.slack_join_channel slack.slack_leave_channel slack.slack_search_messages slack.slack_create_channel +q004,List open pull requests,resource,github.list_pull_requests,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,460,1.0,1,836,1.0,github.list_pull_requests github.search_pull_requests bitbucket.bitbucket_list_prs gitlab.gitlab_list_merge_requests github.create_pull_request playwright.browser_tab_list memory.open_nodes github.pull_request_read github.merge_pull_request github.update_pull_request_branch +q005,Open a pull request on GitHub,verb_product,github.create_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,364,1.0,1,564,1.0,1,1180,1.0,github.create_pull_request github.request_copilot_review github.pull_request_read github.merge_pull_request github.update_pull_request_branch github.pull_request_review_write github.update_pull_request github.add_comment_to_pending_review github.search_pull_requests bitbucket.bitbucket_create_pr +q006,Run a SQL query against postgres,verb_product,postgres.query,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,295,1.0,1,592,1.0,postgres.query mysql.mysql_query sqlite.sqlite_query postgres.explain_query salesforce.salesforce_query postgres.list_schemas postgres.list_tables postgres.describe_table docker.docker_run github.get_check_run +q007,Read a local file from disk,paraphrase,filesystem.read_file,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,390,1.0,1,809,1.0,filesystem.read_file filesystem.read_text_file filesystem.read_media_file figma.figma_get_file_nodes gitlab.gitlab_get_file figma.figma_get_images github.delete_file memory.read_graph brave_search.brave_local_search github.get_file_contents +q008,What files are in this folder,paraphrase,google_drive.gdrive_list_folder,0.0,0.0,0.0,0.1111,0.1923,9,3,717,0.0,2,756,0.0,1,734,1.0,google_drive.gdrive_list_folder slack.list_files github.push_files filesystem.search_files google_drive.gdrive_search github.create_gist cloudflare.cf_purge_cache github.update_gist filesystem.list_directory filesystem.move_file +q009,Remember this fact in the knowledge graph,paraphrase,memory.read_graph,0.0,1.0,1.0,0.5000,0.5339,2,1,154,1.0,1,261,1.0,1,507,1.0,memory.read_graph memory.create_entities memory.delete_relations memory.create_relations memory.add_observations memory.open_nodes memory.search_nodes memory.delete_entities postgres.list_schemas time.get_current_time +q010,Search the web with Brave,verb_product,brave_search.brave_web_search,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,410,1.0,1,684,1.0,brave_search.brave_web_search brave_search.brave_local_search jira.jira_search_issues memory.search_nodes github.update_pull_request_branch filesystem.list_directory_with_sizes confluence.confluence_search github.search_users github.search_orgs grafana.grafana_search_dashboards +q011,Fetch the contents of a URL,verb_product,fetch.fetch,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,330,1.0,1,663,1.0,fetch.fetch filesystem.read_file playwright.browser_navigate puppeteer.puppeteer_navigate git.git_add playwright.browser_snapshot github.get_file_contents git.git_status docker.docker_logs slack.slack_set_channel_topic +q012,Show git status,identifier,git.git_status,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,299,1.0,1,613,1.0,git.git_status git.git_show git.git_diff_unstaged git.git_diff_staged git.git_log git.git_diff sentry.sentry_update_issue git.git_branch git.git_checkout zendesk.zendesk_list_tickets +q013,Commit staged changes,paraphrase,git.git_commit,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,327,1.0,1,663,1.0,git.git_commit git.git_reset git.git_diff_staged github.get_commit git.git_show git.git_log github.push_files +q014,Create a Linear issue,verb_product,linear.linear_create_comment,0.0,1.0,1.0,0.5000,0.6309,2,1,333,1.0,1,463,1.0,1,917,1.0,linear.linear_create_comment linear.linear_create_issue linear.linear_update_issue linear.linear_get_issue linear.linear_assign_issue linear.linear_create_project bitbucket.bitbucket_create_issue gitlab.gitlab_create_issue linear.create_cycle linear.create_label +q015,Make a task in my project tracker Linear,paraphrase,linear.linear_create_project,0.0,0.0,0.0,0.0000,0.0000,146,49,12414,0.0,30,12154,0.0,15,11794,0.0,linear.linear_create_project asana.asana_get_task clickup.clickup_get_task asana.asana_update_task clickup.clickup_update_task asana.asana_create_task clickup.clickup_create_task mysql.mysql_list_tables mysql.mysql_query linear.get_cycle +q016,Post a comment on a GitHub issue,verb_product,figma.figma_post_comment,0.0,1.0,1.0,0.3333,0.5000,3,1,307,1.0,1,442,1.0,1,876,1.0,figma.figma_post_comment github.update_issue_comment github.add_issue_comment linear.linear_create_comment jira.jira_add_comment github.assign_copilot_to_issue trello.trello_add_comment slack.slack_post_message jira.jira_add_worklog github.issue_read +q017,Merge the pull request,ambiguous,github.merge_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,283,1.0,1,468,1.0,1,1121,1.0,github.merge_pull_request gitlab.gitlab_merge_merge_request gitlab.gitlab_get_merge_request github.update_pull_request_branch github.pull_request_read github.request_copilot_review gitlab.gitlab_create_merge_request github.pull_request_review_write github.update_pull_request github.create_pull_request +q018,Trigger a GitHub Actions workflow,verb_product,github.actions_run_trigger,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,458,1.0,1,721,1.0,github.actions_run_trigger github.actions_get github.actions_list circleci.circleci_trigger_pipeline github.actions_get_job_logs github.get_milestone github.delete_milestone github.get_hook github.delete_hook github.get_environment +q019,Get Dependabot alerts,resource,github.list_dependabot_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,412,1.0,1,715,1.0,github.list_dependabot_alerts github.get_dependabot_alert github.list_secret_scanning_alerts github.list_code_scanning_alerts redis.redis_get kubernetes.k8s_get_pod shopify.shopify_get_product shopify.shopify_get_order vercel.vercel_get_deployment grafana.grafana_get_dashboard +q020,List Sentry issues,verb_product,sentry.sentry_list_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,395,1.0,1,796,1.0,sentry.sentry_list_issues sentry.sentry_list_projects sentry.sentry_list_events gitlab.gitlab_list_issues linear.linear_list_issues sentry.sentry_get_issue github.list_issues sentry.sentry_update_issue github.search_issues linear.linear_search_issues +q021,Create a Stripe customer,verb_product,stripe.stripe_create_customer,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,322,1.0,1,669,1.0,stripe.stripe_create_customer stripe.stripe_create_invoice stripe.stripe_get_customer stripe.stripe_create_payment_intent stripe.stripe_list_invoices stripe.stripe_list_subscriptions stripe.stripe_create_product stripe.stripe_create_refund stripe.stripe_create_price stripe.stripe_list_customers +q022,Refund a Stripe payment,paraphrase,stripe.stripe_create_refund,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,331,1.0,1,647,1.0,stripe.stripe_create_refund stripe.stripe_get_payment_intent stripe.stripe_create_payment_intent stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_list_customers stripe.stripe_create_product stripe.stripe_list_subscriptions +q023,Send an email with Gmail,verb_product,gmail.gmail_send_email,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,405,1.0,1,713,1.0,gmail.gmail_send_email gmail.gmail_get_email gmail.gmail_trash_email twilio.twilio_send_sms gmail.gmail_modify_email gmail.gmail_list_labels puppeteer.puppeteer_click puppeteer.puppeteer_fill gmail.gmail_search_emails gmail.gmail_create_draft +q024,Search my Gmail inbox,verb_product,gmail.gmail_search_emails,1.0,1.0,1.0,1.0000,1.0000,1,1,178,1.0,1,315,1.0,1,691,1.0,gmail.gmail_search_emails gmail.gmail_list_labels gmail.gmail_get_email gmail.gmail_create_draft gmail.gmail_trash_email gmail.gmail_send_email gmail.gmail_modify_email mysql.mysql_list_tables mysql.mysql_query brave_search.brave_web_search +q025,Upload a file to Google Drive,paraphrase,google_drive.gdrive_update_file,0.0,0.0,1.0,0.2500,0.4307,4,2,523,0.0,1,446,1.0,1,774,1.0,google_drive.gdrive_update_file google_drive.gdrive_share_file google_drive.gdrive_get_file google_drive.gdrive_create_file slack.slack_upload_file google_drive.gdrive_search google_drive.gdrive_list_folder aws_s3.s3_put_object google_calendar.gcal_list_calendars filesystem.edit_file +q026,Create a calendar event,resource,google_calendar.gcal_create_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,427,1.0,1,801,1.0,google_calendar.gcal_create_event google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars github.pull_request_review_write stripe.stripe_create_invoice vercel.vercel_create_project trello.trello_create_board stripe.stripe_create_customer +q027,List Kubernetes pods,verb_product,kubernetes.k8s_list_pods,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,303,1.0,1,718,1.0,kubernetes.k8s_list_pods kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_apply kubernetes.k8s_get_pod kubernetes.k8s_delete kubernetes.k8s_scale_deployment kubernetes.k8s_pod_logs clickup.clickup_list_tasks +q028,Get logs from a k8s pod,verb_product,kubernetes.k8s_pod_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,262,1.0,1,455,1.0,1,830,1.0,kubernetes.k8s_pod_logs kubernetes.k8s_get_pod docker.docker_logs datadog.datadog_list_logs github.actions_get_job_logs aws_s3.s3_get_object elasticsearch.es_get_doc github.get_tag slack.slack_get_channel_history figma.figma_get_file_nodes +q029,Run a docker container,verb_product,docker.docker_run,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,345,1.0,1,597,1.0,docker.docker_run docker.docker_exec docker.docker_stop docker.docker_rm docker.docker_logs docker.docker_images docker.docker_ps docker.docker_build github.get_check_run github.delete_check_run +q030,Show docker container logs,verb_product,docker.docker_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,345,1.0,1,649,1.0,docker.docker_logs docker.docker_stop docker.docker_rm docker.docker_exec docker.docker_run kubernetes.k8s_pod_logs git.git_log docker.docker_images docker.docker_ps docker.docker_build +q031,Navigate the browser to a URL,ambiguous,playwright.browser_navigate,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,346,1.0,1,617,1.0,playwright.browser_navigate puppeteer.puppeteer_navigate playwright.browser_tab_new fetch.fetch twilio.twilio_make_call playwright.browser_evaluate playwright.browser_press_key playwright.browser_snapshot playwright.browser_click playwright.browser_wait_for +q032,Take a screenshot of the page,ambiguous,playwright.browser_take_screenshot,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,320,1.0,1,608,1.0,playwright.browser_take_screenshot puppeteer.puppeteer_screenshot playwright.browser_snapshot filesystem.read_file playwright.browser_evaluate playwright.browser_press_key playwright.browser_click notion.notion_get_page notion.notion_update_page git.git_status +q033,Create a GitLab merge request,verb_product,gitlab.gitlab_merge_merge_request,0.0,1.0,1.0,0.3333,0.5000,3,1,297,1.0,1,475,1.0,1,975,1.0,gitlab.gitlab_merge_merge_request gitlab.gitlab_get_merge_request gitlab.gitlab_create_merge_request github.merge_pull_request gitlab.gitlab_list_merge_requests gitlab.create_pipeline gitlab.create_job gitlab.create_wiki gitlab.create_snippet gitlab.create_label +q034,Search Notion pages,verb_product,notion.notion_search,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,322,1.0,1,671,1.0,notion.notion_search notion.notion_list_users notion.notion_get_page notion.notion_get_block brave_search.brave_web_search notion.notion_create_database notion.notion_update_page notion.notion_query_database notion.notion_get_comments confluence.confluence_search +q035,Create a Notion page,verb_product,notion.notion_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,230,1.0,1,371,1.0,1,822,1.0,notion.notion_create_page notion.notion_get_page notion.notion_update_page notion.notion_get_comments notion.notion_create_database confluence.confluence_create_page notion.create_comment notion.create_users_token notion.create_data_source confluence.confluence_get_page +q036,Query a Notion database,verb_product,notion.notion_query_database,1.0,1.0,1.0,1.0000,1.0000,1,1,261,1.0,1,386,1.0,1,668,1.0,notion.notion_query_database notion.notion_create_database notion.notion_search notion.notion_create_page notion.notion_list_users notion.notion_get_page notion.notion_get_block notion.notion_update_page notion.notion_get_comments prometheus.prom_query +q037,Create a Zendesk ticket,verb_product,zendesk.zendesk_create_ticket,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,368,1.0,1,714,1.0,zendesk.zendesk_create_ticket zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_list_tickets zendesk.zendesk_search jira.jira_create_issue stripe.stripe_create_invoice vercel.vercel_create_project trello.trello_create_board stripe.stripe_create_customer +q038,Create a HubSpot contact,verb_product,hubspot.hubspot_create_contact,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,663,1.0,hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_create_deal hubspot.hubspot_list_contacts hubspot.hubspot_search_contacts stripe.stripe_create_invoice vercel.vercel_create_project trello.trello_create_board stripe.stripe_create_customer stripe.stripe_create_product +q039,Create a Salesforce lead,verb_product,salesforce.salesforce_convert_lead,0.0,1.0,1.0,0.5000,0.6309,2,1,252,1.0,1,364,1.0,1,710,1.0,salesforce.salesforce_convert_lead salesforce.salesforce_create_lead salesforce.salesforce_create_record salesforce.salesforce_list_opportunities salesforce.salesforce_query salesforce.salesforce_delete_record salesforce.salesforce_update_record salesforce.salesforce_get_record stripe.stripe_create_invoice vercel.vercel_create_project +q040,Run a SOQL query,identifier,salesforce.salesforce_query,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,332,1.0,1,662,1.0,salesforce.salesforce_query sqlite.sqlite_query mysql.mysql_query postgres.query docker.docker_run github.get_check_run github.delete_check_run github.update_check_run prometheus.prom_query github.create_check_run +q041,List Shopify orders,verb_product,shopify.shopify_list_orders,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,320,1.0,1,682,1.0,shopify.shopify_list_orders shopify.shopify_list_customers shopify.shopify_list_products shopify.shopify_get_product shopify.shopify_get_order shopify.shopify_update_product shopify.shopify_create_order shopify.shopify_create_product clickup.clickup_list_tasks trello.trello_list_cards +q042,Create a PagerDuty incident,verb_product,pagerduty.pagerduty_create_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,403,1.0,1,696,1.0,pagerduty.pagerduty_create_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls stripe.stripe_create_invoice vercel.vercel_create_project trello.trello_create_board stripe.stripe_create_customer +q043,Who is on call in PagerDuty,paraphrase,pagerduty.pagerduty_list_oncalls,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,373,1.0,1,724,1.0,pagerduty.pagerduty_list_oncalls twilio.twilio_make_call pagerduty.pagerduty_get_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_create_incident slack.slack_create_channel filesystem.list_allowed_directories memory.search_nodes +q044,Query Datadog metrics,verb_product,datadog.datadog_query_metrics,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,458,1.0,1,774,1.0,datadog.datadog_query_metrics datadog.datadog_list_monitors datadog.datadog_create_monitor datadog.datadog_list_logs datadog.datadog_get_monitor prometheus.prom_list_metrics prometheus.prom_query prometheus.prom_query_range postgres.explain_query salesforce.salesforce_query +q045,Search Datadog logs,verb_product,datadog.datadog_list_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,458,1.0,1,855,1.0,datadog.datadog_list_logs datadog.datadog_list_monitors datadog.datadog_get_monitor datadog.datadog_create_monitor datadog.datadog_query_metrics docker.docker_logs kubernetes.k8s_pod_logs github.actions_get_job_logs brave_search.brave_web_search confluence.confluence_search +q046,List S3 buckets,verb_product,aws_s3.s3_list_buckets,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,258,1.0,1,474,1.0,aws_s3.s3_list_buckets clickup.clickup_list_tasks trello.trello_list_cards linear.linear_list_teams jira.jira_list_projects kubernetes.k8s_list_deployments kubernetes.k8s_list_services gmail.gmail_list_labels cloudflare.cf_list_zones vercel.vercel_list_projects +q047,Upload an object to S3,paraphrase,aws_s3.s3_put_object,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,437,1.0,1,850,1.0,aws_s3.s3_put_object aws_s3.s3_delete_object aws_s3.s3_get_object slack.slack_upload_file github.assign_copilot_to_issue twilio.twilio_send_sms gmail.gmail_send_email slack.slack_add_reaction slack.slack_reply_to_thread puppeteer.puppeteer_click +q048,Purge Cloudflare cache,verb_product,cloudflare.cf_purge_cache,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,340,1.0,1,746,1.0,cloudflare.cf_purge_cache cloudflare.cf_list_zones cloudflare.cf_list_dns cloudflare.cf_create_dns +q049,Create a Cloudflare DNS record,verb_product,cloudflare.cf_create_dns,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,356,1.0,1,708,1.0,cloudflare.cf_create_dns cloudflare.cf_list_dns salesforce.salesforce_create_record cloudflare.cf_list_zones cloudflare.cf_purge_cache salesforce.salesforce_delete_record salesforce.salesforce_update_record salesforce.salesforce_get_record git.git_commit stripe.stripe_create_invoice +q050,List Vercel deployments,verb_product,vercel.vercel_list_deployments,1.0,1.0,1.0,1.0000,1.0000,1,1,173,1.0,1,291,1.0,1,529,1.0,vercel.vercel_list_deployments vercel.vercel_list_projects kubernetes.k8s_list_deployments vercel.vercel_get_deployment vercel.vercel_create_project clickup.clickup_list_tasks trello.trello_list_cards aws_s3.s3_list_buckets linear.linear_list_teams jira.jira_list_projects +q051,Insert a row into Supabase,verb_product,supabase.supabase_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,203,1.0,1,347,1.0,1,665,1.0,supabase.supabase_insert mongodb.mongo_insert supabase.supabase_list_tables supabase.supabase_select supabase.supabase_invoke_fn supabase.supabase_update playwright.browser_type +q052,Export images from Figma,paraphrase,figma.figma_get_images,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,328,1.0,1,637,1.0,figma.figma_get_images figma.figma_get_file_nodes docker.docker_images figma.figma_get_file figma.figma_get_comments figma.figma_post_comment memory.delete_observations aws_s3.s3_get_object memory.delete_relations docker.docker_logs +q053,Send an SMS with Twilio,verb_product,twilio.twilio_send_sms,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,396,1.0,1,725,1.0,twilio.twilio_send_sms twilio.twilio_list_messages gmail.gmail_send_email twilio.twilio_make_call puppeteer.puppeteer_click puppeteer.puppeteer_fill discord.discord_send_message filesystem.list_directory_with_sizes puppeteer.puppeteer_navigate puppeteer.puppeteer_screenshot +q054,Create an Okta user,verb_product,okta.okta_create_user,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,329,1.0,1,622,1.0,okta.okta_create_user okta.okta_get_user okta.okta_deactivate_user okta.okta_list_users linear.create_user slack.slack_get_user_profile asana.asana_create_project linear.get_user linear.delete_user linear.update_user +q055,Instant query Prometheus,verb_product,prometheus.prom_query,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,377,1.0,1,680,1.0,prometheus.prom_query prometheus.prom_query_range prometheus.prom_list_metrics datadog.datadog_query_metrics postgres.explain_query salesforce.salesforce_query sqlite.sqlite_query mysql.mysql_query postgres.query notion.notion_query_database +q056,Search Grafana dashboards,verb_product,grafana.grafana_search_dashboards,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,360,1.0,1,660,1.0,grafana.grafana_search_dashboards grafana.grafana_get_dashboard grafana.grafana_create_annotation brave_search.brave_web_search confluence.confluence_search github.search_users github.search_orgs brave_search.brave_local_search github.search_commits slack.slack_search_messages +q057,Trigger a CircleCI pipeline,verb_product,circleci.circleci_trigger_pipeline,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,373,1.0,1,755,1.0,circleci.circleci_trigger_pipeline circleci.circleci_get_pipeline circleci.circleci_list_pipelines github.actions_run_trigger gitlab.get_pipeline gitlab.delete_pipeline gitlab.update_pipeline gitlab.create_pipeline gitlab.gitlab_create_pipeline hubspot.hubspot_create_deal +q058,Run terraform plan,identifier,terraform.tf_plan,1.0,1.0,1.0,1.0000,1.0000,1,1,172,1.0,1,320,1.0,1,658,1.0,terraform.tf_plan terraform.tf_apply terraform.tf_state_list postgres.explain_query docker.docker_run github.get_check_run github.delete_check_run github.update_check_run github.create_check_run salesforce.salesforce_query +q059,Create an Asana task,verb_product,asana.asana_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,429,1.0,1,825,1.0,asana.asana_create_task asana.asana_get_task asana.asana_update_task asana.asana_create_project clickup.clickup_create_task asana.asana_list_tasks clickup.clickup_get_task clickup.clickup_update_task okta.okta_create_user github.create_issue +q060,Create a Trello card,verb_product,trello.trello_create_card,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,364,1.0,1,679,1.0,trello.trello_create_card trello.trello_add_comment trello.trello_create_board trello.trello_list_cards stripe.stripe_create_invoice vercel.vercel_create_project stripe.stripe_create_customer stripe.stripe_create_product salesforce.salesforce_create_record okta.okta_create_user +q061,Create a ClickUp task,verb_product,clickup.clickup_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,414,1.0,1,719,1.0,clickup.clickup_create_task clickup.clickup_get_task clickup.clickup_update_task asana.asana_create_task asana.asana_get_task asana.asana_update_task clickup.clickup_list_tasks stripe.stripe_create_invoice vercel.vercel_create_project trello.trello_create_board +q062,Create a Confluence page,verb_product,confluence.confluence_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,401,1.0,1,699,1.0,confluence.confluence_create_page confluence.confluence_get_page confluence.confluence_update_page confluence.confluence_search notion.notion_create_page notion.notion_get_page notion.notion_update_page playwright.browser_take_screenshot playwright.browser_evaluate playwright.browser_press_key +q063,Send a Discord message,verb_product,discord.discord_send_message,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,322,1.0,1,682,1.0,discord.discord_send_message discord.discord_add_reaction discord.discord_get_channel discord.discord_list_channels discord.discord_list_members twilio.twilio_send_sms gmail.gmail_send_email slack.slack_post_message gmail.gmail_get_email gmail.gmail_trash_email +q064,Search Elasticsearch,verb_product,elasticsearch.es_search,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,382,1.0,1,672,1.0,elasticsearch.es_search elasticsearch.es_get_doc elasticsearch.es_index_doc brave_search.brave_web_search confluence.confluence_search github.search_users github.search_orgs grafana.grafana_search_dashboards brave_search.brave_local_search github.search_commits +q065,Get a Redis key,verb_product,redis.redis_get,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,281,1.0,1,589,1.0,redis.redis_get redis.redis_set redis.redis_delete redis.redis_keys jira.jira_get_issue aws_s3.s3_get_object figma.figma_get_file playwright.browser_press_key figma.figma_get_comments figma.figma_get_file_nodes +q066,Find MongoDB documents,verb_product,mongodb.mongo_find,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,381,1.0,1,738,1.0,mongodb.mongo_find mongodb.mongo_insert mongodb.mongo_delete mongodb.mongo_update mongodb.mongo_list_collections +q067,List SQLite tables,verb_product,sqlite.sqlite_list_tables,1.0,1.0,1.0,1.0000,1.0000,1,1,142,1.0,1,240,1.0,1,496,1.0,sqlite.sqlite_list_tables mysql.mysql_list_tables supabase.supabase_list_tables postgres.list_tables sqlite.sqlite_describe_table sqlite.sqlite_query clickup.clickup_list_tasks trello.trello_list_cards aws_s3.s3_list_buckets linear.linear_list_teams +q068,Describe a postgres table,verb_product,postgres.describe_table,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,349,1.0,1,597,1.0,postgres.describe_table sqlite.sqlite_describe_table supabase.supabase_select supabase.supabase_insert supabase.supabase_update postgres.explain_query postgres.list_schemas postgres.query postgres.list_tables +q069,github_create_issue,identifier,github.create_issue,1.0,1.0,1.0,1.0000,0.9502,1,1,378,1.0,1,568,1.0,1,1091,1.0,github.create_issue bitbucket.bitbucket_create_issue github.issue_write gitlab.gitlab_create_issue github.issue_read github.add_issue_comment linear.linear_create_issue jira.jira_create_issue linear.linear_create_comment github.assign_copilot_to_issue +q070,createIssue,identifier,,0.0,0.0,0.0,0.0000,0.0000,29,10,2752,0.0,6,2680,0.0,3,2626,0.0, +q071,list-directory,identifier,filesystem.create_directory,0.0,1.0,1.0,0.5000,0.6309,2,1,186,1.0,1,296,1.0,1,607,1.0,filesystem.create_directory filesystem.list_directory filesystem.list_directory_with_sizes filesystem.get_file_info filesystem.directory_tree github.get_file_contents clickup.clickup_list_tasks trello.trello_list_cards aws_s3.s3_list_buckets linear.linear_list_teams +q072,sequential thinking next thought,paraphrase,sequential_thinking.sequentialthinking,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,382,1.0,1,797,1.0,sequential_thinking.sequentialthinking +q073,What time is it in Tokyo,paraphrase,time.get_current_time,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,378,1.0,1,703,1.0,time.get_current_time time.convert_time playwright.browser_wait_for google_calendar.gcal_list_events slack.slack_create_channel filesystem.list_allowed_directories grafana.grafana_create_annotation jira.jira_add_worklog github.get_milestone github.delete_milestone +q074,Convert time between timezones,verb_product,time.convert_time,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,360,1.0,1,704,1.0,time.convert_time salesforce.salesforce_convert_lead time.get_current_time playwright.browser_wait_for google_calendar.gcal_list_events memory.create_relations git.git_diff grafana.grafana_create_annotation jira.jira_add_worklog +q075,Star a GitHub repo,verb_product,github.star_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,226,1.0,1,394,1.0,1,779,1.0,github.star_repository github.delete_repository github.unstar_repository github.fork_repository github.search_issues github.actions_list github.get_label github.list_branches github.list_releases github.get_dependabot_alert +q076,Fork a repository,resource,github.fork_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,235,1.0,1,386,1.0,1,792,1.0,github.fork_repository github.delete_repository github.star_repository github.unstar_repository github.create_repository github.list_repository_collaborators github.get_repository_tree github.list_branches github.list_releases github.list_dependabot_alerts +q077,Get the latest GitHub release,verb_product,github.get_latest_release,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,361,1.0,1,712,1.0,github.get_latest_release gitlab.get_release github.get_release_by_tag gitlab.delete_release gitlab.update_release github.get_teams gitlab.create_release github.get_me github.get_repository_tree github.get_milestone +q078,Create a gist,resource,github.create_gist,1.0,1.0,1.0,1.0000,1.0000,1,1,229,1.0,1,335,1.0,1,649,1.0,github.create_gist github.get_gist github.update_gist stripe.stripe_create_invoice vercel.vercel_create_project trello.trello_create_board stripe.stripe_create_customer stripe.stripe_create_product salesforce.salesforce_create_record okta.okta_create_user +q079,Search GitHub code,verb_product,github.search_code,1.0,1.0,1.0,1.0000,1.0000,1,1,283,1.0,1,385,1.0,1,757,1.0,github.search_code github.list_code_scanning_alerts github.get_code_scanning_alert github.search_users github.search_orgs github.search_commits github.search_issues github.search_pull_requests brave_search.brave_web_search github.search_repositories +q080,List GitHub notifications,verb_product,github.list_notifications,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,354,1.0,1,726,1.0,github.list_notifications github.mark_all_notifications_read github.list_milestones github.list_hooks github.list_environments github.list_packages github.list_artifacts github.list_runners github.list_check_runs github.actions_list +q081,Assign Copilot to an issue,verb_product,github.assign_copilot_to_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,402,1.0,1,833,1.0,github.assign_copilot_to_issue linear.linear_assign_issue jira.jira_assign_issue jira.jira_update_issue jira.jira_transition_issue github.add_issue_comment linear.linear_create_comment jira.jira_add_comment github.request_copilot_review jira.jira_add_worklog +q082,Get secret scanning alerts,resource,github.list_secret_scanning_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,295,1.0,1,470,1.0,1,765,1.0,github.list_secret_scanning_alerts github.get_secret_scanning_alert github.list_code_scanning_alerts github.get_code_scanning_alert github.list_dependabot_alerts redis.redis_get kubernetes.k8s_get_pod shopify.shopify_get_product shopify.shopify_get_order vercel.vercel_get_deployment +q083,Create or update a file on GitHub,verb_product,github.create_or_update_file,1.0,1.0,1.0,1.0000,0.7602,1,1,306,1.0,1,481,1.0,1,889,1.0,github.create_or_update_file filesystem.write_file slack.update_file google_drive.gdrive_update_file github.projects_write figma.figma_get_comments github.label_write figma.figma_post_comment google_drive.gdrive_get_file github.get_file_contents +q084,List collaborators on a repo,resource,github.list_repository_collaborators,1.0,1.0,1.0,1.0000,1.0000,1,1,278,1.0,1,433,1.0,1,932,1.0,github.list_repository_collaborators github.list_commits trello.trello_list_cards pagerduty.pagerduty_list_oncalls github.get_discussion_comments github.update_issue_comment github.request_copilot_review github.actions_list bitbucket.bitbucket_list_prs github.create_pull_request +q085,Reply in a Slack thread,verb_product,slack.slack_reply_to_thread,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,350,1.0,1,632,1.0,slack.slack_reply_to_thread slack.slack_get_thread_replies github.get_notification_details slack.get_reminder slack.delete_reminder slack.get_canvase slack.delete_canvase slack.get_bookmark slack.delete_bookmark slack.get_usergroup +q086,Add an emoji reaction on Slack,verb_product,slack.slack_add_reaction,1.0,1.0,1.0,1.0000,1.0000,1,1,271,1.0,1,428,1.0,1,811,1.0,slack.slack_add_reaction discord.discord_add_reaction trello.trello_add_comment gmail.gmail_modify_email playwright.browser_click jira.jira_add_comment memory.add_observations git.git_add jira.jira_add_worklog github.assign_copilot_to_issue +q087,Search Slack messages,verb_product,slack.slack_search_messages,1.0,1.0,1.0,1.0000,1.0000,1,1,232,1.0,1,373,1.0,1,664,1.0,slack.slack_search_messages gmail.gmail_search_emails slack.slack_get_channel_history twilio.twilio_list_messages brave_search.brave_web_search confluence.confluence_search github.search_users github.search_orgs grafana.grafana_search_dashboards brave_search.brave_local_search +q088,Create a Slack channel,verb_product,slack.slack_create_channel,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,342,1.0,1,751,1.0,slack.slack_create_channel slack.slack_join_channel slack.slack_leave_channel slack.slack_get_channel_history slack.slack_set_channel_topic slack.slack_post_message slack.slack_upload_file slack.slack_reply_to_thread slack.slack_get_thread_replies slack.slack_add_reaction +q089,Transition a Jira issue,verb_product,jira.jira_transition_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,342,1.0,1,778,1.0,jira.jira_transition_issue jira.jira_get_issue jira.jira_assign_issue jira.jira_update_issue jira.jira_add_comment jira.jira_add_worklog jira.jira_create_issue sentry.sentry_get_issue gitlab.gitlab_get_issue sentry.sentry_update_issue +q090,Search Jira with JQL,identifier,jira.jira_search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,241,1.0,1,345,1.0,1,619,1.0,jira.jira_search_issues filesystem.list_directory_with_sizes brave_search.brave_web_search confluence.confluence_search github.search_users github.search_orgs grafana.grafana_search_dashboards jira.jira_list_projects brave_search.brave_local_search github.search_commits +q091,Add a worklog to Jira,verb_product,jira.jira_add_worklog,1.0,1.0,1.0,1.0000,1.0000,1,1,236,1.0,1,389,1.0,1,874,1.0,jira.jira_add_worklog jira.jira_add_comment memory.add_observations git.git_add discord.discord_add_reaction github.add_issue_comment slack.slack_add_reaction github.add_comment_to_pending_review jira.jira_transition_issue linear.linear_create_comment +q092,List Linear teams,verb_product,linear.linear_list_teams,1.0,1.0,1.0,1.0000,1.0000,1,1,137,1.0,1,295,1.0,1,646,1.0,linear.linear_list_teams github.get_teams linear.linear_list_projects linear.linear_list_issues linear.list_cycles linear.list_labels linear.list_users linear.list_comments linear.list_attachments linear.linear_search_issues +q093,Comment on a Linear issue,verb_product,linear.linear_create_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,360,1.0,1,777,1.0,linear.linear_create_comment github.update_issue_comment linear.get_comment linear.delete_comment linear.update_comment github.add_issue_comment jira.jira_add_comment linear.linear_update_issue linear.linear_get_issue linear.linear_assign_issue +q094,Move a local file,paraphrase,filesystem.move_file,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,358,1.0,1,698,1.0,filesystem.move_file filesystem.read_file filesystem.read_text_file filesystem.read_media_file brave_search.brave_local_search gmail.gmail_trash_email figma.figma_get_file google_drive.gdrive_update_file google_drive.gdrive_share_file slack.get_file +q095,Recursively search files by pattern,paraphrase,filesystem.search_files,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,338,1.0,1,721,1.0,filesystem.search_files linear.linear_search_issues google_drive.gdrive_search redis.redis_keys github.search_repositories slack.list_files brave_search.brave_web_search github.push_files confluence.confluence_search github.search_users +q096,Edit a file in place with old and new string,paraphrase,filesystem.edit_file,1.0,1.0,1.0,1.0000,1.0000,1,1,247,1.0,1,358,1.0,1,673,1.0,filesystem.edit_file filesystem.list_directory_with_sizes filesystem.write_file filesystem.move_file playwright.browser_tab_new slack.get_file slack.delete_file twilio.twilio_make_call slack.update_file figma.figma_get_file +q097,Read the whole knowledge graph,paraphrase,memory.read_graph,1.0,1.0,1.0,1.0000,1.0000,1,1,152,1.0,1,259,1.0,1,561,1.0,memory.read_graph memory.delete_relations memory.create_entities memory.create_relations memory.add_observations memory.open_nodes memory.delete_entities memory.search_nodes filesystem.read_file filesystem.read_text_file +q098,Search memory nodes,verb_product,memory.search_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,202,1.0,1,345,1.0,1,635,1.0,memory.search_nodes memory.open_nodes figma.figma_get_file_nodes brave_search.brave_web_search confluence.confluence_search github.search_users github.search_orgs grafana.grafana_search_dashboards brave_search.brave_local_search github.search_commits +q099,Show unstaged git diff,verb_product,git.git_diff_unstaged,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,313,1.0,1,602,1.0,git.git_diff_unstaged git.git_diff git.git_diff_staged git.git_show git.git_log git.git_status git.git_branch git.git_checkout git.git_commit git.git_add +q100,Stage files with git add,identifier,git.git_add,1.0,1.0,1.0,1.0000,1.0000,1,1,197,1.0,1,417,1.0,1,872,1.0,git.git_add filesystem.list_directory_with_sizes salesforce.salesforce_list_opportunities github.push_files github.add_issue_comment github.create_gist slack.list_files github.update_gist jira.jira_add_comment github.add_comment_to_pending_review +q101,Create a Shopify product,verb_product,shopify.shopify_create_product,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,377,1.0,1,702,1.0,shopify.shopify_create_product shopify.shopify_get_product shopify.shopify_update_product shopify.shopify_create_order stripe.stripe_create_product shopify.shopify_get_order shopify.shopify_list_customers shopify.shopify_list_products shopify.shopify_list_orders stripe.stripe_create_price +q102,Cancel a Stripe subscription,verb_product,stripe.stripe_cancel_subscription,1.0,1.0,1.0,1.0000,1.0000,1,1,167,1.0,1,285,1.0,1,610,1.0,stripe.stripe_cancel_subscription stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_list_customers stripe.stripe_create_product stripe.stripe_list_subscriptions stripe.stripe_get_payment_intent stripe.stripe_create_refund +q103,Create a payment intent,resource,stripe.stripe_create_payment_intent,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,340,1.0,1,654,1.0,stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent stripe.stripe_create_refund stripe.stripe_create_invoice vercel.vercel_create_project trello.trello_create_board stripe.stripe_create_customer stripe.stripe_create_product salesforce.salesforce_create_record okta.okta_create_user +q104,Acknowledge a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,405,1.0,1,717,1.0,pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls +q105,Create a Datadog monitor,verb_product,datadog.datadog_create_monitor,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,459,1.0,1,758,1.0,datadog.datadog_create_monitor datadog.datadog_get_monitor datadog.datadog_list_monitors datadog.datadog_list_logs datadog.datadog_query_metrics stripe.stripe_create_invoice vercel.vercel_create_project trello.trello_create_board stripe.stripe_create_customer stripe.stripe_create_product +q106,Resolve a Sentry issue,paraphrase,sentry.sentry_get_issue,0.0,1.0,1.0,0.5000,0.6309,2,1,251,1.0,1,368,1.0,1,721,1.0,sentry.sentry_get_issue sentry.sentry_update_issue sentry.sentry_list_events pagerduty.pagerduty_resolve_incident sentry.sentry_list_projects sentry.sentry_list_issues jira.jira_get_issue jira.jira_assign_issue jira.jira_update_issue gitlab.gitlab_get_issue +q107,Scale a Kubernetes deployment,verb_product,kubernetes.k8s_scale_deployment,1.0,1.0,1.0,1.0000,1.0000,1,1,245,1.0,1,353,1.0,1,726,1.0,kubernetes.k8s_scale_deployment vercel.vercel_get_deployment kubernetes.k8s_apply kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_get_pod kubernetes.k8s_delete kubernetes.k8s_list_pods kubernetes.k8s_pod_logs +q108,Apply a Kubernetes manifest,verb_product,kubernetes.k8s_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,308,1.0,1,727,1.0,kubernetes.k8s_apply terraform.tf_apply kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_get_pod kubernetes.k8s_delete kubernetes.k8s_list_pods kubernetes.k8s_scale_deployment kubernetes.k8s_pod_logs +q109,Stop a docker container,verb_product,docker.docker_stop,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,345,1.0,1,641,1.0,docker.docker_stop docker.docker_rm docker.docker_logs docker.docker_exec docker.docker_run docker.docker_images docker.docker_ps docker.docker_build kubernetes.k8s_pod_logs +q110,Build a docker image,verb_product,docker.docker_build,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,303,1.0,1,601,1.0,docker.docker_build docker.docker_run docker.docker_images docker.docker_stop docker.docker_ps docker.docker_rm docker.docker_logs docker.docker_exec filesystem.read_media_file +q111,Click an element in Playwright,verb_product,playwright.browser_click,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,329,1.0,1,612,1.0,playwright.browser_click puppeteer.puppeteer_click playwright.browser_type playwright.browser_take_screenshot playwright.browser_snapshot clickup.clickup_list_tasks clickup.clickup_get_task playwright.browser_tab_list playwright.browser_tab_close clickup.clickup_create_task +q112,Type text into a browser input,paraphrase,playwright.browser_type,1.0,1.0,1.0,1.0000,0.9502,1,1,235,1.0,1,320,1.0,1,648,1.0,playwright.browser_type playwright.browser_wait_for puppeteer.puppeteer_fill playwright.browser_tab_list playwright.browser_tab_close playwright.browser_navigate playwright.browser_tab_new filesystem.read_text_file supabase.supabase_insert mongodb.mongo_insert +q113,List GitLab projects,verb_product,gitlab.gitlab_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,143,1.0,1,240,1.0,1,623,1.0,gitlab.gitlab_list_projects jira.jira_list_projects vercel.vercel_list_projects linear.linear_list_projects sentry.sentry_list_projects github.projects_list gitlab.list_pipelines gitlab.list_jobs gitlab.list_wikis gitlab.list_snippets +q114,Create a Bitbucket pull request,verb_product,bitbucket.bitbucket_create_pr,1.0,1.0,1.0,1.0000,1.0000,1,1,313,1.0,1,546,1.0,1,1139,1.0,bitbucket.bitbucket_create_pr bitbucket.bitbucket_list_prs bitbucket.bitbucket_create_issue github.pull_request_review_write github.pull_request_read github.merge_pull_request github.request_copilot_review github.create_pull_request github.update_pull_request_branch github.update_pull_request +q115,Create a Gmail draft,verb_product,gmail.gmail_create_draft,1.0,1.0,1.0,1.0000,1.0000,1,1,205,1.0,1,315,1.0,1,758,1.0,gmail.gmail_create_draft gmail.gmail_list_labels gmail.gmail_search_emails gmail.gmail_get_email gmail.gmail_trash_email gmail.gmail_send_email gmail.gmail_modify_email github.create_pull_request stripe.stripe_create_invoice vercel.vercel_create_project +q116,Share a Google Drive file,verb_product,google_drive.gdrive_share_file,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,417,1.0,1,725,1.0,google_drive.gdrive_share_file google_drive.gdrive_update_file google_drive.gdrive_get_file google_drive.gdrive_create_file google_drive.gdrive_search google_drive.gdrive_list_folder google_calendar.gcal_list_calendars figma.figma_get_file google_calendar.gcal_delete_event google_calendar.gcal_update_event +q117,List Google Calendar events,verb_product,google_calendar.gcal_list_events,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,427,1.0,1,753,1.0,google_calendar.gcal_list_events google_calendar.gcal_list_calendars google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event kubernetes.k8s_get_events sentry.sentry_list_events google_drive.gdrive_list_folder google_drive.gdrive_search google_drive.gdrive_update_file +q118,Delete a calendar event,resource,google_calendar.gcal_delete_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,427,1.0,1,735,1.0,google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars redis.redis_delete kubernetes.k8s_delete aws_s3.s3_delete_object github.delete_milestone github.delete_hook +q119,List Vercel projects,verb_product,vercel.vercel_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,151,1.0,1,248,1.0,1,550,1.0,vercel.vercel_list_projects vercel.vercel_list_deployments jira.jira_list_projects linear.linear_list_projects sentry.sentry_list_projects vercel.vercel_get_deployment vercel.vercel_create_project gitlab.gitlab_list_projects github.projects_list github.projects_get +q120,Select rows from Supabase,verb_product,supabase.supabase_select,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,354,1.0,1,660,1.0,supabase.supabase_select supabase.supabase_insert supabase.supabase_update supabase.supabase_list_tables supabase.supabase_invoke_fn memory.delete_observations aws_s3.s3_get_object memory.delete_relations docker.docker_logs elasticsearch.es_get_doc +q121,Post a Figma comment,verb_product,figma.figma_post_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,362,1.0,1,646,1.0,figma.figma_post_comment figma.figma_get_file figma.figma_get_comments slack.slack_post_message figma.figma_get_images figma.figma_get_file_nodes linear.get_comment linear.delete_comment notion.get_comment notion.delete_comment +q122,List Okta users,verb_product,okta.okta_list_users,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,271,1.0,1,606,1.0,okta.okta_list_users okta.okta_create_user okta.okta_get_user okta.okta_deactivate_user notion.notion_list_users slack.slack_list_users linear.list_users notion.list_users_tokens slack.slack_get_users github.search_users +q123,Range query Prometheus,verb_product,prometheus.prom_query_range,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,377,1.0,1,680,1.0,prometheus.prom_query_range prometheus.prom_query prometheus.prom_list_metrics datadog.datadog_query_metrics postgres.explain_query salesforce.salesforce_query sqlite.sqlite_query mysql.mysql_query postgres.query notion.notion_query_database +q124,Create a Grafana annotation,verb_product,grafana.grafana_create_annotation,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,324,1.0,1,638,1.0,grafana.grafana_create_annotation grafana.grafana_search_dashboards grafana.grafana_get_dashboard stripe.stripe_create_invoice vercel.vercel_create_project trello.trello_create_board stripe.stripe_create_customer stripe.stripe_create_product salesforce.salesforce_create_record okta.okta_create_user +q125,Apply terraform,identifier,terraform.tf_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,292,1.0,1,698,1.0,terraform.tf_apply kubernetes.k8s_apply terraform.tf_plan terraform.tf_state_list +q126,List Asana tasks,verb_product,asana.asana_list_tasks,1.0,1.0,1.0,1.0000,1.0000,1,1,192,1.0,1,368,1.0,1,642,1.0,asana.asana_list_tasks clickup.clickup_list_tasks asana.asana_get_task asana.asana_create_project asana.asana_create_task asana.asana_update_task trello.trello_list_cards aws_s3.s3_list_buckets linear.linear_list_teams jira.jira_list_projects +q127,Comment on a Trello card,verb_product,trello.trello_add_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,422,1.0,1,703,1.0,trello.trello_add_comment trello.trello_create_card trello.trello_list_cards trello.trello_create_board github.update_issue_comment figma.figma_post_comment linear.get_comment linear.delete_comment notion.get_comment notion.delete_comment +q128,Update a ClickUp task,verb_product,clickup.clickup_update_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,398,1.0,1,789,1.0,clickup.clickup_update_task clickup.clickup_get_task clickup.clickup_create_task asana.asana_update_task asana.asana_get_task clickup.clickup_list_tasks asana.asana_create_task mongodb.mongo_update shopify.shopify_update_product github.update_milestone +q129,Search Confluence,verb_product,confluence.confluence_search,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,401,1.0,1,691,1.0,confluence.confluence_search confluence.confluence_get_page confluence.confluence_update_page confluence.confluence_create_page brave_search.brave_web_search github.search_users github.search_orgs grafana.grafana_search_dashboards brave_search.brave_local_search github.search_commits +q130,Create a HubSpot deal,verb_product,hubspot.hubspot_create_deal,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,663,1.0,hubspot.hubspot_create_deal hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_list_contacts hubspot.hubspot_search_contacts stripe.stripe_create_invoice vercel.vercel_create_project trello.trello_create_board stripe.stripe_create_customer stripe.stripe_create_product +q131,List Zendesk tickets,verb_product,zendesk.zendesk_list_tickets,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,367,1.0,1,605,1.0,zendesk.zendesk_list_tickets zendesk.zendesk_search zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_create_ticket clickup.clickup_list_tasks trello.trello_list_cards aws_s3.s3_list_buckets linear.linear_list_teams jira.jira_list_projects +q132,List Discord channels,verb_product,discord.discord_list_channels,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,300,1.0,1,592,1.0,discord.discord_list_channels discord.discord_list_members slack.slack_list_channels discord.discord_get_channel discord.discord_send_message discord.discord_add_reaction clickup.clickup_list_tasks trello.trello_list_cards aws_s3.s3_list_buckets linear.linear_list_teams +q133,Set a Redis key,verb_product,redis.redis_set,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,302,1.0,1,600,1.0,redis.redis_set redis.redis_get redis.redis_delete redis.redis_keys slack.slack_set_channel_topic playwright.browser_press_key jira.jira_get_issue aws_s3.s3_delete_object aws_s3.s3_get_object figma.figma_get_file +q134,Insert MongoDB documents,verb_product,mongodb.mongo_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,399,1.0,1,698,1.0,mongodb.mongo_insert mongodb.mongo_delete mongodb.mongo_find mongodb.mongo_update supabase.supabase_insert mongodb.mongo_list_collections +q135,Run a MySQL query,verb_product,mysql.mysql_query,1.0,1.0,1.0,1.0000,1.0000,1,1,181,1.0,1,286,1.0,1,608,1.0,mysql.mysql_query mysql.mysql_list_tables salesforce.salesforce_query sqlite.sqlite_query postgres.query docker.docker_run github.get_check_run github.delete_check_run github.update_check_run prometheus.prom_query +q136,Index a document in Elasticsearch,verb_product,elasticsearch.es_index_doc,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,346,1.0,1,605,1.0,elasticsearch.es_index_doc elasticsearch.es_get_doc elasticsearch.es_search playwright.browser_tab_close github.get_milestone github.delete_milestone github.get_hook github.delete_hook github.get_environment github.delete_environment +q137,Get an S3 object,verb_product,aws_s3.s3_get_object,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,355,1.0,1,668,1.0,aws_s3.s3_get_object aws_s3.s3_delete_object aws_s3.s3_put_object okta.okta_get_user asana.asana_get_task puppeteer.puppeteer_click elasticsearch.es_search okta.okta_create_user puppeteer.puppeteer_fill aws_s3.s3_list_objects +q138,List Cloudflare zones,verb_product,cloudflare.cf_list_zones,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,338,1.0,1,579,1.0,cloudflare.cf_list_zones cloudflare.cf_list_dns cloudflare.cf_purge_cache cloudflare.cf_create_dns clickup.clickup_list_tasks trello.trello_list_cards aws_s3.s3_list_buckets linear.linear_list_teams jira.jira_list_projects kubernetes.k8s_list_deployments +q139,Create a Stripe invoice,verb_product,stripe.stripe_create_invoice,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,353,1.0,1,669,1.0,stripe.stripe_create_invoice stripe.stripe_create_customer stripe.stripe_create_product stripe.stripe_create_refund stripe.stripe_create_price stripe.stripe_create_payment_intent stripe.stripe_list_invoices stripe.stripe_get_customer stripe.stripe_list_customers stripe.stripe_list_subscriptions +q140,List GitHub Actions workflows,verb_product,github.actions_list,1.0,1.0,1.0,1.0000,1.0000,1,1,293,1.0,1,468,1.0,1,845,1.0,github.actions_list github.actions_get github.actions_run_trigger github.actions_get_job_logs github.list_milestones github.list_hooks github.list_environments github.list_packages github.list_artifacts github.list_runners +q141,Get GitHub Actions job logs,verb_product,github.actions_get_job_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,256,1.0,1,421,1.0,1,822,1.0,github.actions_get_job_logs github.actions_get gitlab.get_job github.actions_list kubernetes.k8s_pod_logs github.actions_run_trigger gitlab.delete_job gitlab.update_job docker.docker_logs gitlab.create_job +q142,Create a GitHub repository,verb_product,github.create_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,268,1.0,1,406,1.0,1,949,1.0,github.create_repository github.create_branch github.delete_repository github.star_repository github.unstar_repository github.fork_repository github.create_issue github.list_repository_collaborators github.create_or_update_file github.get_repository_tree +q143,Checkout a git branch,verb_product,git.git_checkout,1.0,1.0,1.0,1.0000,1.0000,1,1,266,1.0,1,429,1.0,1,1001,1.0,git.git_checkout git.git_create_branch github.create_branch git.git_branch gitlab.gitlab_create_merge_request github.update_pull_request_branch github.list_commits github.delete_file github.push_files github.create_or_update_file +q144,Show git log,identifier,git.git_log,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,299,1.0,1,600,1.0,git.git_log git.git_show git.git_diff_unstaged git.git_diff_staged git.git_status git.git_diff git.git_branch git.git_checkout git.git_commit git.git_add +q145,List postgres schemas,verb_product,postgres.list_schemas,1.0,1.0,1.0,1.0000,1.0000,1,1,160,1.0,1,277,1.0,1,515,1.0,postgres.list_schemas postgres.list_tables postgres.explain_query postgres.query postgres.describe_table clickup.clickup_list_tasks trello.trello_list_cards aws_s3.s3_list_buckets linear.linear_list_teams jira.jira_list_projects +q146,Open nodes in the knowledge graph,verb_product,memory.open_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,163,1.0,1,260,1.0,1,531,1.0,memory.open_nodes memory.search_nodes memory.read_graph memory.create_entities memory.delete_relations memory.create_relations memory.add_observations memory.delete_entities figma.figma_get_file_nodes playwright.browser_tab_list +q147,Get file metadata from the filesystem,paraphrase,filesystem.read_file,0.0,1.0,1.0,0.5000,0.6309,2,1,220,1.0,1,334,1.0,1,766,1.0,filesystem.read_file filesystem.get_file_info filesystem.read_text_file filesystem.read_media_file google_drive.gdrive_get_file figma.figma_get_file_nodes gitlab.gitlab_get_file figma.figma_get_images github.get_file_contents filesystem.write_file +q148,Write a new local file,paraphrase,filesystem.write_file,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,384,1.0,1,791,1.0,filesystem.write_file filesystem.read_file filesystem.read_text_file filesystem.edit_file filesystem.read_media_file playwright.browser_tab_new brave_search.brave_local_search github.issue_write figma.figma_get_file google_drive.gdrive_update_file +q149,Search GitHub issues,verb_product,github.search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,233,1.0,1,385,1.0,1,700,1.0,github.search_issues linear.linear_search_issues jira.jira_search_issues github.list_issues github.search_users github.search_orgs github.search_commits github.search_pull_requests gitlab.gitlab_list_issues linear.linear_list_issues +q150,Get the authenticated GitHub user,paraphrase,github.get_teams,0.0,1.0,1.0,0.5000,0.6309,2,1,179,1.0,1,288,1.0,1,569,1.0,github.get_teams github.get_me github.list_notifications okta.okta_get_user slack.slack_get_user_profile linear.get_user github.get_latest_release okta.okta_deactivate_user linear.delete_user okta.okta_create_user diff --git a/experiments/mcp-tool-search/results/bm25_plain.json b/experiments/mcp-tool-search/results/bm25_plain.json new file mode 100644 index 00000000..248467ed --- /dev/null +++ b/experiments/mcp-tool-search/results/bm25_plain.json @@ -0,0 +1,6958 @@ +{ + "summary": { + "n": 150, + "index_s": 0.0009321830000317277, + "rss_mb": 0.046875, + "p50_ms": 0.1412585000366562, + "p95_ms": 0.23318005003716258, + "recall@1": 0.9066666666666666, + "recall@1_lo": 0.86, + "recall@1_hi": 0.9533333333333334, + "recall@3": 0.9733333333333334, + "recall@3_lo": 0.9466666666666667, + "recall@3_hi": 0.9933333333333333, + "recall@5": 0.98, + "recall@5_lo": 0.9533333333333334, + "recall@5_hi": 1.0, + "mrr@10": 0.9401851851851851, + "mrr@10_lo": 0.9066666666666666, + "mrr@10_hi": 0.9707407407407408, + "ndcg@10": 0.9478950871964872, + "ndcg@10_lo": 0.9194780695599083, + "ndcg@10_hi": 0.9739250907995152 + }, + "meta": { + "name": "bm25_plain", + "needs_model": false, + "needs_api": false, + "notes": "Okapi BM25, split on non-alnum only" + }, + "rows": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "bitbucket.bitbucket_create_issue", + "github.issue_write", + "gitlab.gitlab_create_issue", + "github.issue_read", + "github.add_issue_comment", + "linear.linear_create_issue", + "jira.jira_create_issue", + "linear.linear_create_comment", + "github.assign_copilot_to_issue" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 380, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 570, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1093, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "kind": "verb_product", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_create_issue", + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "jira.create_board", + "jira.create_sprint", + "jira.create_filter", + "jira.create_webhook", + "jira.create_attachment", + "jira.create_version" + ], + "hit1": "jira.jira_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 281, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 466, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 951, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "kind": "verb_product", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "slack.slack_post_message", + "twilio.twilio_send_sms", + "slack.slack_add_reaction", + "gmail.gmail_send_email", + "discord.discord_add_reaction", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_search_messages", + "slack.slack_create_channel" + ], + "hit1": "discord.discord_send_message", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 760, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "kind": "resource", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "ranking": [ + "github.list_pull_requests", + "github.search_pull_requests", + "bitbucket.bitbucket_list_prs", + "gitlab.gitlab_list_merge_requests", + "github.create_pull_request", + "playwright.browser_tab_list", + "memory.open_nodes", + "github.pull_request_read", + "github.merge_pull_request", + "github.update_pull_request_branch" + ], + "hit1": "github.list_pull_requests", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 460, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 836, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "ranking": [ + "github.create_pull_request", + "github.request_copilot_review", + "github.pull_request_read", + "github.merge_pull_request", + "github.update_pull_request_branch", + "github.pull_request_review_write", + "github.update_pull_request", + "github.add_comment_to_pending_review", + "github.search_pull_requests", + "bitbucket.bitbucket_create_pr" + ], + "hit1": "github.create_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 364, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 564, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1180, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "kind": "verb_product", + "gold": [ + "postgres.query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "mysql.mysql_query", + "sqlite.sqlite_query", + "postgres.explain_query", + "salesforce.salesforce_query", + "postgres.list_schemas", + "postgres.list_tables", + "postgres.describe_table", + "docker.docker_run", + "github.get_check_run" + ], + "hit1": "postgres.query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 592, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "kind": "paraphrase", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "ranking": [ + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "figma.figma_get_file_nodes", + "gitlab.gitlab_get_file", + "figma.figma_get_images", + "github.delete_file", + "memory.read_graph", + "brave_search.brave_local_search", + "github.get_file_contents" + ], + "hit1": "filesystem.read_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 809, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "kind": "paraphrase", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "ranking": [ + "google_drive.gdrive_list_folder", + "slack.list_files", + "github.push_files", + "filesystem.search_files", + "google_drive.gdrive_search", + "github.create_gist", + "cloudflare.cf_purge_cache", + "github.update_gist", + "filesystem.list_directory", + "filesystem.move_file" + ], + "hit1": "google_drive.gdrive_list_folder", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.1111111111111111, + "ndcg@10": 0.1922943146971571, + "hit_rank": 9, + "k3_calls": 3, + "k3_tokens": 717, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 756, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 734, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "ranking": [ + "memory.read_graph", + "memory.create_entities", + "memory.delete_relations", + "memory.create_relations", + "memory.add_observations", + "memory.open_nodes", + "memory.search_nodes", + "memory.delete_entities", + "postgres.list_schemas", + "time.get_current_time" + ], + "hit1": "memory.read_graph", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.5338931479009518, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 154, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 261, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 507, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "kind": "verb_product", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "ranking": [ + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "jira.jira_search_issues", + "memory.search_nodes", + "github.update_pull_request_branch", + "filesystem.list_directory_with_sizes", + "confluence.confluence_search", + "github.search_users", + "github.search_orgs", + "grafana.grafana_search_dashboards" + ], + "hit1": "brave_search.brave_web_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 410, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "kind": "verb_product", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "ranking": [ + "fetch.fetch", + "filesystem.read_file", + "playwright.browser_navigate", + "puppeteer.puppeteer_navigate", + "git.git_add", + "playwright.browser_snapshot", + "github.get_file_contents", + "git.git_status", + "docker.docker_logs", + "slack.slack_set_channel_topic" + ], + "hit1": "fetch.fetch", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "kind": "identifier", + "gold": [ + "git.git_status" + ], + "alts": [], + "ranking": [ + "git.git_status", + "git.git_show", + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_log", + "git.git_diff", + "sentry.sentry_update_issue", + "git.git_branch", + "git.git_checkout", + "zendesk.zendesk_list_tickets" + ], + "hit1": "git.git_status", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 613, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "kind": "paraphrase", + "gold": [ + "git.git_commit" + ], + "alts": [], + "ranking": [ + "git.git_commit", + "git.git_reset", + "git.git_diff_staged", + "github.get_commit", + "git.git_show", + "git.git_log", + "github.push_files" + ], + "hit1": "git.git_commit", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_comment", + "linear.linear_create_issue", + "linear.linear_update_issue", + "linear.linear_get_issue", + "linear.linear_assign_issue", + "linear.linear_create_project", + "bitbucket.bitbucket_create_issue", + "gitlab.gitlab_create_issue", + "linear.create_cycle", + "linear.create_label" + ], + "hit1": "linear.linear_create_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 917, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "kind": "paraphrase", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_project", + "asana.asana_get_task", + "clickup.clickup_get_task", + "asana.asana_update_task", + "clickup.clickup_update_task", + "asana.asana_create_task", + "clickup.clickup_create_task", + "mysql.mysql_list_tables", + "mysql.mysql_query", + "linear.get_cycle" + ], + "hit1": "linear.linear_create_project", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 146, + "k3_calls": 49, + "k3_tokens": 12414, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 30, + "k5_tokens": 12154, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11794, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "github.update_issue_comment", + "github.add_issue_comment", + "linear.linear_create_comment", + "jira.jira_add_comment", + "github.assign_copilot_to_issue", + "trello.trello_add_comment", + "slack.slack_post_message", + "jira.jira_add_worklog", + "github.issue_read" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 307, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 442, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 876, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "kind": "ambiguous", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "ranking": [ + "github.merge_pull_request", + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_get_merge_request", + "github.update_pull_request_branch", + "github.pull_request_read", + "github.request_copilot_review", + "gitlab.gitlab_create_merge_request", + "github.pull_request_review_write", + "github.update_pull_request", + "github.create_pull_request" + ], + "hit1": "github.merge_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 468, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1121, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "kind": "verb_product", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "ranking": [ + "github.actions_run_trigger", + "github.actions_get", + "github.actions_list", + "circleci.circleci_trigger_pipeline", + "github.actions_get_job_logs", + "github.get_milestone", + "github.delete_milestone", + "github.get_hook", + "github.delete_hook", + "github.get_environment" + ], + "hit1": "github.actions_run_trigger", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "kind": "resource", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "ranking": [ + "github.list_dependabot_alerts", + "github.get_dependabot_alert", + "github.list_secret_scanning_alerts", + "github.list_code_scanning_alerts", + "redis.redis_get", + "kubernetes.k8s_get_pod", + "shopify.shopify_get_product", + "shopify.shopify_get_order", + "vercel.vercel_get_deployment", + "grafana.grafana_get_dashboard" + ], + "hit1": "github.list_dependabot_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 715, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "kind": "verb_product", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_issues", + "sentry.sentry_list_projects", + "sentry.sentry_list_events", + "gitlab.gitlab_list_issues", + "linear.linear_list_issues", + "sentry.sentry_get_issue", + "github.list_issues", + "sentry.sentry_update_issue", + "github.search_issues", + "linear.linear_search_issues" + ], + "hit1": "sentry.sentry_list_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 395, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_customer", + "stripe.stripe_create_invoice", + "stripe.stripe_get_customer", + "stripe.stripe_create_payment_intent", + "stripe.stripe_list_invoices", + "stripe.stripe_list_subscriptions", + "stripe.stripe_create_product", + "stripe.stripe_create_refund", + "stripe.stripe_create_price", + "stripe.stripe_list_customers" + ], + "hit1": "stripe.stripe_create_customer", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "kind": "paraphrase", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_refund", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_list_customers", + "stripe.stripe_create_product", + "stripe.stripe_list_subscriptions" + ], + "hit1": "stripe.stripe_create_refund", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "kind": "verb_product", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "ranking": [ + "gmail.gmail_send_email", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "twilio.twilio_send_sms", + "gmail.gmail_modify_email", + "gmail.gmail_list_labels", + "puppeteer.puppeteer_click", + "puppeteer.puppeteer_fill", + "gmail.gmail_search_emails", + "gmail.gmail_create_draft" + ], + "hit1": "gmail.gmail_send_email", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "kind": "verb_product", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "ranking": [ + "gmail.gmail_search_emails", + "gmail.gmail_list_labels", + "gmail.gmail_get_email", + "gmail.gmail_create_draft", + "gmail.gmail_trash_email", + "gmail.gmail_send_email", + "gmail.gmail_modify_email", + "mysql.mysql_list_tables", + "mysql.mysql_query", + "brave_search.brave_web_search" + ], + "hit1": "gmail.gmail_search_emails", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 691, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "kind": "paraphrase", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_update_file", + "google_drive.gdrive_share_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_create_file", + "slack.slack_upload_file", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "aws_s3.s3_put_object", + "google_calendar.gcal_list_calendars", + "filesystem.edit_file" + ], + "hit1": "google_drive.gdrive_update_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 523, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 446, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_create_event", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "github.pull_request_review_write", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project", + "trello.trello_create_board", + "stripe.stripe_create_customer" + ], + "hit1": "google_calendar.gcal_create_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 801, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_list_pods", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_apply", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_delete", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_pod_logs", + "clickup.clickup_list_tasks" + ], + "hit1": "kubernetes.k8s_list_pods", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_get_pod", + "docker.docker_logs", + "datadog.datadog_list_logs", + "github.actions_get_job_logs", + "aws_s3.s3_get_object", + "elasticsearch.es_get_doc", + "github.get_tag", + "slack.slack_get_channel_history", + "figma.figma_get_file_nodes" + ], + "hit1": "kubernetes.k8s_pod_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 830, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "ranking": [ + "docker.docker_run", + "docker.docker_exec", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_images", + "docker.docker_ps", + "docker.docker_build", + "github.get_check_run", + "github.delete_check_run" + ], + "hit1": "docker.docker_run", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "kind": "verb_product", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "ranking": [ + "docker.docker_logs", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_exec", + "docker.docker_run", + "kubernetes.k8s_pod_logs", + "git.git_log", + "docker.docker_images", + "docker.docker_ps", + "docker.docker_build" + ], + "hit1": "docker.docker_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 649, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "kind": "ambiguous", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "ranking": [ + "playwright.browser_navigate", + "puppeteer.puppeteer_navigate", + "playwright.browser_tab_new", + "fetch.fetch", + "twilio.twilio_make_call", + "playwright.browser_evaluate", + "playwright.browser_press_key", + "playwright.browser_snapshot", + "playwright.browser_click", + "playwright.browser_wait_for" + ], + "hit1": "playwright.browser_navigate", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 346, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 617, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "kind": "ambiguous", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "ranking": [ + "playwright.browser_take_screenshot", + "puppeteer.puppeteer_screenshot", + "playwright.browser_snapshot", + "filesystem.read_file", + "playwright.browser_evaluate", + "playwright.browser_press_key", + "playwright.browser_click", + "notion.notion_get_page", + "notion.notion_update_page", + "git.git_status" + ], + "hit1": "playwright.browser_take_screenshot", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 608, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_get_merge_request", + "gitlab.gitlab_create_merge_request", + "github.merge_pull_request", + "gitlab.gitlab_list_merge_requests", + "gitlab.create_pipeline", + "gitlab.create_job", + "gitlab.create_wiki", + "gitlab.create_snippet", + "gitlab.create_label" + ], + "hit1": "gitlab.gitlab_merge_merge_request", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 975, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "kind": "verb_product", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "ranking": [ + "notion.notion_search", + "notion.notion_list_users", + "notion.notion_get_page", + "notion.notion_get_block", + "brave_search.brave_web_search", + "notion.notion_create_database", + "notion.notion_update_page", + "notion.notion_query_database", + "notion.notion_get_comments", + "confluence.confluence_search" + ], + "hit1": "notion.notion_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 671, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "kind": "verb_product", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "ranking": [ + "notion.notion_create_page", + "notion.notion_get_page", + "notion.notion_update_page", + "notion.notion_get_comments", + "notion.notion_create_database", + "confluence.confluence_create_page", + "notion.create_comment", + "notion.create_users_token", + "notion.create_data_source", + "confluence.confluence_get_page" + ], + "hit1": "notion.notion_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 822, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "kind": "verb_product", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "ranking": [ + "notion.notion_query_database", + "notion.notion_create_database", + "notion.notion_search", + "notion.notion_create_page", + "notion.notion_list_users", + "notion.notion_get_page", + "notion.notion_get_block", + "notion.notion_update_page", + "notion.notion_get_comments", + "prometheus.prom_query" + ], + "hit1": "notion.notion_query_database", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 668, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_search", + "jira.jira_create_issue", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project", + "trello.trello_create_board", + "stripe.stripe_create_customer" + ], + "hit1": "zendesk.zendesk_create_ticket", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_create_deal", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_search_contacts", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project", + "trello.trello_create_board", + "stripe.stripe_create_customer", + "stripe.stripe_create_product" + ], + "hit1": "hubspot.hubspot_create_contact", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "kind": "verb_product", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_convert_lead", + "salesforce.salesforce_create_lead", + "salesforce.salesforce_create_record", + "salesforce.salesforce_list_opportunities", + "salesforce.salesforce_query", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_update_record", + "salesforce.salesforce_get_record", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project" + ], + "hit1": "salesforce.salesforce_convert_lead", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 710, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "kind": "identifier", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "postgres.query", + "docker.docker_run", + "github.get_check_run", + "github.delete_check_run", + "github.update_check_run", + "prometheus.prom_query", + "github.create_check_run" + ], + "hit1": "salesforce.salesforce_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 332, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "kind": "verb_product", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "ranking": [ + "shopify.shopify_list_orders", + "shopify.shopify_list_customers", + "shopify.shopify_list_products", + "shopify.shopify_get_product", + "shopify.shopify_get_order", + "shopify.shopify_update_product", + "shopify.shopify_create_order", + "shopify.shopify_create_product", + "clickup.clickup_list_tasks", + "trello.trello_list_cards" + ], + "hit1": "shopify.shopify_list_orders", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project", + "trello.trello_create_board", + "stripe.stripe_create_customer" + ], + "hit1": "pagerduty.pagerduty_create_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 696, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "kind": "paraphrase", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_list_oncalls", + "twilio.twilio_make_call", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_create_incident", + "slack.slack_create_channel", + "filesystem.list_allowed_directories", + "memory.search_nodes" + ], + "hit1": "pagerduty.pagerduty_list_oncalls", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "kind": "verb_product", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "ranking": [ + "datadog.datadog_query_metrics", + "datadog.datadog_list_monitors", + "datadog.datadog_create_monitor", + "datadog.datadog_list_logs", + "datadog.datadog_get_monitor", + "prometheus.prom_list_metrics", + "prometheus.prom_query", + "prometheus.prom_query_range", + "postgres.explain_query", + "salesforce.salesforce_query" + ], + "hit1": "datadog.datadog_query_metrics", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "kind": "verb_product", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "ranking": [ + "datadog.datadog_list_logs", + "datadog.datadog_list_monitors", + "datadog.datadog_get_monitor", + "datadog.datadog_create_monitor", + "datadog.datadog_query_metrics", + "docker.docker_logs", + "kubernetes.k8s_pod_logs", + "github.actions_get_job_logs", + "brave_search.brave_web_search", + "confluence.confluence_search" + ], + "hit1": "datadog.datadog_list_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 855, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_list_buckets", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "linear.linear_list_teams", + "jira.jira_list_projects", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "gmail.gmail_list_labels", + "cloudflare.cf_list_zones", + "vercel.vercel_list_projects" + ], + "hit1": "aws_s3.s3_list_buckets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 258, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 474, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "kind": "paraphrase", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_put_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_get_object", + "slack.slack_upload_file", + "github.assign_copilot_to_issue", + "twilio.twilio_send_sms", + "gmail.gmail_send_email", + "slack.slack_add_reaction", + "slack.slack_reply_to_thread", + "puppeteer.puppeteer_click" + ], + "hit1": "aws_s3.s3_put_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 850, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "cloudflare.cf_list_dns", + "cloudflare.cf_create_dns" + ], + "hit1": "cloudflare.cf_purge_cache", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "salesforce.salesforce_create_record", + "cloudflare.cf_list_zones", + "cloudflare.cf_purge_cache", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_update_record", + "salesforce.salesforce_get_record", + "git.git_commit", + "stripe.stripe_create_invoice" + ], + "hit1": "cloudflare.cf_create_dns", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 708, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_deployments", + "vercel.vercel_list_projects", + "kubernetes.k8s_list_deployments", + "vercel.vercel_get_deployment", + "vercel.vercel_create_project", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "aws_s3.s3_list_buckets", + "linear.linear_list_teams", + "jira.jira_list_projects" + ], + "hit1": "vercel.vercel_list_deployments", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 173, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 529, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "ranking": [ + "supabase.supabase_insert", + "mongodb.mongo_insert", + "supabase.supabase_list_tables", + "supabase.supabase_select", + "supabase.supabase_invoke_fn", + "supabase.supabase_update", + "playwright.browser_type" + ], + "hit1": "supabase.supabase_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 203, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 347, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "kind": "paraphrase", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "ranking": [ + "figma.figma_get_images", + "figma.figma_get_file_nodes", + "docker.docker_images", + "figma.figma_get_file", + "figma.figma_get_comments", + "figma.figma_post_comment", + "memory.delete_observations", + "aws_s3.s3_get_object", + "memory.delete_relations", + "docker.docker_logs" + ], + "hit1": "figma.figma_get_images", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 637, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "kind": "verb_product", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "ranking": [ + "twilio.twilio_send_sms", + "twilio.twilio_list_messages", + "gmail.gmail_send_email", + "twilio.twilio_make_call", + "puppeteer.puppeteer_click", + "puppeteer.puppeteer_fill", + "discord.discord_send_message", + "filesystem.list_directory_with_sizes", + "puppeteer.puppeteer_navigate", + "puppeteer.puppeteer_screenshot" + ], + "hit1": "twilio.twilio_send_sms", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "kind": "verb_product", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "ranking": [ + "okta.okta_create_user", + "okta.okta_get_user", + "okta.okta_deactivate_user", + "okta.okta_list_users", + "linear.create_user", + "slack.slack_get_user_profile", + "asana.asana_create_project", + "linear.get_user", + "linear.delete_user", + "linear.update_user" + ], + "hit1": "okta.okta_create_user", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 622, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query", + "prometheus.prom_query_range", + "prometheus.prom_list_metrics", + "datadog.datadog_query_metrics", + "postgres.explain_query", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "postgres.query", + "notion.notion_query_database" + ], + "hit1": "prometheus.prom_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "kind": "verb_product", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "ranking": [ + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "grafana.grafana_create_annotation", + "brave_search.brave_web_search", + "confluence.confluence_search", + "github.search_users", + "github.search_orgs", + "brave_search.brave_local_search", + "github.search_commits", + "slack.slack_search_messages" + ], + "hit1": "grafana.grafana_search_dashboards", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "kind": "verb_product", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "ranking": [ + "circleci.circleci_trigger_pipeline", + "circleci.circleci_get_pipeline", + "circleci.circleci_list_pipelines", + "github.actions_run_trigger", + "gitlab.get_pipeline", + "gitlab.delete_pipeline", + "gitlab.update_pipeline", + "gitlab.create_pipeline", + "gitlab.gitlab_create_pipeline", + "hubspot.hubspot_create_deal" + ], + "hit1": "circleci.circleci_trigger_pipeline", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "kind": "identifier", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "ranking": [ + "terraform.tf_plan", + "terraform.tf_apply", + "terraform.tf_state_list", + "postgres.explain_query", + "docker.docker_run", + "github.get_check_run", + "github.delete_check_run", + "github.update_check_run", + "github.create_check_run", + "salesforce.salesforce_query" + ], + "hit1": "terraform.tf_plan", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "kind": "verb_product", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "ranking": [ + "asana.asana_create_task", + "asana.asana_get_task", + "asana.asana_update_task", + "asana.asana_create_project", + "clickup.clickup_create_task", + "asana.asana_list_tasks", + "clickup.clickup_get_task", + "clickup.clickup_update_task", + "okta.okta_create_user", + "github.create_issue" + ], + "hit1": "asana.asana_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "ranking": [ + "trello.trello_create_card", + "trello.trello_add_comment", + "trello.trello_create_board", + "trello.trello_list_cards", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project", + "stripe.stripe_create_customer", + "stripe.stripe_create_product", + "salesforce.salesforce_create_record", + "okta.okta_create_user" + ], + "hit1": "trello.trello_create_card", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 679, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_create_task", + "clickup.clickup_get_task", + "clickup.clickup_update_task", + "asana.asana_create_task", + "asana.asana_get_task", + "asana.asana_update_task", + "clickup.clickup_list_tasks", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project", + "trello.trello_create_board" + ], + "hit1": "clickup.clickup_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 414, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 719, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "kind": "verb_product", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "ranking": [ + "confluence.confluence_create_page", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "confluence.confluence_search", + "notion.notion_create_page", + "notion.notion_get_page", + "notion.notion_update_page", + "playwright.browser_take_screenshot", + "playwright.browser_evaluate", + "playwright.browser_press_key" + ], + "hit1": "confluence.confluence_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "kind": "verb_product", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "discord.discord_add_reaction", + "discord.discord_get_channel", + "discord.discord_list_channels", + "discord.discord_list_members", + "twilio.twilio_send_sms", + "gmail.gmail_send_email", + "slack.slack_post_message", + "gmail.gmail_get_email", + "gmail.gmail_trash_email" + ], + "hit1": "discord.discord_send_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_search", + "elasticsearch.es_get_doc", + "elasticsearch.es_index_doc", + "brave_search.brave_web_search", + "confluence.confluence_search", + "github.search_users", + "github.search_orgs", + "grafana.grafana_search_dashboards", + "brave_search.brave_local_search", + "github.search_commits" + ], + "hit1": "elasticsearch.es_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 672, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "redis.redis_set", + "redis.redis_delete", + "redis.redis_keys", + "jira.jira_get_issue", + "aws_s3.s3_get_object", + "figma.figma_get_file", + "playwright.browser_press_key", + "figma.figma_get_comments", + "figma.figma_get_file_nodes" + ], + "hit1": "redis.redis_get", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 589, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_find", + "mongodb.mongo_insert", + "mongodb.mongo_delete", + "mongodb.mongo_update", + "mongodb.mongo_list_collections" + ], + "hit1": "mongodb.mongo_find", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "kind": "verb_product", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "ranking": [ + "sqlite.sqlite_list_tables", + "mysql.mysql_list_tables", + "supabase.supabase_list_tables", + "postgres.list_tables", + "sqlite.sqlite_describe_table", + "sqlite.sqlite_query", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "aws_s3.s3_list_buckets", + "linear.linear_list_teams" + ], + "hit1": "sqlite.sqlite_list_tables", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 142, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 240, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 496, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "kind": "verb_product", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "ranking": [ + "postgres.describe_table", + "sqlite.sqlite_describe_table", + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update", + "postgres.explain_query", + "postgres.list_schemas", + "postgres.query", + "postgres.list_tables" + ], + "hit1": "postgres.describe_table", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "bitbucket.bitbucket_create_issue", + "github.issue_write", + "gitlab.gitlab_create_issue", + "github.issue_read", + "github.add_issue_comment", + "linear.linear_create_issue", + "jira.jira_create_issue", + "linear.linear_create_comment", + "github.assign_copilot_to_issue" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 378, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 568, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1091, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 29, + "k3_calls": 10, + "k3_tokens": 2752, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 6, + "k5_tokens": 2680, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 3, + "k10_tokens": 2626, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "kind": "identifier", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "ranking": [ + "filesystem.create_directory", + "filesystem.list_directory", + "filesystem.list_directory_with_sizes", + "filesystem.get_file_info", + "filesystem.directory_tree", + "github.get_file_contents", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "aws_s3.s3_list_buckets", + "linear.linear_list_teams" + ], + "hit1": "filesystem.create_directory", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 607, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "kind": "paraphrase", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "ranking": [ + "sequential_thinking.sequentialthinking" + ], + "hit1": "sequential_thinking.sequentialthinking", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "kind": "paraphrase", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "ranking": [ + "time.get_current_time", + "time.convert_time", + "playwright.browser_wait_for", + "google_calendar.gcal_list_events", + "slack.slack_create_channel", + "filesystem.list_allowed_directories", + "grafana.grafana_create_annotation", + "jira.jira_add_worklog", + "github.get_milestone", + "github.delete_milestone" + ], + "hit1": "time.get_current_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "kind": "verb_product", + "gold": [ + "time.convert_time" + ], + "alts": [], + "ranking": [ + "time.convert_time", + "salesforce.salesforce_convert_lead", + "time.get_current_time", + "playwright.browser_wait_for", + "google_calendar.gcal_list_events", + "memory.create_relations", + "git.git_diff", + "grafana.grafana_create_annotation", + "jira.jira_add_worklog" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "kind": "verb_product", + "gold": [ + "github.star_repository" + ], + "alts": [], + "ranking": [ + "github.star_repository", + "github.delete_repository", + "github.unstar_repository", + "github.fork_repository", + "github.search_issues", + "github.actions_list", + "github.get_label", + "github.list_branches", + "github.list_releases", + "github.get_dependabot_alert" + ], + "hit1": "github.star_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 394, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 779, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "kind": "resource", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "ranking": [ + "github.fork_repository", + "github.delete_repository", + "github.star_repository", + "github.unstar_repository", + "github.create_repository", + "github.list_repository_collaborators", + "github.get_repository_tree", + "github.list_branches", + "github.list_releases", + "github.list_dependabot_alerts" + ], + "hit1": "github.fork_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 792, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "kind": "verb_product", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "ranking": [ + "github.get_latest_release", + "gitlab.get_release", + "github.get_release_by_tag", + "gitlab.delete_release", + "gitlab.update_release", + "github.get_teams", + "gitlab.create_release", + "github.get_me", + "github.get_repository_tree", + "github.get_milestone" + ], + "hit1": "github.get_latest_release", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 361, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 712, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "kind": "resource", + "gold": [ + "github.create_gist" + ], + "alts": [], + "ranking": [ + "github.create_gist", + "github.get_gist", + "github.update_gist", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project", + "trello.trello_create_board", + "stripe.stripe_create_customer", + "stripe.stripe_create_product", + "salesforce.salesforce_create_record", + "okta.okta_create_user" + ], + "hit1": "github.create_gist", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 649, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "kind": "verb_product", + "gold": [ + "github.search_code" + ], + "alts": [], + "ranking": [ + "github.search_code", + "github.list_code_scanning_alerts", + "github.get_code_scanning_alert", + "github.search_users", + "github.search_orgs", + "github.search_commits", + "github.search_issues", + "github.search_pull_requests", + "brave_search.brave_web_search", + "github.search_repositories" + ], + "hit1": "github.search_code", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "kind": "verb_product", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "ranking": [ + "github.list_notifications", + "github.mark_all_notifications_read", + "github.list_milestones", + "github.list_hooks", + "github.list_environments", + "github.list_packages", + "github.list_artifacts", + "github.list_runners", + "github.list_check_runs", + "github.actions_list" + ], + "hit1": "github.list_notifications", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "kind": "verb_product", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "ranking": [ + "github.assign_copilot_to_issue", + "linear.linear_assign_issue", + "jira.jira_assign_issue", + "jira.jira_update_issue", + "jira.jira_transition_issue", + "github.add_issue_comment", + "linear.linear_create_comment", + "jira.jira_add_comment", + "github.request_copilot_review", + "jira.jira_add_worklog" + ], + "hit1": "github.assign_copilot_to_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 833, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "kind": "resource", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "ranking": [ + "github.list_secret_scanning_alerts", + "github.get_secret_scanning_alert", + "github.list_code_scanning_alerts", + "github.get_code_scanning_alert", + "github.list_dependabot_alerts", + "redis.redis_get", + "kubernetes.k8s_get_pod", + "shopify.shopify_get_product", + "shopify.shopify_get_order", + "vercel.vercel_get_deployment" + ], + "hit1": "github.list_secret_scanning_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "ranking": [ + "github.create_or_update_file", + "filesystem.write_file", + "slack.update_file", + "google_drive.gdrive_update_file", + "github.projects_write", + "figma.figma_get_comments", + "github.label_write", + "figma.figma_post_comment", + "google_drive.gdrive_get_file", + "github.get_file_contents" + ], + "hit1": "github.create_or_update_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 306, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 481, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 889, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "kind": "resource", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "ranking": [ + "github.list_repository_collaborators", + "github.list_commits", + "trello.trello_list_cards", + "pagerduty.pagerduty_list_oncalls", + "github.get_discussion_comments", + "github.update_issue_comment", + "github.request_copilot_review", + "github.actions_list", + "bitbucket.bitbucket_list_prs", + "github.create_pull_request" + ], + "hit1": "github.list_repository_collaborators", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 278, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 433, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 932, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "kind": "verb_product", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "ranking": [ + "slack.slack_reply_to_thread", + "slack.slack_get_thread_replies", + "github.get_notification_details", + "slack.get_reminder", + "slack.delete_reminder", + "slack.get_canvase", + "slack.delete_canvase", + "slack.get_bookmark", + "slack.delete_bookmark", + "slack.get_usergroup" + ], + "hit1": "slack.slack_reply_to_thread", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "kind": "verb_product", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "ranking": [ + "slack.slack_add_reaction", + "discord.discord_add_reaction", + "trello.trello_add_comment", + "gmail.gmail_modify_email", + "playwright.browser_click", + "jira.jira_add_comment", + "memory.add_observations", + "git.git_add", + "jira.jira_add_worklog", + "github.assign_copilot_to_issue" + ], + "hit1": "slack.slack_add_reaction", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 271, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 811, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "kind": "verb_product", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "ranking": [ + "slack.slack_search_messages", + "gmail.gmail_search_emails", + "slack.slack_get_channel_history", + "twilio.twilio_list_messages", + "brave_search.brave_web_search", + "confluence.confluence_search", + "github.search_users", + "github.search_orgs", + "grafana.grafana_search_dashboards", + "brave_search.brave_local_search" + ], + "hit1": "slack.slack_search_messages", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 664, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "kind": "verb_product", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "ranking": [ + "slack.slack_create_channel", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_get_channel_history", + "slack.slack_set_channel_topic", + "slack.slack_post_message", + "slack.slack_upload_file", + "slack.slack_reply_to_thread", + "slack.slack_get_thread_replies", + "slack.slack_add_reaction" + ], + "hit1": "slack.slack_create_channel", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "kind": "verb_product", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_transition_issue", + "jira.jira_get_issue", + "jira.jira_assign_issue", + "jira.jira_update_issue", + "jira.jira_add_comment", + "jira.jira_add_worklog", + "jira.jira_create_issue", + "sentry.sentry_get_issue", + "gitlab.gitlab_get_issue", + "sentry.sentry_update_issue" + ], + "hit1": "jira.jira_transition_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 778, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "kind": "identifier", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "ranking": [ + "jira.jira_search_issues", + "filesystem.list_directory_with_sizes", + "brave_search.brave_web_search", + "confluence.confluence_search", + "github.search_users", + "github.search_orgs", + "grafana.grafana_search_dashboards", + "jira.jira_list_projects", + "brave_search.brave_local_search", + "github.search_commits" + ], + "hit1": "jira.jira_search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 241, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 619, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "kind": "verb_product", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "ranking": [ + "jira.jira_add_worklog", + "jira.jira_add_comment", + "memory.add_observations", + "git.git_add", + "discord.discord_add_reaction", + "github.add_issue_comment", + "slack.slack_add_reaction", + "github.add_comment_to_pending_review", + "jira.jira_transition_issue", + "linear.linear_create_comment" + ], + "hit1": "jira.jira_add_worklog", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 236, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 874, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "kind": "verb_product", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "ranking": [ + "linear.linear_list_teams", + "github.get_teams", + "linear.linear_list_projects", + "linear.linear_list_issues", + "linear.list_cycles", + "linear.list_labels", + "linear.list_users", + "linear.list_comments", + "linear.list_attachments", + "linear.linear_search_issues" + ], + "hit1": "linear.linear_list_teams", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 137, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 646, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "ranking": [ + "linear.linear_create_comment", + "github.update_issue_comment", + "linear.get_comment", + "linear.delete_comment", + "linear.update_comment", + "github.add_issue_comment", + "jira.jira_add_comment", + "linear.linear_update_issue", + "linear.linear_get_issue", + "linear.linear_assign_issue" + ], + "hit1": "linear.linear_create_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "kind": "paraphrase", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "ranking": [ + "filesystem.move_file", + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "brave_search.brave_local_search", + "gmail.gmail_trash_email", + "figma.figma_get_file", + "google_drive.gdrive_update_file", + "google_drive.gdrive_share_file", + "slack.get_file" + ], + "hit1": "filesystem.move_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "kind": "paraphrase", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "ranking": [ + "filesystem.search_files", + "linear.linear_search_issues", + "google_drive.gdrive_search", + "redis.redis_keys", + "github.search_repositories", + "slack.list_files", + "brave_search.brave_web_search", + "github.push_files", + "confluence.confluence_search", + "github.search_users" + ], + "hit1": "filesystem.search_files", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "kind": "paraphrase", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "ranking": [ + "filesystem.edit_file", + "filesystem.list_directory_with_sizes", + "filesystem.write_file", + "filesystem.move_file", + "playwright.browser_tab_new", + "slack.get_file", + "slack.delete_file", + "twilio.twilio_make_call", + "slack.update_file", + "figma.figma_get_file" + ], + "hit1": "filesystem.edit_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "ranking": [ + "memory.read_graph", + "memory.delete_relations", + "memory.create_entities", + "memory.create_relations", + "memory.add_observations", + "memory.open_nodes", + "memory.delete_entities", + "memory.search_nodes", + "filesystem.read_file", + "filesystem.read_text_file" + ], + "hit1": "memory.read_graph", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 259, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 561, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "kind": "verb_product", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "memory.open_nodes", + "figma.figma_get_file_nodes", + "brave_search.brave_web_search", + "confluence.confluence_search", + "github.search_users", + "github.search_orgs", + "grafana.grafana_search_dashboards", + "brave_search.brave_local_search", + "github.search_commits" + ], + "hit1": "memory.search_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 635, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "kind": "verb_product", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "ranking": [ + "git.git_diff_unstaged", + "git.git_diff", + "git.git_diff_staged", + "git.git_show", + "git.git_log", + "git.git_status", + "git.git_branch", + "git.git_checkout", + "git.git_commit", + "git.git_add" + ], + "hit1": "git.git_diff_unstaged", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 313, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 602, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "kind": "identifier", + "gold": [ + "git.git_add" + ], + "alts": [], + "ranking": [ + "git.git_add", + "filesystem.list_directory_with_sizes", + "salesforce.salesforce_list_opportunities", + "github.push_files", + "github.add_issue_comment", + "github.create_gist", + "slack.list_files", + "github.update_gist", + "jira.jira_add_comment", + "github.add_comment_to_pending_review" + ], + "hit1": "git.git_add", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 197, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 872, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "kind": "verb_product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "ranking": [ + "shopify.shopify_create_product", + "shopify.shopify_get_product", + "shopify.shopify_update_product", + "shopify.shopify_create_order", + "stripe.stripe_create_product", + "shopify.shopify_get_order", + "shopify.shopify_list_customers", + "shopify.shopify_list_products", + "shopify.shopify_list_orders", + "stripe.stripe_create_price" + ], + "hit1": "shopify.shopify_create_product", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "kind": "verb_product", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "ranking": [ + "stripe.stripe_cancel_subscription", + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_list_customers", + "stripe.stripe_create_product", + "stripe.stripe_list_subscriptions", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_refund" + ], + "hit1": "stripe.stripe_cancel_subscription", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 167, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 285, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 610, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "kind": "resource", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_refund", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project", + "trello.trello_create_board", + "stripe.stripe_create_customer", + "stripe.stripe_create_product", + "salesforce.salesforce_create_record", + "okta.okta_create_user" + ], + "hit1": "stripe.stripe_create_payment_intent", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 654, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "kind": "verb_product", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "ranking": [ + "datadog.datadog_create_monitor", + "datadog.datadog_get_monitor", + "datadog.datadog_list_monitors", + "datadog.datadog_list_logs", + "datadog.datadog_query_metrics", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project", + "trello.trello_create_board", + "stripe.stripe_create_customer", + "stripe.stripe_create_product" + ], + "hit1": "datadog.datadog_create_monitor", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "kind": "paraphrase", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "ranking": [ + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "sentry.sentry_list_events", + "pagerduty.pagerduty_resolve_incident", + "sentry.sentry_list_projects", + "sentry.sentry_list_issues", + "jira.jira_get_issue", + "jira.jira_assign_issue", + "jira.jira_update_issue", + "gitlab.gitlab_get_issue" + ], + "hit1": "sentry.sentry_get_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_scale_deployment", + "vercel.vercel_get_deployment", + "kubernetes.k8s_apply", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_delete", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_pod_logs" + ], + "hit1": "kubernetes.k8s_scale_deployment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_apply", + "terraform.tf_apply", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_delete", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_pod_logs" + ], + "hit1": "kubernetes.k8s_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 308, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "ranking": [ + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_exec", + "docker.docker_run", + "docker.docker_images", + "docker.docker_ps", + "docker.docker_build", + "kubernetes.k8s_pod_logs" + ], + "hit1": "docker.docker_stop", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "kind": "verb_product", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "ranking": [ + "docker.docker_build", + "docker.docker_run", + "docker.docker_images", + "docker.docker_stop", + "docker.docker_ps", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_exec", + "filesystem.read_media_file" + ], + "hit1": "docker.docker_build", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "kind": "verb_product", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "ranking": [ + "playwright.browser_click", + "puppeteer.puppeteer_click", + "playwright.browser_type", + "playwright.browser_take_screenshot", + "playwright.browser_snapshot", + "clickup.clickup_list_tasks", + "clickup.clickup_get_task", + "playwright.browser_tab_list", + "playwright.browser_tab_close", + "clickup.clickup_create_task" + ], + "hit1": "playwright.browser_click", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 612, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "kind": "paraphrase", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "ranking": [ + "playwright.browser_type", + "playwright.browser_wait_for", + "puppeteer.puppeteer_fill", + "playwright.browser_tab_list", + "playwright.browser_tab_close", + "playwright.browser_navigate", + "playwright.browser_tab_new", + "filesystem.read_text_file", + "supabase.supabase_insert", + "mongodb.mongo_insert" + ], + "hit1": "playwright.browser_type", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 648, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_list_projects", + "jira.jira_list_projects", + "vercel.vercel_list_projects", + "linear.linear_list_projects", + "sentry.sentry_list_projects", + "github.projects_list", + "gitlab.list_pipelines", + "gitlab.list_jobs", + "gitlab.list_wikis", + "gitlab.list_snippets" + ], + "hit1": "gitlab.gitlab_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 143, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 240, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 623, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "kind": "verb_product", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "ranking": [ + "bitbucket.bitbucket_create_pr", + "bitbucket.bitbucket_list_prs", + "bitbucket.bitbucket_create_issue", + "github.pull_request_review_write", + "github.pull_request_read", + "github.merge_pull_request", + "github.request_copilot_review", + "github.create_pull_request", + "github.update_pull_request_branch", + "github.update_pull_request" + ], + "hit1": "bitbucket.bitbucket_create_pr", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 546, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1139, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "kind": "verb_product", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "ranking": [ + "gmail.gmail_create_draft", + "gmail.gmail_list_labels", + "gmail.gmail_search_emails", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "gmail.gmail_send_email", + "gmail.gmail_modify_email", + "github.create_pull_request", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project" + ], + "hit1": "gmail.gmail_create_draft", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "kind": "verb_product", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_share_file", + "google_drive.gdrive_update_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "google_calendar.gcal_list_calendars", + "figma.figma_get_file", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event" + ], + "hit1": "google_drive.gdrive_share_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "kind": "verb_product", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "kubernetes.k8s_get_events", + "sentry.sentry_list_events", + "google_drive.gdrive_list_folder", + "google_drive.gdrive_search", + "google_drive.gdrive_update_file" + ], + "hit1": "google_calendar.gcal_list_events", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "redis.redis_delete", + "kubernetes.k8s_delete", + "aws_s3.s3_delete_object", + "github.delete_milestone", + "github.delete_hook" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 735, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_projects", + "vercel.vercel_list_deployments", + "jira.jira_list_projects", + "linear.linear_list_projects", + "sentry.sentry_list_projects", + "vercel.vercel_get_deployment", + "vercel.vercel_create_project", + "gitlab.gitlab_list_projects", + "github.projects_list", + "github.projects_get" + ], + "hit1": "vercel.vercel_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 248, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "ranking": [ + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update", + "supabase.supabase_list_tables", + "supabase.supabase_invoke_fn", + "memory.delete_observations", + "aws_s3.s3_get_object", + "memory.delete_relations", + "docker.docker_logs", + "elasticsearch.es_get_doc" + ], + "hit1": "supabase.supabase_select", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "kind": "verb_product", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "figma.figma_get_file", + "figma.figma_get_comments", + "slack.slack_post_message", + "figma.figma_get_images", + "figma.figma_get_file_nodes", + "linear.get_comment", + "linear.delete_comment", + "notion.get_comment", + "notion.delete_comment" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 646, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "kind": "verb_product", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "ranking": [ + "okta.okta_list_users", + "okta.okta_create_user", + "okta.okta_get_user", + "okta.okta_deactivate_user", + "notion.notion_list_users", + "slack.slack_list_users", + "linear.list_users", + "notion.list_users_tokens", + "slack.slack_get_users", + "github.search_users" + ], + "hit1": "okta.okta_list_users", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 271, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query_range", + "prometheus.prom_query", + "prometheus.prom_list_metrics", + "datadog.datadog_query_metrics", + "postgres.explain_query", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "postgres.query", + "notion.notion_query_database" + ], + "hit1": "prometheus.prom_query_range", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "kind": "verb_product", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "ranking": [ + "grafana.grafana_create_annotation", + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project", + "trello.trello_create_board", + "stripe.stripe_create_customer", + "stripe.stripe_create_product", + "salesforce.salesforce_create_record", + "okta.okta_create_user" + ], + "hit1": "grafana.grafana_create_annotation", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 638, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "kind": "identifier", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "ranking": [ + "terraform.tf_apply", + "kubernetes.k8s_apply", + "terraform.tf_plan", + "terraform.tf_state_list" + ], + "hit1": "terraform.tf_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "kind": "verb_product", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "ranking": [ + "asana.asana_list_tasks", + "clickup.clickup_list_tasks", + "asana.asana_get_task", + "asana.asana_create_project", + "asana.asana_create_task", + "asana.asana_update_task", + "trello.trello_list_cards", + "aws_s3.s3_list_buckets", + "linear.linear_list_teams", + "jira.jira_list_projects" + ], + "hit1": "asana.asana_list_tasks", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 642, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "ranking": [ + "trello.trello_add_comment", + "trello.trello_create_card", + "trello.trello_list_cards", + "trello.trello_create_board", + "github.update_issue_comment", + "figma.figma_post_comment", + "linear.get_comment", + "linear.delete_comment", + "notion.get_comment", + "notion.delete_comment" + ], + "hit1": "trello.trello_add_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 422, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_update_task", + "clickup.clickup_get_task", + "clickup.clickup_create_task", + "asana.asana_update_task", + "asana.asana_get_task", + "clickup.clickup_list_tasks", + "asana.asana_create_task", + "mongodb.mongo_update", + "shopify.shopify_update_product", + "github.update_milestone" + ], + "hit1": "clickup.clickup_update_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 398, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 789, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "kind": "verb_product", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "ranking": [ + "confluence.confluence_search", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "confluence.confluence_create_page", + "brave_search.brave_web_search", + "github.search_users", + "github.search_orgs", + "grafana.grafana_search_dashboards", + "brave_search.brave_local_search", + "github.search_commits" + ], + "hit1": "confluence.confluence_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 691, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_deal", + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_search_contacts", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project", + "trello.trello_create_board", + "stripe.stripe_create_customer", + "stripe.stripe_create_product" + ], + "hit1": "hubspot.hubspot_create_deal", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_search", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_create_ticket", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "aws_s3.s3_list_buckets", + "linear.linear_list_teams", + "jira.jira_list_projects" + ], + "hit1": "zendesk.zendesk_list_tickets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 605, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "kind": "verb_product", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "ranking": [ + "discord.discord_list_channels", + "discord.discord_list_members", + "slack.slack_list_channels", + "discord.discord_get_channel", + "discord.discord_send_message", + "discord.discord_add_reaction", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "aws_s3.s3_list_buckets", + "linear.linear_list_teams" + ], + "hit1": "discord.discord_list_channels", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 592, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "ranking": [ + "redis.redis_set", + "redis.redis_get", + "redis.redis_delete", + "redis.redis_keys", + "slack.slack_set_channel_topic", + "playwright.browser_press_key", + "jira.jira_get_issue", + "aws_s3.s3_delete_object", + "aws_s3.s3_get_object", + "figma.figma_get_file" + ], + "hit1": "redis.redis_set", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert", + "mongodb.mongo_delete", + "mongodb.mongo_find", + "mongodb.mongo_update", + "supabase.supabase_insert", + "mongodb.mongo_list_collections" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "kind": "verb_product", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "ranking": [ + "mysql.mysql_query", + "mysql.mysql_list_tables", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "postgres.query", + "docker.docker_run", + "github.get_check_run", + "github.delete_check_run", + "github.update_check_run", + "prometheus.prom_query" + ], + "hit1": "mysql.mysql_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 181, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 286, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 608, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "elasticsearch.es_search", + "playwright.browser_tab_close", + "github.get_milestone", + "github.delete_milestone", + "github.get_hook", + "github.delete_hook", + "github.get_environment", + "github.delete_environment" + ], + "hit1": "elasticsearch.es_index_doc", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 346, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 605, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_put_object", + "okta.okta_get_user", + "asana.asana_get_task", + "puppeteer.puppeteer_click", + "elasticsearch.es_search", + "okta.okta_create_user", + "puppeteer.puppeteer_fill", + "aws_s3.s3_list_objects" + ], + "hit1": "aws_s3.s3_get_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 668, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones", + "cloudflare.cf_list_dns", + "cloudflare.cf_purge_cache", + "cloudflare.cf_create_dns", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "aws_s3.s3_list_buckets", + "linear.linear_list_teams", + "jira.jira_list_projects", + "kubernetes.k8s_list_deployments" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 579, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_create_customer", + "stripe.stripe_create_product", + "stripe.stripe_create_refund", + "stripe.stripe_create_price", + "stripe.stripe_create_payment_intent", + "stripe.stripe_list_invoices", + "stripe.stripe_get_customer", + "stripe.stripe_list_customers", + "stripe.stripe_list_subscriptions" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "kind": "verb_product", + "gold": [ + "github.actions_list" + ], + "alts": [], + "ranking": [ + "github.actions_list", + "github.actions_get", + "github.actions_run_trigger", + "github.actions_get_job_logs", + "github.list_milestones", + "github.list_hooks", + "github.list_environments", + "github.list_packages", + "github.list_artifacts", + "github.list_runners" + ], + "hit1": "github.actions_list", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 468, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 845, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "kind": "verb_product", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "ranking": [ + "github.actions_get_job_logs", + "github.actions_get", + "gitlab.get_job", + "github.actions_list", + "kubernetes.k8s_pod_logs", + "github.actions_run_trigger", + "gitlab.delete_job", + "gitlab.update_job", + "docker.docker_logs", + "gitlab.create_job" + ], + "hit1": "github.actions_get_job_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 421, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 822, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "kind": "verb_product", + "gold": [ + "github.create_repository" + ], + "alts": [], + "ranking": [ + "github.create_repository", + "github.create_branch", + "github.delete_repository", + "github.star_repository", + "github.unstar_repository", + "github.fork_repository", + "github.create_issue", + "github.list_repository_collaborators", + "github.create_or_update_file", + "github.get_repository_tree" + ], + "hit1": "github.create_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 406, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 949, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "kind": "verb_product", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "ranking": [ + "git.git_checkout", + "git.git_create_branch", + "github.create_branch", + "git.git_branch", + "gitlab.gitlab_create_merge_request", + "github.update_pull_request_branch", + "github.list_commits", + "github.delete_file", + "github.push_files", + "github.create_or_update_file" + ], + "hit1": "git.git_checkout", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 266, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1001, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "kind": "identifier", + "gold": [ + "git.git_log" + ], + "alts": [], + "ranking": [ + "git.git_log", + "git.git_show", + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_status", + "git.git_diff", + "git.git_branch", + "git.git_checkout", + "git.git_commit", + "git.git_add" + ], + "hit1": "git.git_log", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "kind": "verb_product", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "ranking": [ + "postgres.list_schemas", + "postgres.list_tables", + "postgres.explain_query", + "postgres.query", + "postgres.describe_table", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "aws_s3.s3_list_buckets", + "linear.linear_list_teams", + "jira.jira_list_projects" + ], + "hit1": "postgres.list_schemas", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 160, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 515, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "kind": "verb_product", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "ranking": [ + "memory.open_nodes", + "memory.search_nodes", + "memory.read_graph", + "memory.create_entities", + "memory.delete_relations", + "memory.create_relations", + "memory.add_observations", + "memory.delete_entities", + "figma.figma_get_file_nodes", + "playwright.browser_tab_list" + ], + "hit1": "memory.open_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 260, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 531, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "kind": "paraphrase", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "ranking": [ + "filesystem.read_file", + "filesystem.get_file_info", + "filesystem.read_text_file", + "filesystem.read_media_file", + "google_drive.gdrive_get_file", + "figma.figma_get_file_nodes", + "gitlab.gitlab_get_file", + "figma.figma_get_images", + "github.get_file_contents", + "filesystem.write_file" + ], + "hit1": "filesystem.read_file", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 220, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 766, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "kind": "paraphrase", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "ranking": [ + "filesystem.write_file", + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.edit_file", + "filesystem.read_media_file", + "playwright.browser_tab_new", + "brave_search.brave_local_search", + "github.issue_write", + "figma.figma_get_file", + "google_drive.gdrive_update_file" + ], + "hit1": "filesystem.write_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 791, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "kind": "verb_product", + "gold": [ + "github.search_issues" + ], + "alts": [], + "ranking": [ + "github.search_issues", + "linear.linear_search_issues", + "jira.jira_search_issues", + "github.list_issues", + "github.search_users", + "github.search_orgs", + "github.search_commits", + "github.search_pull_requests", + "gitlab.gitlab_list_issues", + "linear.linear_list_issues" + ], + "hit1": "github.search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 233, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 700, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "kind": "paraphrase", + "gold": [ + "github.get_me" + ], + "alts": [], + "ranking": [ + "github.get_teams", + "github.get_me", + "github.list_notifications", + "okta.okta_get_user", + "slack.slack_get_user_profile", + "linear.get_user", + "github.get_latest_release", + "okta.okta_deactivate_user", + "linear.delete_user", + "okta.okta_create_user" + ], + "hit1": "github.get_teams", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 288, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 569, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "cost": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.4, + "p95_calls": 1.0, + "success_within_1": 0.9733333333333334, + "success_within_2": 0.98, + "success_within_3": 0.9866666666666667, + "mean_tokens": 331.55333333333334, + "p95_tokens": 323.99999999999966, + "total_tokens": 49733, + "tokens_per_successful_query": 331.55333333333334, + "total_tokens_successful": 49733 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.2333333333333334, + "p95_calls": 1.0, + "success_within_1": 0.98, + "success_within_2": 0.9866666666666667, + "success_within_3": 0.9866666666666667, + "mean_tokens": 465.3, + "p95_tokens": 478.29999999999984, + "total_tokens": 69795, + "tokens_per_successful_query": 465.3, + "total_tokens_successful": 69795 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.1066666666666667, + "p95_calls": 1.0, + "success_within_1": 0.9866666666666667, + "success_within_2": 0.9866666666666667, + "success_within_3": 0.9933333333333333, + "mean_tokens": 803.7666666666667, + "p95_tokens": 989.2999999999995, + "total_tokens": 120565, + "tokens_per_successful_query": 803.7666666666667, + "total_tokens_successful": 120565 + } + } +} diff --git a/experiments/mcp-tool-search/results/bm25_tfidf_hybrid.csv b/experiments/mcp-tool-search/results/bm25_tfidf_hybrid.csv new file mode 100644 index 00000000..0690303c --- /dev/null +++ b/experiments/mcp-tool-search/results/bm25_tfidf_hybrid.csv @@ -0,0 +1,151 @@ +id,query,kind,hit1,recall@1,recall@3,recall@5,mrr@10,ndcg@10,hit_rank,k3_calls,k3_tokens,k3_success1,k5_calls,k5_tokens,k5_success1,k10_calls,k10_tokens,k10_success1,ranking +q001,Create a GitHub issue,verb_product,gitlab.gitlab_create_issue,0.0,1.0,1.0,0.5000,0.6063,2,1,349,1.0,1,521,1.0,1,972,1.0,gitlab.gitlab_create_issue github.create_issue github.add_issue_comment github.issue_read gitlab.gitlab_get_issue github.update_issue_comment github.issue_write bitbucket.bitbucket_create_issue linear.linear_create_comment jira.jira_get_issue +q002,Create a Jira ticket,verb_product,jira.jira_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,281,1.0,1,466,1.0,1,951,1.0,jira.jira_create_issue zendesk.zendesk_get_ticket zendesk.zendesk_create_ticket zendesk.zendesk_update_ticket jira.create_filter jira.create_attachment jira.create_board jira.create_sprint jira.create_version jira.create_webhook +q003,Send a Slack message,verb_product,discord.discord_send_message,0.0,1.0,1.0,0.5000,0.6309,2,1,250,1.0,1,427,1.0,1,753,1.0,discord.discord_send_message slack.slack_post_message gmail.gmail_send_email twilio.twilio_send_sms slack.slack_add_reaction slack.get_file discord.discord_add_reaction slack.delete_file slack.slack_get_users slack.slack_list_users +q004,List open pull requests,resource,github.list_pull_requests,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,460,1.0,1,836,1.0,github.list_pull_requests github.search_pull_requests bitbucket.bitbucket_list_prs gitlab.gitlab_list_merge_requests github.create_pull_request memory.open_nodes playwright.browser_tab_list github.pull_request_read github.merge_pull_request github.update_pull_request_branch +q005,Open a pull request on GitHub,verb_product,github.create_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,364,1.0,1,611,1.0,1,1144,1.0,github.create_pull_request github.request_copilot_review github.pull_request_read github.merge_pull_request github.update_pull_request github.pull_request_review_write github.update_pull_request_branch github.search_pull_requests github.list_pull_requests github.add_comment_to_pending_review +q006,Run a SQL query against postgres,verb_product,postgres.query,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,295,1.0,1,553,1.0,postgres.query mysql.mysql_query sqlite.sqlite_query postgres.explain_query salesforce.salesforce_query postgres.list_schemas postgres.list_tables postgres.describe_table github.get_check_run github.delete_check_run +q007,Read a local file from disk,paraphrase,filesystem.read_file,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,428,1.0,1,780,1.0,filesystem.read_file filesystem.read_text_file filesystem.read_media_file gitlab.gitlab_get_file github.delete_file memory.read_graph github.issue_read brave_search.brave_local_search slack.get_file github.pull_request_read +q008,What files are in this folder,paraphrase,google_drive.gdrive_list_folder,0.0,0.0,0.0,0.1250,0.2938,8,3,712,0.0,2,745,0.0,1,723,1.0,google_drive.gdrive_list_folder slack.list_files github.push_files filesystem.search_files google_drive.gdrive_search github.create_gist github.update_gist filesystem.list_directory git.git_add filesystem.directory_tree +q009,Remember this fact in the knowledge graph,paraphrase,memory.read_graph,0.0,1.0,1.0,0.5000,0.4932,2,1,154,1.0,1,262,1.0,1,544,1.0,memory.read_graph memory.create_entities memory.delete_relations memory.create_relations memory.search_nodes memory.open_nodes memory.add_observations memory.delete_entities filesystem.read_file slack.slack_list_channels +q010,Search the web with Brave,verb_product,brave_search.brave_web_search,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,367,1.0,1,710,1.0,brave_search.brave_web_search brave_search.brave_local_search jira.jira_search_issues filesystem.list_directory_with_sizes github.search_users memory.search_nodes github.update_pull_request_branch github.search_issues github.search_orgs github.search_commits +q011,Fetch the contents of a URL,verb_product,fetch.fetch,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,372,1.0,1,640,1.0,fetch.fetch filesystem.read_file playwright.browser_navigate github.get_file_contents puppeteer.puppeteer_navigate git.git_add docker.docker_logs git.git_status playwright.browser_tab_new playwright.browser_snapshot +q012,Show git status,identifier,git.git_show,0.0,1.0,1.0,0.5000,0.6309,2,1,183,1.0,1,299,1.0,1,645,1.0,git.git_show git.git_status git.git_diff_staged git.git_log git.git_diff_unstaged git.git_diff git.git_branch sentry.sentry_update_issue slack.list_files linear.list_users +q013,Commit staged changes,paraphrase,git.git_commit,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,327,1.0,1,663,1.0,git.git_commit git.git_reset git.git_diff_staged github.get_commit git.git_show git.git_log github.push_files +q014,Create a Linear issue,verb_product,linear.linear_create_comment,0.0,0.0,1.0,0.2500,0.4307,4,2,574,0.0,1,463,1.0,1,921,1.0,linear.linear_create_comment linear.linear_get_issue linear.linear_update_issue linear.linear_create_issue linear.linear_assign_issue gitlab.gitlab_create_issue linear.linear_create_project linear.create_comment linear.create_user linear.create_label +q015,Make a task in my project tracker Linear,paraphrase,linear.linear_create_project,0.0,0.0,0.0,0.0000,0.0000,145,49,12448,0.0,29,11769,0.0,15,11810,0.0,linear.linear_create_project clickup.clickup_get_task asana.asana_get_task clickup.clickup_update_task asana.asana_update_task clickup.clickup_create_task asana.asana_create_task mysql.mysql_list_tables github.projects_get mysql.mysql_query +q016,Post a comment on a GitHub issue,verb_product,github.update_issue_comment,0.0,1.0,1.0,0.3333,0.5000,3,1,307,1.0,1,444,1.0,1,819,1.0,github.update_issue_comment figma.figma_post_comment github.add_issue_comment slack.slack_post_message linear.linear_create_comment jira.jira_add_comment github.assign_copilot_to_issue linear.get_comment notion.get_comment github.issue_read +q017,Merge the pull request,ambiguous,github.merge_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,283,1.0,1,495,1.0,1,1121,1.0,github.merge_pull_request gitlab.gitlab_merge_merge_request gitlab.gitlab_get_merge_request github.update_pull_request_branch gitlab.gitlab_create_merge_request github.pull_request_read github.update_pull_request github.request_copilot_review github.pull_request_review_write github.create_pull_request +q018,Trigger a GitHub Actions workflow,verb_product,github.actions_run_trigger,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,458,1.0,1,795,1.0,github.actions_run_trigger github.actions_get github.actions_list github.actions_get_job_logs circleci.circleci_trigger_pipeline github.delete_repository github.list_tags github.get_label github.search_users github.projects_get +q019,Get Dependabot alerts,resource,github.list_dependabot_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,420,1.0,1,684,1.0,github.list_dependabot_alerts github.get_dependabot_alert github.list_secret_scanning_alerts github.list_code_scanning_alerts slack.get_file linear.get_comment github.get_milestone linear.get_user notion.get_comment gitlab.get_label +q020,List Sentry issues,verb_product,sentry.sentry_list_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,236,1.0,1,380,1.0,1,796,1.0,sentry.sentry_list_issues sentry.sentry_list_projects sentry.sentry_get_issue sentry.sentry_list_events gitlab.gitlab_list_issues github.list_issues sentry.sentry_update_issue github.search_issues linear.linear_search_issues linear.linear_list_issues +q021,Create a Stripe customer,verb_product,stripe.stripe_create_customer,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,300,1.0,1,669,1.0,stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_list_subscriptions stripe.stripe_create_payment_intent stripe.stripe_create_product stripe.stripe_create_refund stripe.stripe_create_price stripe.stripe_list_customers +q022,Refund a Stripe payment,paraphrase,stripe.stripe_create_refund,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,370,1.0,1,647,1.0,stripe.stripe_create_refund stripe.stripe_get_payment_intent stripe.stripe_create_payment_intent stripe.stripe_create_product stripe.stripe_create_customer stripe.stripe_list_customers stripe.stripe_get_customer stripe.stripe_list_invoices stripe.stripe_create_invoice stripe.stripe_list_subscriptions +q023,Send an email with Gmail,verb_product,gmail.gmail_send_email,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,344,1.0,1,711,1.0,gmail.gmail_send_email gmail.gmail_get_email gmail.gmail_trash_email gmail.gmail_list_labels twilio.twilio_send_sms gmail.gmail_modify_email gmail.gmail_create_draft discord.discord_send_message gmail.gmail_search_emails puppeteer.puppeteer_click +q024,Search my Gmail inbox,verb_product,gmail.gmail_search_emails,1.0,1.0,1.0,1.0000,1.0000,1,1,178,1.0,1,315,1.0,1,652,1.0,gmail.gmail_search_emails gmail.gmail_list_labels gmail.gmail_get_email gmail.gmail_create_draft gmail.gmail_trash_email gmail.gmail_send_email mysql.mysql_list_tables gmail.gmail_modify_email mysql.mysql_query github.search_users +q025,Upload a file to Google Drive,paraphrase,google_drive.gdrive_update_file,0.0,0.0,1.0,0.2500,0.4307,4,2,523,0.0,1,446,1.0,1,735,1.0,google_drive.gdrive_update_file google_drive.gdrive_get_file slack.slack_upload_file google_drive.gdrive_create_file google_drive.gdrive_share_file google_drive.gdrive_search google_drive.gdrive_list_folder aws_s3.s3_put_object slack.get_file google_calendar.gcal_list_calendars +q026,Create a calendar event,resource,google_calendar.gcal_delete_event,0.0,1.0,1.0,0.3333,0.5000,3,1,300,1.0,1,427,1.0,1,962,1.0,google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars github.pull_request_review_write slack.create_file linear.create_comment github.create_milestone linear.create_user +q027,List Kubernetes pods,verb_product,kubernetes.k8s_list_pods,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,324,1.0,1,699,1.0,kubernetes.k8s_list_pods kubernetes.k8s_list_deployments kubernetes.k8s_get_events kubernetes.k8s_list_services kubernetes.k8s_delete kubernetes.k8s_get_pod kubernetes.k8s_apply kubernetes.k8s_pod_logs kubernetes.k8s_scale_deployment jira.jira_list_projects +q028,Get logs from a k8s pod,verb_product,kubernetes.k8s_pod_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,262,1.0,1,455,1.0,1,834,1.0,kubernetes.k8s_pod_logs kubernetes.k8s_get_pod docker.docker_logs datadog.datadog_list_logs github.actions_get_job_logs git.git_log aws_s3.s3_get_object gitlab.gitlab_get_file github.get_tag elasticsearch.es_get_doc +q029,Run a docker container,verb_product,docker.docker_run,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,345,1.0,1,597,1.0,docker.docker_run docker.docker_exec docker.docker_stop docker.docker_rm docker.docker_logs docker.docker_images docker.docker_ps docker.docker_build github.get_check_run github.delete_check_run +q030,Show docker container logs,verb_product,docker.docker_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,345,1.0,1,645,1.0,docker.docker_logs docker.docker_stop docker.docker_rm docker.docker_exec docker.docker_run kubernetes.k8s_pod_logs git.git_log docker.docker_images git.git_show docker.docker_ps +q031,Navigate the browser to a URL,ambiguous,playwright.browser_navigate,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,294,1.0,1,596,1.0,playwright.browser_navigate puppeteer.puppeteer_navigate playwright.browser_tab_new fetch.fetch playwright.browser_tab_list twilio.twilio_make_call playwright.browser_press_key playwright.browser_click playwright.browser_tab_close playwright.browser_evaluate +q032,Take a screenshot of the page,ambiguous,playwright.browser_take_screenshot,1.0,1.0,1.0,1.0000,1.0000,1,1,226,1.0,1,334,1.0,1,678,1.0,playwright.browser_take_screenshot puppeteer.puppeteer_screenshot notion.notion_get_page playwright.browser_snapshot notion.notion_update_page confluence.confluence_get_page filesystem.read_file confluence.confluence_update_page notion.notion_create_page git.git_status +q033,Create a GitLab merge request,verb_product,gitlab.gitlab_merge_merge_request,0.0,1.0,1.0,0.3333,0.5000,3,1,297,1.0,1,475,1.0,1,939,1.0,gitlab.gitlab_merge_merge_request gitlab.gitlab_get_merge_request gitlab.gitlab_create_merge_request github.merge_pull_request gitlab.gitlab_list_merge_requests gitlab.gitlab_create_issue gitlab.gitlab_create_pipeline github.create_pull_request github.request_copilot_review gitlab.gitlab_list_projects +q034,Search Notion pages,verb_product,notion.notion_search,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,286,1.0,1,639,1.0,notion.notion_search notion.notion_list_users notion.get_comment notion.notion_get_page notion.delete_comment github.search_users notion.update_comment notion.notion_create_database notion.create_comment notion.notion_get_block +q035,Create a Notion page,verb_product,notion.notion_get_page,0.0,1.0,1.0,0.3333,0.5000,3,1,230,1.0,1,382,1.0,1,707,1.0,notion.notion_get_page notion.notion_update_page notion.notion_create_page notion.notion_get_comments notion.create_comment notion.notion_create_database confluence.confluence_create_page notion.notion_list_users notion.get_comment confluence.confluence_get_page +q036,Query a Notion database,verb_product,notion.notion_query_database,1.0,1.0,1.0,1.0000,1.0000,1,1,279,1.0,1,386,1.0,1,709,1.0,notion.notion_query_database notion.notion_create_database notion.notion_create_page notion.notion_search notion.notion_list_users notion.get_comment notion.notion_get_page notion.delete_comment notion.update_comment notion.create_comment +q037,Create a Zendesk ticket,verb_product,zendesk.zendesk_get_ticket,0.0,1.0,1.0,0.3333,0.5000,3,1,257,1.0,1,368,1.0,1,875,1.0,zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_create_ticket zendesk.zendesk_search zendesk.zendesk_list_tickets jira.jira_create_issue slack.create_file linear.create_comment github.create_milestone linear.create_user +q038,Create a HubSpot contact,verb_product,hubspot.hubspot_get_contact,0.0,1.0,1.0,0.5000,0.6309,2,1,255,1.0,1,364,1.0,1,899,1.0,hubspot.hubspot_get_contact hubspot.hubspot_create_contact hubspot.hubspot_create_deal hubspot.hubspot_list_contacts hubspot.hubspot_search_contacts github.create_repository github.create_gist github.create_branch github.create_issue github.create_or_update_file +q039,Create a Salesforce lead,verb_product,salesforce.salesforce_convert_lead,0.0,1.0,1.0,0.5000,0.6309,2,1,252,1.0,1,416,1.0,1,796,1.0,salesforce.salesforce_convert_lead salesforce.salesforce_create_lead salesforce.salesforce_create_record salesforce.salesforce_delete_record salesforce.salesforce_update_record salesforce.salesforce_get_record salesforce.salesforce_list_opportunities salesforce.salesforce_query slack.create_file linear.create_comment +q040,Run a SOQL query,identifier,salesforce.salesforce_query,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,310,1.0,1,662,1.0,salesforce.salesforce_query github.get_check_run github.delete_check_run github.update_check_run sqlite.sqlite_query postgres.query github.create_check_run mysql.mysql_query docker.docker_run prometheus.prom_query +q041,List Shopify orders,verb_product,shopify.shopify_list_orders,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,338,1.0,1,662,1.0,shopify.shopify_list_orders shopify.shopify_list_customers shopify.shopify_list_products shopify.shopify_get_product shopify.shopify_update_product shopify.shopify_get_order shopify.shopify_create_product shopify.shopify_create_order jira.jira_list_projects slack.list_files +q042,Create a PagerDuty incident,verb_product,pagerduty.pagerduty_get_incident,0.0,1.0,1.0,0.5000,0.6309,2,1,255,1.0,1,403,1.0,1,857,1.0,pagerduty.pagerduty_get_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls slack.create_file linear.create_comment github.create_milestone linear.create_user +q043,Who is on call in PagerDuty,paraphrase,pagerduty.pagerduty_list_oncalls,1.0,1.0,1.0,1.0000,1.0000,1,1,227,1.0,1,419,1.0,1,772,1.0,pagerduty.pagerduty_list_oncalls pagerduty.pagerduty_get_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_create_incident twilio.twilio_make_call pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident slack.slack_create_channel filesystem.list_allowed_directories github.update_issue_comment +q044,Query Datadog metrics,verb_product,datadog.datadog_query_metrics,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,458,1.0,1,804,1.0,datadog.datadog_query_metrics datadog.datadog_list_monitors datadog.datadog_create_monitor datadog.datadog_list_logs datadog.datadog_get_monitor prometheus.prom_list_metrics prometheus.prom_query notion.notion_query_database prometheus.prom_query_range salesforce.salesforce_query +q045,Search Datadog logs,verb_product,datadog.datadog_list_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,458,1.0,1,824,1.0,datadog.datadog_list_logs datadog.datadog_list_monitors datadog.datadog_get_monitor datadog.datadog_create_monitor datadog.datadog_query_metrics kubernetes.k8s_pod_logs docker.docker_logs github.actions_get_job_logs github.search_users git.git_log +q046,List S3 buckets,verb_product,aws_s3.s3_list_buckets,1.0,1.0,1.0,1.0000,1.0000,1,1,170,1.0,1,312,1.0,1,616,1.0,aws_s3.s3_list_buckets jira.jira_list_projects slack.list_files github.list_labels linear.list_users gitlab.list_labels gitlab.gitlab_list_projects notion.notion_list_users linear.list_labels github.projects_list +q047,Upload an object to S3,paraphrase,aws_s3.s3_put_object,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,437,1.0,1,862,1.0,aws_s3.s3_put_object aws_s3.s3_delete_object aws_s3.s3_get_object slack.slack_upload_file github.assign_copilot_to_issue gmail.gmail_send_email twilio.twilio_send_sms slack.slack_add_reaction slack.slack_reply_to_thread jira.jira_update_issue +q048,Purge Cloudflare cache,verb_product,cloudflare.cf_purge_cache,1.0,1.0,1.0,1.0000,1.0000,1,1,231,1.0,1,340,1.0,1,746,1.0,cloudflare.cf_purge_cache cloudflare.cf_list_zones cloudflare.cf_create_dns cloudflare.cf_list_dns +q049,Create a Cloudflare DNS record,verb_product,cloudflare.cf_create_dns,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,360,1.0,1,757,1.0,cloudflare.cf_create_dns cloudflare.cf_list_dns salesforce.salesforce_create_record cloudflare.cf_list_zones salesforce.salesforce_delete_record salesforce.salesforce_update_record salesforce.salesforce_get_record cloudflare.cf_purge_cache git.git_commit slack.create_file +q050,List Vercel deployments,verb_product,vercel.vercel_list_deployments,1.0,1.0,1.0,1.0000,1.0000,1,1,173,1.0,1,291,1.0,1,618,1.0,vercel.vercel_list_deployments kubernetes.k8s_list_deployments vercel.vercel_list_projects vercel.vercel_create_project vercel.vercel_get_deployment jira.jira_list_projects slack.list_files github.list_labels linear.list_users gitlab.list_labels +q051,Insert a row into Supabase,verb_product,supabase.supabase_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,203,1.0,1,362,1.0,1,665,1.0,supabase.supabase_insert mongodb.mongo_insert supabase.supabase_list_tables supabase.supabase_update supabase.supabase_invoke_fn supabase.supabase_select playwright.browser_type +q052,Export images from Figma,paraphrase,figma.figma_get_images,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,328,1.0,1,777,1.0,figma.figma_get_images docker.docker_images figma.figma_get_file figma.figma_get_file_nodes figma.figma_get_comments figma.figma_post_comment github.delete_file github.create_branch aws_s3.s3_get_object gitlab.gitlab_get_file +q053,Send an SMS with Twilio,verb_product,twilio.twilio_send_sms,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,409,1.0,1,725,1.0,twilio.twilio_send_sms twilio.twilio_list_messages gmail.gmail_send_email twilio.twilio_make_call discord.discord_send_message puppeteer.puppeteer_click filesystem.list_directory_with_sizes puppeteer.puppeteer_fill puppeteer.puppeteer_screenshot puppeteer.puppeteer_navigate +q054,Create an Okta user,verb_product,okta.okta_create_user,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,329,1.0,1,683,1.0,okta.okta_create_user okta.okta_get_user okta.okta_deactivate_user okta.okta_list_users linear.create_user linear.get_user linear.delete_user linear.update_user slack.slack_get_user_profile github.create_issue +q055,Instant query Prometheus,verb_product,prometheus.prom_query,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,407,1.0,1,680,1.0,prometheus.prom_query prometheus.prom_query_range prometheus.prom_list_metrics notion.notion_query_database datadog.datadog_query_metrics salesforce.salesforce_query sqlite.sqlite_query postgres.query postgres.explain_query mysql.mysql_query +q056,Search Grafana dashboards,verb_product,grafana.grafana_search_dashboards,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,353,1.0,1,636,1.0,grafana.grafana_search_dashboards grafana.grafana_get_dashboard grafana.grafana_create_annotation github.search_users github.search_issues github.search_orgs confluence.confluence_search github.search_commits slack.slack_search_messages linear.linear_search_issues +q057,Trigger a CircleCI pipeline,verb_product,circleci.circleci_trigger_pipeline,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,325,1.0,1,755,1.0,circleci.circleci_trigger_pipeline circleci.circleci_get_pipeline gitlab.gitlab_create_pipeline circleci.circleci_list_pipelines gitlab.get_pipeline gitlab.delete_pipeline gitlab.update_pipeline gitlab.create_pipeline github.actions_run_trigger hubspot.hubspot_create_deal +q058,Run terraform plan,identifier,terraform.tf_plan,1.0,1.0,1.0,1.0000,1.0000,1,1,172,1.0,1,280,1.0,1,684,1.0,terraform.tf_plan terraform.tf_apply terraform.tf_state_list github.get_check_run github.delete_check_run github.update_check_run postgres.explain_query github.create_check_run docker.docker_run github.actions_get +q059,Create an Asana task,verb_product,asana.asana_get_task,0.0,1.0,1.0,0.5000,0.6309,2,1,267,1.0,1,405,1.0,1,825,1.0,asana.asana_get_task asana.asana_create_task asana.asana_update_task asana.asana_create_project asana.asana_list_tasks clickup.clickup_create_task clickup.clickup_get_task clickup.clickup_update_task github.create_issue okta.okta_create_user +q060,Create a Trello card,verb_product,trello.trello_create_card,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,413,1.0,1,903,1.0,trello.trello_create_card trello.trello_add_comment trello.trello_create_board trello.trello_list_cards slack.create_file linear.create_comment github.create_milestone linear.create_user notion.create_comment gitlab.create_label +q061,Create a ClickUp task,verb_product,clickup.clickup_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,362,1.0,1,770,1.0,clickup.clickup_create_task clickup.clickup_get_task clickup.clickup_update_task clickup.clickup_list_tasks asana.asana_get_task asana.asana_create_task asana.asana_update_task playwright.browser_click puppeteer.puppeteer_click slack.create_file +q062,Create a Confluence page,verb_product,confluence.confluence_get_page,0.0,1.0,1.0,0.5000,0.6309,2,1,260,1.0,1,367,1.0,1,755,1.0,confluence.confluence_get_page confluence.confluence_create_page confluence.confluence_update_page confluence.confluence_search notion.notion_get_page notion.notion_create_page notion.notion_update_page playwright.browser_take_screenshot notion.notion_get_comments github.list_commits +q063,Send a Discord message,verb_product,discord.discord_send_message,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,322,1.0,1,740,1.0,discord.discord_send_message discord.discord_add_reaction discord.discord_get_channel discord.discord_list_channels discord.discord_list_members gmail.gmail_send_email twilio.twilio_send_sms slack.slack_post_message github.delete_file gmail.gmail_get_email +q064,Search Elasticsearch,verb_product,elasticsearch.es_search,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,375,1.0,1,658,1.0,elasticsearch.es_search elasticsearch.es_get_doc elasticsearch.es_index_doc github.search_users github.search_issues github.search_orgs github.search_commits confluence.confluence_search slack.slack_search_messages linear.linear_search_issues +q065,Get a Redis key,verb_product,redis.redis_get,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,281,1.0,1,589,1.0,redis.redis_get redis.redis_set redis.redis_delete redis.redis_keys jira.jira_get_issue playwright.browser_press_key figma.figma_get_file aws_s3.s3_get_object figma.figma_get_comments figma.figma_get_file_nodes +q066,Find MongoDB documents,verb_product,mongodb.mongo_find,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,381,1.0,1,738,1.0,mongodb.mongo_find mongodb.mongo_update mongodb.mongo_delete mongodb.mongo_insert mongodb.mongo_list_collections +q067,List SQLite tables,verb_product,sqlite.sqlite_list_tables,1.0,1.0,1.0,1.0000,1.0000,1,1,147,1.0,1,245,1.0,1,543,1.0,sqlite.sqlite_list_tables sqlite.sqlite_query supabase.supabase_list_tables sqlite.sqlite_describe_table postgres.list_tables mysql.mysql_list_tables jira.jira_list_projects slack.list_files github.list_labels linear.list_users +q068,Describe a postgres table,verb_product,postgres.describe_table,1.0,1.0,1.0,1.0000,1.0000,1,1,215,1.0,1,349,1.0,1,597,1.0,postgres.describe_table sqlite.sqlite_describe_table supabase.supabase_update supabase.supabase_select supabase.supabase_insert postgres.list_schemas postgres.list_tables postgres.query postgres.explain_query +q069,github_create_issue,identifier,gitlab.gitlab_create_issue,0.0,1.0,1.0,0.5000,0.5940,2,1,347,1.0,1,511,1.0,1,931,1.0,gitlab.gitlab_create_issue github.create_issue github.add_issue_comment bitbucket.bitbucket_create_issue gitlab.gitlab_get_issue linear.linear_create_comment jira.jira_get_issue github.issue_read github.issue_write jira.jira_update_issue +q070,createIssue,identifier,gitlab.gitlab_create_issue,0.0,1.0,1.0,0.5000,0.4992,2,1,332,1.0,1,472,1.0,1,880,1.0,gitlab.gitlab_create_issue github.create_issue bitbucket.bitbucket_create_issue gitlab.gitlab_get_issue linear.linear_create_comment jira.jira_get_issue jira.jira_update_issue sentry.sentry_get_issue jira.jira_create_issue github.add_issue_comment +q071,list-directory,identifier,filesystem.create_directory,0.0,1.0,1.0,0.5000,0.6309,2,1,182,1.0,1,348,1.0,1,654,1.0,filesystem.create_directory filesystem.list_directory filesystem.get_file_info filesystem.list_directory_with_sizes github.get_file_contents filesystem.directory_tree jira.jira_list_projects slack.list_files github.list_labels linear.list_users +q072,sequential thinking next thought,paraphrase,sequential_thinking.sequentialthinking,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,382,1.0,1,797,1.0,sequential_thinking.sequentialthinking +q073,What time is it in Tokyo,paraphrase,time.get_current_time,1.0,1.0,1.0,1.0000,1.0000,1,1,222,1.0,1,378,1.0,1,699,1.0,time.get_current_time time.convert_time slack.slack_create_channel playwright.browser_wait_for google_calendar.gcal_list_events filesystem.list_allowed_directories grafana.grafana_create_annotation jira.jira_add_worklog slack.get_file slack.delete_file +q074,Convert time between timezones,verb_product,time.convert_time,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,360,1.0,1,704,1.0,time.convert_time salesforce.salesforce_convert_lead time.get_current_time playwright.browser_wait_for google_calendar.gcal_list_events memory.create_relations git.git_diff grafana.grafana_create_annotation jira.jira_add_worklog +q075,Star a GitHub repo,verb_product,github.star_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,354,1.0,1,733,1.0,github.star_repository github.delete_repository github.get_label git.git_branch github.list_tags github.list_labels github.list_branches github.search_issues github.get_tag github.actions_list +q076,Fork a repository,resource,github.fork_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,248,1.0,1,386,1.0,1,825,1.0,github.fork_repository github.delete_repository github.create_repository github.unstar_repository github.star_repository github.list_repository_collaborators github.get_repository_tree github.list_labels github.list_branches github.delete_file +q077,Get the latest GitHub release,verb_product,github.get_latest_release,1.0,1.0,1.0,1.0000,1.0000,1,1,201,1.0,1,361,1.0,1,753,1.0,github.get_latest_release gitlab.get_release gitlab.delete_release gitlab.update_release github.get_release_by_tag gitlab.create_release github.get_teams github.get_repository_tree github.get_label github.projects_get +q078,Create a gist,resource,github.get_gist,0.0,1.0,1.0,0.3333,0.5000,3,1,229,1.0,1,421,1.0,1,911,1.0,github.get_gist github.update_gist github.create_gist slack.create_file linear.create_comment github.create_milestone linear.create_user notion.create_comment gitlab.create_label jira.create_filter +q079,Search GitHub code,verb_product,github.search_code,1.0,1.0,1.0,1.0000,1.0000,1,1,283,1.0,1,419,1.0,1,725,1.0,github.search_code github.list_code_scanning_alerts github.get_code_scanning_alert github.search_users github.search_issues github.search_orgs github.search_commits github.search_pull_requests github.search_repositories hubspot.hubspot_search_contacts +q080,List GitHub notifications,verb_product,github.list_notifications,1.0,1.0,1.0,1.0000,1.0000,1,1,193,1.0,1,328,1.0,1,687,1.0,github.list_notifications github.mark_all_notifications_read github.list_tags github.list_labels github.projects_list github.list_branches git.git_branch github.actions_list github.list_releases github.list_issues +q081,Assign Copilot to an issue,verb_product,github.assign_copilot_to_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,434,1.0,1,875,1.0,github.assign_copilot_to_issue linear.linear_assign_issue jira.jira_assign_issue github.request_copilot_review jira.jira_update_issue github.add_issue_comment linear.linear_create_comment github.create_issue jira.jira_add_comment jira.jira_transition_issue +q082,Get secret scanning alerts,resource,github.list_secret_scanning_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,295,1.0,1,470,1.0,1,733,1.0,github.list_secret_scanning_alerts github.get_secret_scanning_alert github.list_code_scanning_alerts github.list_dependabot_alerts github.get_code_scanning_alert slack.get_file linear.get_comment github.get_milestone linear.get_user notion.get_comment +q083,Create or update a file on GitHub,verb_product,github.create_or_update_file,1.0,1.0,1.0,1.0000,0.7602,1,1,341,1.0,1,545,1.0,1,1020,1.0,github.create_or_update_file slack.update_file github.update_issue_comment github.get_file_contents slack.create_file filesystem.write_file github.projects_write github.label_write github.delete_file google_drive.gdrive_update_file +q084,List collaborators on a repo,resource,github.list_repository_collaborators,1.0,1.0,1.0,1.0000,1.0000,1,1,307,1.0,1,445,1.0,1,962,1.0,github.list_repository_collaborators github.list_commits github.update_issue_comment pagerduty.pagerduty_list_oncalls trello.trello_list_cards github.get_discussion_comments github.create_pull_request github.request_copilot_review github.assign_copilot_to_issue github.list_labels +q085,Reply in a Slack thread,verb_product,slack.slack_reply_to_thread,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,346,1.0,1,702,1.0,slack.slack_reply_to_thread slack.slack_get_thread_replies github.get_notification_details slack.get_file slack.delete_file slack.update_file slack.create_file github.dismiss_notification slack.list_files slack.slack_list_users +q086,Add an emoji reaction on Slack,verb_product,slack.slack_add_reaction,1.0,1.0,1.0,1.0000,1.0000,1,1,268,1.0,1,467,1.0,1,812,1.0,slack.slack_add_reaction discord.discord_add_reaction jira.jira_add_comment github.add_issue_comment gmail.gmail_modify_email trello.trello_add_comment git.git_add playwright.browser_click memory.add_observations jira.jira_add_worklog +q087,Search Slack messages,verb_product,slack.slack_search_messages,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,334,1.0,1,632,1.0,slack.slack_search_messages gmail.gmail_search_emails twilio.twilio_list_messages slack.slack_get_channel_history github.search_users github.search_issues github.search_orgs slack.get_file github.search_commits confluence.confluence_search +q088,Create a Slack channel,verb_product,slack.slack_create_channel,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,339,1.0,1,718,1.0,slack.slack_create_channel slack.slack_join_channel slack.slack_leave_channel slack.slack_get_channel_history slack.slack_post_message slack.slack_set_channel_topic discord.discord_get_channel slack.slack_upload_file slack.create_file slack.slack_get_thread_replies +q089,Transition a Jira issue,verb_product,jira.jira_transition_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,207,1.0,1,342,1.0,1,781,1.0,jira.jira_transition_issue jira.jira_get_issue jira.jira_update_issue jira.jira_assign_issue jira.jira_add_comment jira.jira_create_issue jira.jira_add_worklog gitlab.gitlab_get_issue sentry.sentry_get_issue gitlab.gitlab_create_issue +q090,Search Jira with JQL,identifier,jira.jira_search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,202,1.0,1,322,1.0,1,605,1.0,jira.jira_search_issues filesystem.list_directory_with_sizes github.search_users github.search_issues jira.jira_list_projects github.search_orgs github.search_commits confluence.confluence_search slack.slack_search_messages linear.linear_search_issues +q091,Add a worklog to Jira,verb_product,jira.jira_add_worklog,1.0,1.0,1.0,1.0000,1.0000,1,1,285,1.0,1,406,1.0,1,839,1.0,jira.jira_add_worklog jira.jira_add_comment github.add_issue_comment git.git_add memory.add_observations linear.linear_create_comment github.add_comment_to_pending_review discord.discord_add_reaction slack.slack_add_reaction jira.jira_list_projects +q092,List Linear teams,verb_product,linear.linear_list_teams,1.0,1.0,1.0,1.0000,1.0000,1,1,137,1.0,1,283,1.0,1,639,1.0,linear.linear_list_teams github.get_teams linear.linear_list_projects linear.list_users linear.list_labels linear.list_comments linear.list_attachments linear.get_comment linear.linear_create_project linear.list_cycles +q093,Comment on a Linear issue,verb_product,linear.linear_create_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,360,1.0,1,798,1.0,linear.linear_create_comment github.update_issue_comment linear.get_comment linear.delete_comment linear.update_comment linear.create_comment github.add_issue_comment jira.jira_add_comment linear.linear_get_issue linear.linear_update_issue +q094,Move a local file,paraphrase,filesystem.move_file,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,358,1.0,1,684,1.0,filesystem.move_file filesystem.read_file filesystem.read_text_file filesystem.read_media_file brave_search.brave_local_search slack.get_file gmail.gmail_trash_email slack.delete_file slack.update_file slack.create_file +q095,Recursively search files by pattern,paraphrase,filesystem.search_files,1.0,1.0,1.0,1.0000,1.0000,1,1,192,1.0,1,322,1.0,1,716,1.0,filesystem.search_files redis.redis_keys linear.linear_search_issues google_drive.gdrive_search slack.list_files github.search_repositories github.search_users github.push_files github.search_issues github.search_orgs +q096,Edit a file in place with old and new string,paraphrase,filesystem.edit_file,1.0,1.0,1.0,1.0000,1.0000,1,1,247,1.0,1,353,1.0,1,735,1.0,filesystem.edit_file filesystem.list_directory_with_sizes filesystem.write_file slack.get_file slack.delete_file playwright.browser_tab_new slack.update_file twilio.twilio_make_call slack.create_file github.create_repository +q097,Read the whole knowledge graph,paraphrase,memory.read_graph,1.0,1.0,1.0,1.0000,1.0000,1,1,152,1.0,1,260,1.0,1,561,1.0,memory.read_graph memory.create_entities memory.delete_relations memory.create_relations memory.search_nodes filesystem.read_file memory.open_nodes memory.delete_entities memory.add_observations filesystem.read_text_file +q098,Search memory nodes,verb_product,memory.search_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,202,1.0,1,338,1.0,1,621,1.0,memory.search_nodes memory.open_nodes figma.figma_get_file_nodes github.search_users github.search_issues github.search_orgs github.search_commits confluence.confluence_search slack.slack_search_messages linear.linear_search_issues +q099,Show unstaged git diff,verb_product,git.git_diff_unstaged,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,303,1.0,1,621,1.0,git.git_diff_unstaged git.git_diff git.git_diff_staged git.git_show git.git_status git.git_log git.git_branch git.git_checkout git.git_commit git.git_create_branch +q100,Stage files with git add,identifier,git.git_add,1.0,1.0,1.0,1.0000,1.0000,1,1,220,1.0,1,377,1.0,1,774,1.0,git.git_add filesystem.list_directory_with_sizes slack.list_files salesforce.salesforce_list_opportunities github.add_issue_comment jira.jira_add_comment github.push_files memory.add_observations jira.jira_add_worklog filesystem.search_files +q101,Create a Shopify product,verb_product,shopify.shopify_get_product,0.0,1.0,1.0,0.3333,0.5000,3,1,240,1.0,1,377,1.0,1,702,1.0,shopify.shopify_get_product shopify.shopify_update_product shopify.shopify_create_product stripe.stripe_create_product shopify.shopify_create_order shopify.shopify_get_order shopify.shopify_list_customers shopify.shopify_list_orders shopify.shopify_list_products stripe.stripe_create_price +q102,Cancel a Stripe subscription,verb_product,stripe.stripe_cancel_subscription,1.0,1.0,1.0,1.0000,1.0000,1,1,206,1.0,1,321,1.0,1,610,1.0,stripe.stripe_cancel_subscription stripe.stripe_create_product stripe.stripe_create_customer stripe.stripe_list_customers stripe.stripe_get_customer stripe.stripe_list_invoices stripe.stripe_create_invoice stripe.stripe_list_subscriptions stripe.stripe_get_payment_intent stripe.stripe_create_refund +q103,Create a payment intent,resource,stripe.stripe_get_payment_intent,0.0,1.0,1.0,0.5000,0.6309,2,1,234,1.0,1,426,1.0,1,916,1.0,stripe.stripe_get_payment_intent stripe.stripe_create_payment_intent stripe.stripe_create_refund slack.create_file linear.create_comment github.create_milestone linear.create_user notion.create_comment gitlab.create_label jira.create_filter +q104,Acknowledge a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,261,1.0,1,405,1.0,1,717,1.0,pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls +q105,Create a Datadog monitor,verb_product,datadog.datadog_get_monitor,0.0,1.0,1.0,0.5000,0.6309,2,1,259,1.0,1,459,1.0,1,948,1.0,datadog.datadog_get_monitor datadog.datadog_create_monitor datadog.datadog_list_monitors datadog.datadog_list_logs datadog.datadog_query_metrics slack.create_file linear.create_comment github.create_milestone linear.create_user notion.create_comment +q106,Resolve a Sentry issue,paraphrase,sentry.sentry_get_issue,0.0,1.0,1.0,0.5000,0.6309,2,1,251,1.0,1,368,1.0,1,745,1.0,sentry.sentry_get_issue sentry.sentry_update_issue sentry.sentry_list_events pagerduty.pagerduty_resolve_incident sentry.sentry_list_projects sentry.sentry_list_issues gitlab.gitlab_get_issue jira.jira_get_issue jira.jira_update_issue gitlab.gitlab_create_issue +q107,Scale a Kubernetes deployment,verb_product,kubernetes.k8s_scale_deployment,1.0,1.0,1.0,1.0000,1.0000,1,1,230,1.0,1,374,1.0,1,726,1.0,kubernetes.k8s_scale_deployment vercel.vercel_get_deployment kubernetes.k8s_get_events kubernetes.k8s_list_deployments kubernetes.k8s_delete kubernetes.k8s_get_pod kubernetes.k8s_list_services kubernetes.k8s_list_pods kubernetes.k8s_apply kubernetes.k8s_pod_logs +q108,Apply a Kubernetes manifest,verb_product,kubernetes.k8s_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,344,1.0,1,727,1.0,kubernetes.k8s_apply terraform.tf_apply kubernetes.k8s_get_events kubernetes.k8s_list_deployments kubernetes.k8s_delete kubernetes.k8s_get_pod kubernetes.k8s_list_services kubernetes.k8s_list_pods kubernetes.k8s_pod_logs kubernetes.k8s_scale_deployment +q109,Stop a docker container,verb_product,docker.docker_stop,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,345,1.0,1,641,1.0,docker.docker_stop docker.docker_rm docker.docker_logs docker.docker_exec docker.docker_run docker.docker_images docker.docker_ps docker.docker_build kubernetes.k8s_pod_logs +q110,Build a docker image,verb_product,docker.docker_build,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,303,1.0,1,601,1.0,docker.docker_build docker.docker_run docker.docker_images docker.docker_ps docker.docker_stop docker.docker_rm docker.docker_logs docker.docker_exec filesystem.read_media_file +q111,Click an element in Playwright,verb_product,playwright.browser_click,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,329,1.0,1,654,1.0,playwright.browser_click puppeteer.puppeteer_click playwright.browser_type playwright.browser_take_screenshot playwright.browser_snapshot clickup.clickup_create_task clickup.clickup_list_tasks clickup.clickup_get_task clickup.clickup_update_task playwright.browser_tab_list +q112,Type text into a browser input,paraphrase,playwright.browser_type,1.0,1.0,1.0,1.0000,0.9502,1,1,235,1.0,1,337,1.0,1,647,1.0,playwright.browser_type playwright.browser_wait_for puppeteer.puppeteer_fill github.projects_list playwright.browser_tab_list filesystem.read_text_file playwright.browser_navigate playwright.browser_tab_new playwright.browser_tab_close mongodb.mongo_insert +q113,List GitLab projects,verb_product,gitlab.gitlab_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,171,1.0,1,255,1.0,1,609,1.0,gitlab.gitlab_list_projects github.projects_list jira.jira_list_projects linear.linear_list_projects vercel.vercel_list_projects sentry.sentry_list_projects gitlab.gitlab_list_issues github.projects_get gitlab.gitlab_create_issue gitlab.gitlab_list_merge_requests +q114,Create a Bitbucket pull request,verb_product,bitbucket.bitbucket_create_pr,1.0,1.0,1.0,1.0000,1.0000,1,1,313,1.0,1,508,1.0,1,1091,1.0,bitbucket.bitbucket_create_pr bitbucket.bitbucket_list_prs bitbucket.bitbucket_create_issue github.pull_request_read github.merge_pull_request bitbucket.bitbucket_list_repos github.update_pull_request_branch github.update_pull_request github.create_pull_request github.pull_request_review_write +q115,Create a Gmail draft,verb_product,gmail.gmail_create_draft,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,315,1.0,1,844,1.0,gmail.gmail_create_draft gmail.gmail_list_labels gmail.gmail_get_email gmail.gmail_search_emails gmail.gmail_trash_email gmail.gmail_send_email github.create_pull_request gmail.gmail_modify_email slack.create_file linear.create_comment +q116,Share a Google Drive file,verb_product,google_drive.gdrive_share_file,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,417,1.0,1,684,1.0,google_drive.gdrive_share_file google_drive.gdrive_update_file google_drive.gdrive_get_file google_drive.gdrive_create_file google_drive.gdrive_search google_drive.gdrive_list_folder slack.get_file google_calendar.gcal_list_calendars slack.delete_file slack.update_file +q117,List Google Calendar events,verb_product,google_calendar.gcal_list_events,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,427,1.0,1,753,1.0,google_calendar.gcal_list_events google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_list_calendars google_calendar.gcal_create_event kubernetes.k8s_get_events sentry.sentry_list_events google_drive.gdrive_search google_drive.gdrive_list_folder google_drive.gdrive_update_file +q118,Delete a calendar event,resource,google_calendar.gcal_delete_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,427,1.0,1,796,1.0,google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars slack.delete_file github.delete_repository github.pull_request_review_write linear.delete_comment github.delete_milestone +q119,List Vercel projects,verb_product,vercel.vercel_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,175,1.0,1,269,1.0,1,550,1.0,vercel.vercel_list_projects vercel.vercel_create_project vercel.vercel_list_deployments jira.jira_list_projects vercel.vercel_get_deployment gitlab.gitlab_list_projects github.projects_list linear.linear_list_projects sentry.sentry_list_projects github.projects_get +q120,Select rows from Supabase,verb_product,supabase.supabase_select,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,354,1.0,1,815,1.0,supabase.supabase_select supabase.supabase_insert supabase.supabase_update supabase.supabase_list_tables supabase.supabase_invoke_fn github.delete_file github.create_branch aws_s3.s3_get_object gitlab.gitlab_get_file github.get_tag +q121,Post a Figma comment,verb_product,figma.figma_post_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,350,1.0,1,658,1.0,figma.figma_post_comment figma.figma_get_file slack.slack_post_message figma.figma_get_comments figma.figma_get_file_nodes figma.figma_get_images linear.get_comment notion.get_comment linear.delete_comment linear.update_comment +q122,List Okta users,verb_product,okta.okta_list_users,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,306,1.0,1,606,1.0,okta.okta_list_users okta.okta_get_user okta.okta_create_user okta.okta_deactivate_user linear.list_users notion.notion_list_users slack.slack_list_users slack.slack_get_users github.search_users notion.list_users_tokens +q123,Range query Prometheus,verb_product,prometheus.prom_query_range,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,407,1.0,1,680,1.0,prometheus.prom_query_range prometheus.prom_query prometheus.prom_list_metrics notion.notion_query_database datadog.datadog_query_metrics salesforce.salesforce_query sqlite.sqlite_query postgres.query postgres.explain_query mysql.mysql_query +q124,Create a Grafana annotation,verb_product,grafana.grafana_create_annotation,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,410,1.0,1,900,1.0,grafana.grafana_create_annotation grafana.grafana_search_dashboards grafana.grafana_get_dashboard slack.create_file linear.create_comment github.create_milestone linear.create_user notion.create_comment gitlab.create_label jira.create_filter +q125,Apply terraform,identifier,terraform.tf_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,292,1.0,1,698,1.0,terraform.tf_apply kubernetes.k8s_apply terraform.tf_plan terraform.tf_state_list +q126,List Asana tasks,verb_product,asana.asana_list_tasks,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,368,1.0,1,708,1.0,asana.asana_list_tasks clickup.clickup_list_tasks asana.asana_create_project asana.asana_get_task asana.asana_create_task asana.asana_update_task jira.jira_list_projects slack.list_files github.list_labels linear.list_users +q127,Comment on a Trello card,verb_product,trello.trello_add_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,241,1.0,1,422,1.0,1,695,1.0,trello.trello_add_comment trello.trello_create_card trello.trello_create_board trello.trello_list_cards github.update_issue_comment linear.get_comment notion.get_comment linear.delete_comment linear.update_comment notion.delete_comment +q128,Update a ClickUp task,verb_product,clickup.clickup_update_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,399,1.0,1,740,1.0,clickup.clickup_update_task clickup.clickup_get_task clickup.clickup_create_task asana.asana_update_task clickup.clickup_list_tasks asana.asana_get_task playwright.browser_click puppeteer.puppeteer_click asana.asana_create_task slack.update_file +q129,Search Confluence,verb_product,confluence.confluence_search,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,362,1.0,1,677,1.0,confluence.confluence_search confluence.confluence_get_page confluence.confluence_update_page confluence.confluence_create_page github.search_users github.search_issues github.search_orgs github.search_commits slack.slack_search_messages linear.linear_search_issues +q130,Create a HubSpot deal,verb_product,hubspot.hubspot_create_deal,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,899,1.0,hubspot.hubspot_create_deal hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_list_contacts hubspot.hubspot_search_contacts github.create_repository github.create_gist github.create_branch github.create_issue github.create_or_update_file +q131,List Zendesk tickets,verb_product,zendesk.zendesk_list_tickets,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,367,1.0,1,694,1.0,zendesk.zendesk_list_tickets zendesk.zendesk_search zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_create_ticket jira.jira_list_projects slack.list_files github.list_labels linear.list_users gitlab.list_labels +q132,List Discord channels,verb_product,discord.discord_list_channels,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,300,1.0,1,639,1.0,discord.discord_list_channels slack.slack_list_channels discord.discord_get_channel discord.discord_list_members discord.discord_send_message discord.discord_add_reaction jira.jira_list_projects slack.list_files github.list_labels linear.list_users +q133,Set a Redis key,verb_product,redis.redis_set,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,302,1.0,1,595,1.0,redis.redis_set redis.redis_get redis.redis_delete redis.redis_keys slack.slack_set_channel_topic playwright.browser_press_key jira.jira_get_issue jira.jira_update_issue aws_s3.s3_delete_object figma.figma_get_file +q134,Insert MongoDB documents,verb_product,mongodb.mongo_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,256,1.0,1,381,1.0,1,698,1.0,mongodb.mongo_insert mongodb.mongo_update mongodb.mongo_delete mongodb.mongo_find mongodb.mongo_list_collections supabase.supabase_insert +q135,Run a MySQL query,verb_product,mysql.mysql_query,1.0,1.0,1.0,1.0000,1.0000,1,1,179,1.0,1,281,1.0,1,619,1.0,mysql.mysql_query postgres.query sqlite.sqlite_query mysql.mysql_list_tables postgres.explain_query github.get_check_run github.delete_check_run github.update_check_run salesforce.salesforce_query github.create_check_run +q136,Index a document in Elasticsearch,verb_product,elasticsearch.es_index_doc,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,344,1.0,1,607,1.0,elasticsearch.es_index_doc elasticsearch.es_get_doc elasticsearch.es_search playwright.browser_tab_close slack.get_file slack.delete_file linear.get_comment github.get_milestone linear.get_user notion.get_comment +q137,Get an S3 object,verb_product,aws_s3.s3_get_object,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,355,1.0,1,742,1.0,aws_s3.s3_get_object aws_s3.s3_delete_object aws_s3.s3_put_object okta.okta_get_user asana.asana_get_task jira.jira_update_issue github.create_issue asana.asana_create_project okta.okta_create_user elasticsearch.es_search +q138,List Cloudflare zones,verb_product,cloudflare.cf_list_zones,1.0,1.0,1.0,1.0000,1.0000,1,1,215,1.0,1,319,1.0,1,663,1.0,cloudflare.cf_list_zones cloudflare.cf_list_dns cloudflare.cf_create_dns cloudflare.cf_purge_cache jira.jira_list_projects slack.list_files github.list_labels linear.list_users gitlab.list_labels gitlab.gitlab_list_projects +q139,Create a Stripe invoice,verb_product,stripe.stripe_create_invoice,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,353,1.0,1,669,1.0,stripe.stripe_create_invoice stripe.stripe_create_product stripe.stripe_create_customer stripe.stripe_create_refund stripe.stripe_create_price stripe.stripe_list_customers stripe.stripe_create_payment_intent stripe.stripe_get_customer stripe.stripe_list_invoices stripe.stripe_list_subscriptions +q140,List GitHub Actions workflows,verb_product,github.actions_list,1.0,1.0,1.0,1.0000,1.0000,1,1,268,1.0,1,457,1.0,1,782,1.0,github.actions_list github.actions_get github.actions_get_job_logs github.actions_run_trigger github.list_tags github.list_labels github.projects_list github.list_branches git.git_branch github.list_releases +q141,Get GitHub Actions job logs,verb_product,github.actions_get_job_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,377,1.0,1,822,1.0,github.actions_get_job_logs gitlab.get_job github.actions_list github.actions_get gitlab.delete_job gitlab.update_job gitlab.create_job github.actions_run_trigger kubernetes.k8s_pod_logs docker.docker_logs +q142,Create a GitHub repository,verb_product,github.create_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,235,1.0,1,406,1.0,1,837,1.0,github.create_repository github.delete_repository github.star_repository github.unstar_repository github.create_branch github.create_issue github.fork_repository github.list_repository_collaborators github.list_labels github.list_tags +q143,Checkout a git branch,verb_product,git.git_checkout,1.0,1.0,1.0,1.0000,1.0000,1,1,266,1.0,1,407,1.0,1,1001,1.0,git.git_checkout github.create_branch git.git_create_branch git.git_branch github.update_pull_request_branch gitlab.gitlab_create_merge_request github.delete_file github.list_commits github.create_or_update_file github.push_files +q144,Show git log,identifier,git.git_log,1.0,1.0,1.0,1.0000,1.0000,1,1,194,1.0,1,299,1.0,1,619,1.0,git.git_log git.git_show git.git_status git.git_diff_staged git.git_diff_unstaged git.git_diff git.git_branch git.git_checkout git.git_commit git.git_create_branch +q145,List postgres schemas,verb_product,postgres.list_schemas,1.0,1.0,1.0,1.0000,1.0000,1,1,158,1.0,1,277,1.0,1,604,1.0,postgres.list_schemas postgres.list_tables postgres.query postgres.explain_query postgres.describe_table jira.jira_list_projects slack.list_files github.list_labels linear.list_users gitlab.list_labels +q146,Open nodes in the knowledge graph,verb_product,memory.open_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,163,1.0,1,260,1.0,1,531,1.0,memory.open_nodes memory.search_nodes memory.read_graph memory.create_entities memory.delete_relations memory.create_relations memory.add_observations memory.delete_entities figma.figma_get_file_nodes playwright.browser_tab_list +q147,Get file metadata from the filesystem,paraphrase,filesystem.get_file_info,1.0,1.0,1.0,1.0000,1.0000,1,1,220,1.0,1,334,1.0,1,769,1.0,filesystem.get_file_info filesystem.read_file filesystem.read_text_file google_drive.gdrive_get_file filesystem.read_media_file gitlab.gitlab_get_file slack.get_file github.delete_file filesystem.write_file github.get_file_contents +q148,Write a new local file,paraphrase,filesystem.write_file,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,428,1.0,1,752,1.0,filesystem.write_file filesystem.read_file filesystem.read_text_file filesystem.read_media_file github.issue_write filesystem.edit_file playwright.browser_tab_new brave_search.brave_local_search slack.get_file slack.delete_file +q149,Search GitHub issues,verb_product,github.search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,263,1.0,1,407,1.0,1,710,1.0,github.search_issues linear.linear_search_issues github.list_issues gitlab.gitlab_list_issues jira.jira_search_issues github.search_users github.search_orgs github.search_commits github.search_pull_requests sentry.sentry_list_issues +q150,Get the authenticated GitHub user,paraphrase,github.get_teams,0.0,1.0,1.0,0.3333,0.5000,3,1,179,1.0,1,284,1.0,1,606,1.0,github.get_teams github.list_notifications github.get_me linear.get_user okta.okta_get_user slack.slack_get_user_profile linear.delete_user linear.update_user linear.create_user okta.okta_deactivate_user diff --git a/experiments/mcp-tool-search/results/bm25_tfidf_hybrid.json b/experiments/mcp-tool-search/results/bm25_tfidf_hybrid.json new file mode 100644 index 00000000..8e6413c3 --- /dev/null +++ b/experiments/mcp-tool-search/results/bm25_tfidf_hybrid.json @@ -0,0 +1,6969 @@ +{ + "summary": { + "n": 150, + "index_s": 0.002145009000059872, + "rss_mb": 0.10546875, + "p50_ms": 0.37560699996674884, + "p95_ms": 0.65002184998093, + "recall@1": 0.82, + "recall@1_lo": 0.76, + "recall@1_hi": 0.88, + "recall@3": 0.9733333333333334, + "recall@3_lo": 0.9466666666666667, + "recall@3_hi": 0.9933333333333333, + "recall@5": 0.9866666666666667, + "recall@5_lo": 0.9666666666666667, + "recall@5_hi": 1.0, + "mrr@10": 0.8919444444444444, + "mrr@10_lo": 0.8538888888888889, + "mrr@10_hi": 0.9272222222222223, + "ndcg@10": 0.913323974786248, + "ndcg@10_lo": 0.8835932038147185, + "ndcg@10_hi": 0.941183384439175 + }, + "meta": { + "name": "bm25_tfidf_hybrid", + "needs_model": false, + "needs_api": false, + "notes": "StackOne-style 20/80 BM25+TF-IDF" + }, + "rows": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "gitlab.gitlab_create_issue", + "github.create_issue", + "github.add_issue_comment", + "github.issue_read", + "gitlab.gitlab_get_issue", + "github.update_issue_comment", + "github.issue_write", + "bitbucket.bitbucket_create_issue", + "linear.linear_create_comment", + "jira.jira_get_issue" + ], + "hit1": "gitlab.gitlab_create_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6063228553749076, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 349, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 521, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 972, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "kind": "verb_product", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_create_issue", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_update_ticket", + "jira.create_filter", + "jira.create_attachment", + "jira.create_board", + "jira.create_sprint", + "jira.create_version", + "jira.create_webhook" + ], + "hit1": "jira.jira_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 281, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 466, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 951, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "kind": "verb_product", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "slack.slack_post_message", + "gmail.gmail_send_email", + "twilio.twilio_send_sms", + "slack.slack_add_reaction", + "slack.get_file", + "discord.discord_add_reaction", + "slack.delete_file", + "slack.slack_get_users", + "slack.slack_list_users" + ], + "hit1": "discord.discord_send_message", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "kind": "resource", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "ranking": [ + "github.list_pull_requests", + "github.search_pull_requests", + "bitbucket.bitbucket_list_prs", + "gitlab.gitlab_list_merge_requests", + "github.create_pull_request", + "memory.open_nodes", + "playwright.browser_tab_list", + "github.pull_request_read", + "github.merge_pull_request", + "github.update_pull_request_branch" + ], + "hit1": "github.list_pull_requests", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 460, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 836, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "ranking": [ + "github.create_pull_request", + "github.request_copilot_review", + "github.pull_request_read", + "github.merge_pull_request", + "github.update_pull_request", + "github.pull_request_review_write", + "github.update_pull_request_branch", + "github.search_pull_requests", + "github.list_pull_requests", + "github.add_comment_to_pending_review" + ], + "hit1": "github.create_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 364, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 611, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1144, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "kind": "verb_product", + "gold": [ + "postgres.query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "mysql.mysql_query", + "sqlite.sqlite_query", + "postgres.explain_query", + "salesforce.salesforce_query", + "postgres.list_schemas", + "postgres.list_tables", + "postgres.describe_table", + "github.get_check_run", + "github.delete_check_run" + ], + "hit1": "postgres.query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 553, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "kind": "paraphrase", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "ranking": [ + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "gitlab.gitlab_get_file", + "github.delete_file", + "memory.read_graph", + "github.issue_read", + "brave_search.brave_local_search", + "slack.get_file", + "github.pull_request_read" + ], + "hit1": "filesystem.read_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 780, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "kind": "paraphrase", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "ranking": [ + "google_drive.gdrive_list_folder", + "slack.list_files", + "github.push_files", + "filesystem.search_files", + "google_drive.gdrive_search", + "github.create_gist", + "github.update_gist", + "filesystem.list_directory", + "git.git_add", + "filesystem.directory_tree" + ], + "hit1": "google_drive.gdrive_list_folder", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.125, + "ndcg@10": 0.2938406966301003, + "hit_rank": 8, + "k3_calls": 3, + "k3_tokens": 712, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 745, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 723, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "ranking": [ + "memory.read_graph", + "memory.create_entities", + "memory.delete_relations", + "memory.create_relations", + "memory.search_nodes", + "memory.open_nodes", + "memory.add_observations", + "memory.delete_entities", + "filesystem.read_file", + "slack.slack_list_channels" + ], + "hit1": "memory.read_graph", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.49320831104542096, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 154, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 544, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "kind": "verb_product", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "ranking": [ + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "jira.jira_search_issues", + "filesystem.list_directory_with_sizes", + "github.search_users", + "memory.search_nodes", + "github.update_pull_request_branch", + "github.search_issues", + "github.search_orgs", + "github.search_commits" + ], + "hit1": "brave_search.brave_web_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 710, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "kind": "verb_product", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "ranking": [ + "fetch.fetch", + "filesystem.read_file", + "playwright.browser_navigate", + "github.get_file_contents", + "puppeteer.puppeteer_navigate", + "git.git_add", + "docker.docker_logs", + "git.git_status", + "playwright.browser_tab_new", + "playwright.browser_snapshot" + ], + "hit1": "fetch.fetch", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 372, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 640, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "kind": "identifier", + "gold": [ + "git.git_status" + ], + "alts": [], + "ranking": [ + "git.git_show", + "git.git_status", + "git.git_diff_staged", + "git.git_log", + "git.git_diff_unstaged", + "git.git_diff", + "git.git_branch", + "sentry.sentry_update_issue", + "slack.list_files", + "linear.list_users" + ], + "hit1": "git.git_show", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "kind": "paraphrase", + "gold": [ + "git.git_commit" + ], + "alts": [], + "ranking": [ + "git.git_commit", + "git.git_reset", + "git.git_diff_staged", + "github.get_commit", + "git.git_show", + "git.git_log", + "github.push_files" + ], + "hit1": "git.git_commit", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_comment", + "linear.linear_get_issue", + "linear.linear_update_issue", + "linear.linear_create_issue", + "linear.linear_assign_issue", + "gitlab.gitlab_create_issue", + "linear.linear_create_project", + "linear.create_comment", + "linear.create_user", + "linear.create_label" + ], + "hit1": "linear.linear_create_comment", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 574, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 921, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "kind": "paraphrase", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_project", + "clickup.clickup_get_task", + "asana.asana_get_task", + "clickup.clickup_update_task", + "asana.asana_update_task", + "clickup.clickup_create_task", + "asana.asana_create_task", + "mysql.mysql_list_tables", + "github.projects_get", + "mysql.mysql_query" + ], + "hit1": "linear.linear_create_project", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 145, + "k3_calls": 49, + "k3_tokens": 12448, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 29, + "k5_tokens": 11769, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11810, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "ranking": [ + "github.update_issue_comment", + "figma.figma_post_comment", + "github.add_issue_comment", + "slack.slack_post_message", + "linear.linear_create_comment", + "jira.jira_add_comment", + "github.assign_copilot_to_issue", + "linear.get_comment", + "notion.get_comment", + "github.issue_read" + ], + "hit1": "github.update_issue_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 307, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 444, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 819, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "kind": "ambiguous", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "ranking": [ + "github.merge_pull_request", + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_get_merge_request", + "github.update_pull_request_branch", + "gitlab.gitlab_create_merge_request", + "github.pull_request_read", + "github.update_pull_request", + "github.request_copilot_review", + "github.pull_request_review_write", + "github.create_pull_request" + ], + "hit1": "github.merge_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 495, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1121, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "kind": "verb_product", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "ranking": [ + "github.actions_run_trigger", + "github.actions_get", + "github.actions_list", + "github.actions_get_job_logs", + "circleci.circleci_trigger_pipeline", + "github.delete_repository", + "github.list_tags", + "github.get_label", + "github.search_users", + "github.projects_get" + ], + "hit1": "github.actions_run_trigger", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 795, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "kind": "resource", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "ranking": [ + "github.list_dependabot_alerts", + "github.get_dependabot_alert", + "github.list_secret_scanning_alerts", + "github.list_code_scanning_alerts", + "slack.get_file", + "linear.get_comment", + "github.get_milestone", + "linear.get_user", + "notion.get_comment", + "gitlab.get_label" + ], + "hit1": "github.list_dependabot_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 420, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "kind": "verb_product", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_issues", + "sentry.sentry_list_projects", + "sentry.sentry_get_issue", + "sentry.sentry_list_events", + "gitlab.gitlab_list_issues", + "github.list_issues", + "sentry.sentry_update_issue", + "github.search_issues", + "linear.linear_search_issues", + "linear.linear_list_issues" + ], + "hit1": "sentry.sentry_list_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 236, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_list_subscriptions", + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_product", + "stripe.stripe_create_refund", + "stripe.stripe_create_price", + "stripe.stripe_list_customers" + ], + "hit1": "stripe.stripe_create_customer", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "kind": "paraphrase", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_refund", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_product", + "stripe.stripe_create_customer", + "stripe.stripe_list_customers", + "stripe.stripe_get_customer", + "stripe.stripe_list_invoices", + "stripe.stripe_create_invoice", + "stripe.stripe_list_subscriptions" + ], + "hit1": "stripe.stripe_create_refund", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "kind": "verb_product", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "ranking": [ + "gmail.gmail_send_email", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "gmail.gmail_list_labels", + "twilio.twilio_send_sms", + "gmail.gmail_modify_email", + "gmail.gmail_create_draft", + "discord.discord_send_message", + "gmail.gmail_search_emails", + "puppeteer.puppeteer_click" + ], + "hit1": "gmail.gmail_send_email", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 711, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "kind": "verb_product", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "ranking": [ + "gmail.gmail_search_emails", + "gmail.gmail_list_labels", + "gmail.gmail_get_email", + "gmail.gmail_create_draft", + "gmail.gmail_trash_email", + "gmail.gmail_send_email", + "mysql.mysql_list_tables", + "gmail.gmail_modify_email", + "mysql.mysql_query", + "github.search_users" + ], + "hit1": "gmail.gmail_search_emails", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 652, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "kind": "paraphrase", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_update_file", + "google_drive.gdrive_get_file", + "slack.slack_upload_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_share_file", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "aws_s3.s3_put_object", + "slack.get_file", + "google_calendar.gcal_list_calendars" + ], + "hit1": "google_drive.gdrive_update_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 523, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 446, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 735, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "github.pull_request_review_write", + "slack.create_file", + "linear.create_comment", + "github.create_milestone", + "linear.create_user" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 962, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_list_pods", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_get_events", + "kubernetes.k8s_list_services", + "kubernetes.k8s_delete", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_apply", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_scale_deployment", + "jira.jira_list_projects" + ], + "hit1": "kubernetes.k8s_list_pods", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_get_pod", + "docker.docker_logs", + "datadog.datadog_list_logs", + "github.actions_get_job_logs", + "git.git_log", + "aws_s3.s3_get_object", + "gitlab.gitlab_get_file", + "github.get_tag", + "elasticsearch.es_get_doc" + ], + "hit1": "kubernetes.k8s_pod_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 834, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "ranking": [ + "docker.docker_run", + "docker.docker_exec", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_images", + "docker.docker_ps", + "docker.docker_build", + "github.get_check_run", + "github.delete_check_run" + ], + "hit1": "docker.docker_run", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "kind": "verb_product", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "ranking": [ + "docker.docker_logs", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_exec", + "docker.docker_run", + "kubernetes.k8s_pod_logs", + "git.git_log", + "docker.docker_images", + "git.git_show", + "docker.docker_ps" + ], + "hit1": "docker.docker_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "kind": "ambiguous", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "ranking": [ + "playwright.browser_navigate", + "puppeteer.puppeteer_navigate", + "playwright.browser_tab_new", + "fetch.fetch", + "playwright.browser_tab_list", + "twilio.twilio_make_call", + "playwright.browser_press_key", + "playwright.browser_click", + "playwright.browser_tab_close", + "playwright.browser_evaluate" + ], + "hit1": "playwright.browser_navigate", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 596, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "kind": "ambiguous", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "ranking": [ + "playwright.browser_take_screenshot", + "puppeteer.puppeteer_screenshot", + "notion.notion_get_page", + "playwright.browser_snapshot", + "notion.notion_update_page", + "confluence.confluence_get_page", + "filesystem.read_file", + "confluence.confluence_update_page", + "notion.notion_create_page", + "git.git_status" + ], + "hit1": "playwright.browser_take_screenshot", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 678, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_get_merge_request", + "gitlab.gitlab_create_merge_request", + "github.merge_pull_request", + "gitlab.gitlab_list_merge_requests", + "gitlab.gitlab_create_issue", + "gitlab.gitlab_create_pipeline", + "github.create_pull_request", + "github.request_copilot_review", + "gitlab.gitlab_list_projects" + ], + "hit1": "gitlab.gitlab_merge_merge_request", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 939, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "kind": "verb_product", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "ranking": [ + "notion.notion_search", + "notion.notion_list_users", + "notion.get_comment", + "notion.notion_get_page", + "notion.delete_comment", + "github.search_users", + "notion.update_comment", + "notion.notion_create_database", + "notion.create_comment", + "notion.notion_get_block" + ], + "hit1": "notion.notion_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 286, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 639, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "kind": "verb_product", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "ranking": [ + "notion.notion_get_page", + "notion.notion_update_page", + "notion.notion_create_page", + "notion.notion_get_comments", + "notion.create_comment", + "notion.notion_create_database", + "confluence.confluence_create_page", + "notion.notion_list_users", + "notion.get_comment", + "confluence.confluence_get_page" + ], + "hit1": "notion.notion_get_page", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "kind": "verb_product", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "ranking": [ + "notion.notion_query_database", + "notion.notion_create_database", + "notion.notion_create_page", + "notion.notion_search", + "notion.notion_list_users", + "notion.get_comment", + "notion.notion_get_page", + "notion.delete_comment", + "notion.update_comment", + "notion.create_comment" + ], + "hit1": "notion.notion_query_database", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 279, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_search", + "zendesk.zendesk_list_tickets", + "jira.jira_create_issue", + "slack.create_file", + "linear.create_comment", + "github.create_milestone", + "linear.create_user" + ], + "hit1": "zendesk.zendesk_get_ticket", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 875, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_get_contact", + "hubspot.hubspot_create_contact", + "hubspot.hubspot_create_deal", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_search_contacts", + "github.create_repository", + "github.create_gist", + "github.create_branch", + "github.create_issue", + "github.create_or_update_file" + ], + "hit1": "hubspot.hubspot_get_contact", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 899, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "kind": "verb_product", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_convert_lead", + "salesforce.salesforce_create_lead", + "salesforce.salesforce_create_record", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_update_record", + "salesforce.salesforce_get_record", + "salesforce.salesforce_list_opportunities", + "salesforce.salesforce_query", + "slack.create_file", + "linear.create_comment" + ], + "hit1": "salesforce.salesforce_convert_lead", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "kind": "identifier", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_query", + "github.get_check_run", + "github.delete_check_run", + "github.update_check_run", + "sqlite.sqlite_query", + "postgres.query", + "github.create_check_run", + "mysql.mysql_query", + "docker.docker_run", + "prometheus.prom_query" + ], + "hit1": "salesforce.salesforce_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 310, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "kind": "verb_product", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "ranking": [ + "shopify.shopify_list_orders", + "shopify.shopify_list_customers", + "shopify.shopify_list_products", + "shopify.shopify_get_product", + "shopify.shopify_update_product", + "shopify.shopify_get_order", + "shopify.shopify_create_product", + "shopify.shopify_create_order", + "jira.jira_list_projects", + "slack.list_files" + ], + "hit1": "shopify.shopify_list_orders", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "slack.create_file", + "linear.create_comment", + "github.create_milestone", + "linear.create_user" + ], + "hit1": "pagerduty.pagerduty_get_incident", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 857, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "kind": "paraphrase", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_list_oncalls", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_create_incident", + "twilio.twilio_make_call", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "slack.slack_create_channel", + "filesystem.list_allowed_directories", + "github.update_issue_comment" + ], + "hit1": "pagerduty.pagerduty_list_oncalls", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 419, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 772, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "kind": "verb_product", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "ranking": [ + "datadog.datadog_query_metrics", + "datadog.datadog_list_monitors", + "datadog.datadog_create_monitor", + "datadog.datadog_list_logs", + "datadog.datadog_get_monitor", + "prometheus.prom_list_metrics", + "prometheus.prom_query", + "notion.notion_query_database", + "prometheus.prom_query_range", + "salesforce.salesforce_query" + ], + "hit1": "datadog.datadog_query_metrics", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 804, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "kind": "verb_product", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "ranking": [ + "datadog.datadog_list_logs", + "datadog.datadog_list_monitors", + "datadog.datadog_get_monitor", + "datadog.datadog_create_monitor", + "datadog.datadog_query_metrics", + "kubernetes.k8s_pod_logs", + "docker.docker_logs", + "github.actions_get_job_logs", + "github.search_users", + "git.git_log" + ], + "hit1": "datadog.datadog_list_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 824, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_list_buckets", + "jira.jira_list_projects", + "slack.list_files", + "github.list_labels", + "linear.list_users", + "gitlab.list_labels", + "gitlab.gitlab_list_projects", + "notion.notion_list_users", + "linear.list_labels", + "github.projects_list" + ], + "hit1": "aws_s3.s3_list_buckets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 170, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 616, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "kind": "paraphrase", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_put_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_get_object", + "slack.slack_upload_file", + "github.assign_copilot_to_issue", + "gmail.gmail_send_email", + "twilio.twilio_send_sms", + "slack.slack_add_reaction", + "slack.slack_reply_to_thread", + "jira.jira_update_issue" + ], + "hit1": "aws_s3.s3_put_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 862, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns" + ], + "hit1": "cloudflare.cf_purge_cache", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "salesforce.salesforce_create_record", + "cloudflare.cf_list_zones", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_update_record", + "salesforce.salesforce_get_record", + "cloudflare.cf_purge_cache", + "git.git_commit", + "slack.create_file" + ], + "hit1": "cloudflare.cf_create_dns", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_deployments", + "kubernetes.k8s_list_deployments", + "vercel.vercel_list_projects", + "vercel.vercel_create_project", + "vercel.vercel_get_deployment", + "jira.jira_list_projects", + "slack.list_files", + "github.list_labels", + "linear.list_users", + "gitlab.list_labels" + ], + "hit1": "vercel.vercel_list_deployments", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 173, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 618, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "ranking": [ + "supabase.supabase_insert", + "mongodb.mongo_insert", + "supabase.supabase_list_tables", + "supabase.supabase_update", + "supabase.supabase_invoke_fn", + "supabase.supabase_select", + "playwright.browser_type" + ], + "hit1": "supabase.supabase_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 203, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "kind": "paraphrase", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "ranking": [ + "figma.figma_get_images", + "docker.docker_images", + "figma.figma_get_file", + "figma.figma_get_file_nodes", + "figma.figma_get_comments", + "figma.figma_post_comment", + "github.delete_file", + "github.create_branch", + "aws_s3.s3_get_object", + "gitlab.gitlab_get_file" + ], + "hit1": "figma.figma_get_images", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "kind": "verb_product", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "ranking": [ + "twilio.twilio_send_sms", + "twilio.twilio_list_messages", + "gmail.gmail_send_email", + "twilio.twilio_make_call", + "discord.discord_send_message", + "puppeteer.puppeteer_click", + "filesystem.list_directory_with_sizes", + "puppeteer.puppeteer_fill", + "puppeteer.puppeteer_screenshot", + "puppeteer.puppeteer_navigate" + ], + "hit1": "twilio.twilio_send_sms", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "kind": "verb_product", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "ranking": [ + "okta.okta_create_user", + "okta.okta_get_user", + "okta.okta_deactivate_user", + "okta.okta_list_users", + "linear.create_user", + "linear.get_user", + "linear.delete_user", + "linear.update_user", + "slack.slack_get_user_profile", + "github.create_issue" + ], + "hit1": "okta.okta_create_user", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 683, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query", + "prometheus.prom_query_range", + "prometheus.prom_list_metrics", + "notion.notion_query_database", + "datadog.datadog_query_metrics", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "postgres.query", + "postgres.explain_query", + "mysql.mysql_query" + ], + "hit1": "prometheus.prom_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "kind": "verb_product", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "ranking": [ + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "grafana.grafana_create_annotation", + "github.search_users", + "github.search_issues", + "github.search_orgs", + "confluence.confluence_search", + "github.search_commits", + "slack.slack_search_messages", + "linear.linear_search_issues" + ], + "hit1": "grafana.grafana_search_dashboards", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 636, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "kind": "verb_product", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "ranking": [ + "circleci.circleci_trigger_pipeline", + "circleci.circleci_get_pipeline", + "gitlab.gitlab_create_pipeline", + "circleci.circleci_list_pipelines", + "gitlab.get_pipeline", + "gitlab.delete_pipeline", + "gitlab.update_pipeline", + "gitlab.create_pipeline", + "github.actions_run_trigger", + "hubspot.hubspot_create_deal" + ], + "hit1": "circleci.circleci_trigger_pipeline", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 325, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "kind": "identifier", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "ranking": [ + "terraform.tf_plan", + "terraform.tf_apply", + "terraform.tf_state_list", + "github.get_check_run", + "github.delete_check_run", + "github.update_check_run", + "postgres.explain_query", + "github.create_check_run", + "docker.docker_run", + "github.actions_get" + ], + "hit1": "terraform.tf_plan", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "kind": "verb_product", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "ranking": [ + "asana.asana_get_task", + "asana.asana_create_task", + "asana.asana_update_task", + "asana.asana_create_project", + "asana.asana_list_tasks", + "clickup.clickup_create_task", + "clickup.clickup_get_task", + "clickup.clickup_update_task", + "github.create_issue", + "okta.okta_create_user" + ], + "hit1": "asana.asana_get_task", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "ranking": [ + "trello.trello_create_card", + "trello.trello_add_comment", + "trello.trello_create_board", + "trello.trello_list_cards", + "slack.create_file", + "linear.create_comment", + "github.create_milestone", + "linear.create_user", + "notion.create_comment", + "gitlab.create_label" + ], + "hit1": "trello.trello_create_card", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 413, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 903, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_create_task", + "clickup.clickup_get_task", + "clickup.clickup_update_task", + "clickup.clickup_list_tasks", + "asana.asana_get_task", + "asana.asana_create_task", + "asana.asana_update_task", + "playwright.browser_click", + "puppeteer.puppeteer_click", + "slack.create_file" + ], + "hit1": "clickup.clickup_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 770, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "kind": "verb_product", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "ranking": [ + "confluence.confluence_get_page", + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "confluence.confluence_search", + "notion.notion_get_page", + "notion.notion_create_page", + "notion.notion_update_page", + "playwright.browser_take_screenshot", + "notion.notion_get_comments", + "github.list_commits" + ], + "hit1": "confluence.confluence_get_page", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "kind": "verb_product", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "discord.discord_add_reaction", + "discord.discord_get_channel", + "discord.discord_list_channels", + "discord.discord_list_members", + "gmail.gmail_send_email", + "twilio.twilio_send_sms", + "slack.slack_post_message", + "github.delete_file", + "gmail.gmail_get_email" + ], + "hit1": "discord.discord_send_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_search", + "elasticsearch.es_get_doc", + "elasticsearch.es_index_doc", + "github.search_users", + "github.search_issues", + "github.search_orgs", + "github.search_commits", + "confluence.confluence_search", + "slack.slack_search_messages", + "linear.linear_search_issues" + ], + "hit1": "elasticsearch.es_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "redis.redis_set", + "redis.redis_delete", + "redis.redis_keys", + "jira.jira_get_issue", + "playwright.browser_press_key", + "figma.figma_get_file", + "aws_s3.s3_get_object", + "figma.figma_get_comments", + "figma.figma_get_file_nodes" + ], + "hit1": "redis.redis_get", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 589, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_find", + "mongodb.mongo_update", + "mongodb.mongo_delete", + "mongodb.mongo_insert", + "mongodb.mongo_list_collections" + ], + "hit1": "mongodb.mongo_find", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "kind": "verb_product", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "ranking": [ + "sqlite.sqlite_list_tables", + "sqlite.sqlite_query", + "supabase.supabase_list_tables", + "sqlite.sqlite_describe_table", + "postgres.list_tables", + "mysql.mysql_list_tables", + "jira.jira_list_projects", + "slack.list_files", + "github.list_labels", + "linear.list_users" + ], + "hit1": "sqlite.sqlite_list_tables", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 147, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 245, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 543, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "kind": "verb_product", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "ranking": [ + "postgres.describe_table", + "sqlite.sqlite_describe_table", + "supabase.supabase_update", + "supabase.supabase_select", + "supabase.supabase_insert", + "postgres.list_schemas", + "postgres.list_tables", + "postgres.query", + "postgres.explain_query" + ], + "hit1": "postgres.describe_table", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "gitlab.gitlab_create_issue", + "github.create_issue", + "github.add_issue_comment", + "bitbucket.bitbucket_create_issue", + "gitlab.gitlab_get_issue", + "linear.linear_create_comment", + "jira.jira_get_issue", + "github.issue_read", + "github.issue_write", + "jira.jira_update_issue" + ], + "hit1": "gitlab.gitlab_create_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.5940445580826669, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 347, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 511, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 931, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "ranking": [ + "gitlab.gitlab_create_issue", + "github.create_issue", + "bitbucket.bitbucket_create_issue", + "gitlab.gitlab_get_issue", + "linear.linear_create_comment", + "jira.jira_get_issue", + "jira.jira_update_issue", + "sentry.sentry_get_issue", + "jira.jira_create_issue", + "github.add_issue_comment" + ], + "hit1": "gitlab.gitlab_create_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.4991774411495835, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 332, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 472, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 880, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "kind": "identifier", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "ranking": [ + "filesystem.create_directory", + "filesystem.list_directory", + "filesystem.get_file_info", + "filesystem.list_directory_with_sizes", + "github.get_file_contents", + "filesystem.directory_tree", + "jira.jira_list_projects", + "slack.list_files", + "github.list_labels", + "linear.list_users" + ], + "hit1": "filesystem.create_directory", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 654, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "kind": "paraphrase", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "ranking": [ + "sequential_thinking.sequentialthinking" + ], + "hit1": "sequential_thinking.sequentialthinking", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "kind": "paraphrase", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "ranking": [ + "time.get_current_time", + "time.convert_time", + "slack.slack_create_channel", + "playwright.browser_wait_for", + "google_calendar.gcal_list_events", + "filesystem.list_allowed_directories", + "grafana.grafana_create_annotation", + "jira.jira_add_worklog", + "slack.get_file", + "slack.delete_file" + ], + "hit1": "time.get_current_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 222, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "kind": "verb_product", + "gold": [ + "time.convert_time" + ], + "alts": [], + "ranking": [ + "time.convert_time", + "salesforce.salesforce_convert_lead", + "time.get_current_time", + "playwright.browser_wait_for", + "google_calendar.gcal_list_events", + "memory.create_relations", + "git.git_diff", + "grafana.grafana_create_annotation", + "jira.jira_add_worklog" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "kind": "verb_product", + "gold": [ + "github.star_repository" + ], + "alts": [], + "ranking": [ + "github.star_repository", + "github.delete_repository", + "github.get_label", + "git.git_branch", + "github.list_tags", + "github.list_labels", + "github.list_branches", + "github.search_issues", + "github.get_tag", + "github.actions_list" + ], + "hit1": "github.star_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "kind": "resource", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "ranking": [ + "github.fork_repository", + "github.delete_repository", + "github.create_repository", + "github.unstar_repository", + "github.star_repository", + "github.list_repository_collaborators", + "github.get_repository_tree", + "github.list_labels", + "github.list_branches", + "github.delete_file" + ], + "hit1": "github.fork_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 248, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "kind": "verb_product", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "ranking": [ + "github.get_latest_release", + "gitlab.get_release", + "gitlab.delete_release", + "gitlab.update_release", + "github.get_release_by_tag", + "gitlab.create_release", + "github.get_teams", + "github.get_repository_tree", + "github.get_label", + "github.projects_get" + ], + "hit1": "github.get_latest_release", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 201, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 361, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "kind": "resource", + "gold": [ + "github.create_gist" + ], + "alts": [], + "ranking": [ + "github.get_gist", + "github.update_gist", + "github.create_gist", + "slack.create_file", + "linear.create_comment", + "github.create_milestone", + "linear.create_user", + "notion.create_comment", + "gitlab.create_label", + "jira.create_filter" + ], + "hit1": "github.get_gist", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 421, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 911, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "kind": "verb_product", + "gold": [ + "github.search_code" + ], + "alts": [], + "ranking": [ + "github.search_code", + "github.list_code_scanning_alerts", + "github.get_code_scanning_alert", + "github.search_users", + "github.search_issues", + "github.search_orgs", + "github.search_commits", + "github.search_pull_requests", + "github.search_repositories", + "hubspot.hubspot_search_contacts" + ], + "hit1": "github.search_code", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 419, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "kind": "verb_product", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "ranking": [ + "github.list_notifications", + "github.mark_all_notifications_read", + "github.list_tags", + "github.list_labels", + "github.projects_list", + "github.list_branches", + "git.git_branch", + "github.actions_list", + "github.list_releases", + "github.list_issues" + ], + "hit1": "github.list_notifications", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "kind": "verb_product", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "ranking": [ + "github.assign_copilot_to_issue", + "linear.linear_assign_issue", + "jira.jira_assign_issue", + "github.request_copilot_review", + "jira.jira_update_issue", + "github.add_issue_comment", + "linear.linear_create_comment", + "github.create_issue", + "jira.jira_add_comment", + "jira.jira_transition_issue" + ], + "hit1": "github.assign_copilot_to_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 434, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 875, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "kind": "resource", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "ranking": [ + "github.list_secret_scanning_alerts", + "github.get_secret_scanning_alert", + "github.list_code_scanning_alerts", + "github.list_dependabot_alerts", + "github.get_code_scanning_alert", + "slack.get_file", + "linear.get_comment", + "github.get_milestone", + "linear.get_user", + "notion.get_comment" + ], + "hit1": "github.list_secret_scanning_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "ranking": [ + "github.create_or_update_file", + "slack.update_file", + "github.update_issue_comment", + "github.get_file_contents", + "slack.create_file", + "filesystem.write_file", + "github.projects_write", + "github.label_write", + "github.delete_file", + "google_drive.gdrive_update_file" + ], + "hit1": "github.create_or_update_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 341, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 545, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1020, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "kind": "resource", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "ranking": [ + "github.list_repository_collaborators", + "github.list_commits", + "github.update_issue_comment", + "pagerduty.pagerduty_list_oncalls", + "trello.trello_list_cards", + "github.get_discussion_comments", + "github.create_pull_request", + "github.request_copilot_review", + "github.assign_copilot_to_issue", + "github.list_labels" + ], + "hit1": "github.list_repository_collaborators", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 307, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 445, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 962, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "kind": "verb_product", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "ranking": [ + "slack.slack_reply_to_thread", + "slack.slack_get_thread_replies", + "github.get_notification_details", + "slack.get_file", + "slack.delete_file", + "slack.update_file", + "slack.create_file", + "github.dismiss_notification", + "slack.list_files", + "slack.slack_list_users" + ], + "hit1": "slack.slack_reply_to_thread", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 346, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "kind": "verb_product", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "ranking": [ + "slack.slack_add_reaction", + "discord.discord_add_reaction", + "jira.jira_add_comment", + "github.add_issue_comment", + "gmail.gmail_modify_email", + "trello.trello_add_comment", + "git.git_add", + "playwright.browser_click", + "memory.add_observations", + "jira.jira_add_worklog" + ], + "hit1": "slack.slack_add_reaction", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 467, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 812, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "kind": "verb_product", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "ranking": [ + "slack.slack_search_messages", + "gmail.gmail_search_emails", + "twilio.twilio_list_messages", + "slack.slack_get_channel_history", + "github.search_users", + "github.search_issues", + "github.search_orgs", + "slack.get_file", + "github.search_commits", + "confluence.confluence_search" + ], + "hit1": "slack.slack_search_messages", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "kind": "verb_product", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "ranking": [ + "slack.slack_create_channel", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_get_channel_history", + "slack.slack_post_message", + "slack.slack_set_channel_topic", + "discord.discord_get_channel", + "slack.slack_upload_file", + "slack.create_file", + "slack.slack_get_thread_replies" + ], + "hit1": "slack.slack_create_channel", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "kind": "verb_product", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_transition_issue", + "jira.jira_get_issue", + "jira.jira_update_issue", + "jira.jira_assign_issue", + "jira.jira_add_comment", + "jira.jira_create_issue", + "jira.jira_add_worklog", + "gitlab.gitlab_get_issue", + "sentry.sentry_get_issue", + "gitlab.gitlab_create_issue" + ], + "hit1": "jira.jira_transition_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 207, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "kind": "identifier", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "ranking": [ + "jira.jira_search_issues", + "filesystem.list_directory_with_sizes", + "github.search_users", + "github.search_issues", + "jira.jira_list_projects", + "github.search_orgs", + "github.search_commits", + "confluence.confluence_search", + "slack.slack_search_messages", + "linear.linear_search_issues" + ], + "hit1": "jira.jira_search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 605, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "kind": "verb_product", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "ranking": [ + "jira.jira_add_worklog", + "jira.jira_add_comment", + "github.add_issue_comment", + "git.git_add", + "memory.add_observations", + "linear.linear_create_comment", + "github.add_comment_to_pending_review", + "discord.discord_add_reaction", + "slack.slack_add_reaction", + "jira.jira_list_projects" + ], + "hit1": "jira.jira_add_worklog", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 285, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 406, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 839, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "kind": "verb_product", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "ranking": [ + "linear.linear_list_teams", + "github.get_teams", + "linear.linear_list_projects", + "linear.list_users", + "linear.list_labels", + "linear.list_comments", + "linear.list_attachments", + "linear.get_comment", + "linear.linear_create_project", + "linear.list_cycles" + ], + "hit1": "linear.linear_list_teams", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 137, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 283, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 639, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "ranking": [ + "linear.linear_create_comment", + "github.update_issue_comment", + "linear.get_comment", + "linear.delete_comment", + "linear.update_comment", + "linear.create_comment", + "github.add_issue_comment", + "jira.jira_add_comment", + "linear.linear_get_issue", + "linear.linear_update_issue" + ], + "hit1": "linear.linear_create_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 798, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "kind": "paraphrase", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "ranking": [ + "filesystem.move_file", + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "brave_search.brave_local_search", + "slack.get_file", + "gmail.gmail_trash_email", + "slack.delete_file", + "slack.update_file", + "slack.create_file" + ], + "hit1": "filesystem.move_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "kind": "paraphrase", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "ranking": [ + "filesystem.search_files", + "redis.redis_keys", + "linear.linear_search_issues", + "google_drive.gdrive_search", + "slack.list_files", + "github.search_repositories", + "github.search_users", + "github.push_files", + "github.search_issues", + "github.search_orgs" + ], + "hit1": "filesystem.search_files", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 716, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "kind": "paraphrase", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "ranking": [ + "filesystem.edit_file", + "filesystem.list_directory_with_sizes", + "filesystem.write_file", + "slack.get_file", + "slack.delete_file", + "playwright.browser_tab_new", + "slack.update_file", + "twilio.twilio_make_call", + "slack.create_file", + "github.create_repository" + ], + "hit1": "filesystem.edit_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 735, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "ranking": [ + "memory.read_graph", + "memory.create_entities", + "memory.delete_relations", + "memory.create_relations", + "memory.search_nodes", + "filesystem.read_file", + "memory.open_nodes", + "memory.delete_entities", + "memory.add_observations", + "filesystem.read_text_file" + ], + "hit1": "memory.read_graph", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 260, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 561, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "kind": "verb_product", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "memory.open_nodes", + "figma.figma_get_file_nodes", + "github.search_users", + "github.search_issues", + "github.search_orgs", + "github.search_commits", + "confluence.confluence_search", + "slack.slack_search_messages", + "linear.linear_search_issues" + ], + "hit1": "memory.search_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 621, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "kind": "verb_product", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "ranking": [ + "git.git_diff_unstaged", + "git.git_diff", + "git.git_diff_staged", + "git.git_show", + "git.git_status", + "git.git_log", + "git.git_branch", + "git.git_checkout", + "git.git_commit", + "git.git_create_branch" + ], + "hit1": "git.git_diff_unstaged", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 621, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "kind": "identifier", + "gold": [ + "git.git_add" + ], + "alts": [], + "ranking": [ + "git.git_add", + "filesystem.list_directory_with_sizes", + "slack.list_files", + "salesforce.salesforce_list_opportunities", + "github.add_issue_comment", + "jira.jira_add_comment", + "github.push_files", + "memory.add_observations", + "jira.jira_add_worklog", + "filesystem.search_files" + ], + "hit1": "git.git_add", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 220, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "kind": "verb_product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "ranking": [ + "shopify.shopify_get_product", + "shopify.shopify_update_product", + "shopify.shopify_create_product", + "stripe.stripe_create_product", + "shopify.shopify_create_order", + "shopify.shopify_get_order", + "shopify.shopify_list_customers", + "shopify.shopify_list_orders", + "shopify.shopify_list_products", + "stripe.stripe_create_price" + ], + "hit1": "shopify.shopify_get_product", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "kind": "verb_product", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "ranking": [ + "stripe.stripe_cancel_subscription", + "stripe.stripe_create_product", + "stripe.stripe_create_customer", + "stripe.stripe_list_customers", + "stripe.stripe_get_customer", + "stripe.stripe_list_invoices", + "stripe.stripe_create_invoice", + "stripe.stripe_list_subscriptions", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_refund" + ], + "hit1": "stripe.stripe_cancel_subscription", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 610, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "kind": "resource", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "ranking": [ + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_refund", + "slack.create_file", + "linear.create_comment", + "github.create_milestone", + "linear.create_user", + "notion.create_comment", + "gitlab.create_label", + "jira.create_filter" + ], + "hit1": "stripe.stripe_get_payment_intent", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 426, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 916, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "kind": "verb_product", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "ranking": [ + "datadog.datadog_get_monitor", + "datadog.datadog_create_monitor", + "datadog.datadog_list_monitors", + "datadog.datadog_list_logs", + "datadog.datadog_query_metrics", + "slack.create_file", + "linear.create_comment", + "github.create_milestone", + "linear.create_user", + "notion.create_comment" + ], + "hit1": "datadog.datadog_get_monitor", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 948, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "kind": "paraphrase", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "ranking": [ + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "sentry.sentry_list_events", + "pagerduty.pagerduty_resolve_incident", + "sentry.sentry_list_projects", + "sentry.sentry_list_issues", + "gitlab.gitlab_get_issue", + "jira.jira_get_issue", + "jira.jira_update_issue", + "gitlab.gitlab_create_issue" + ], + "hit1": "sentry.sentry_get_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 745, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_scale_deployment", + "vercel.vercel_get_deployment", + "kubernetes.k8s_get_events", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_delete", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_list_services", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_apply", + "kubernetes.k8s_pod_logs" + ], + "hit1": "kubernetes.k8s_scale_deployment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 374, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_apply", + "terraform.tf_apply", + "kubernetes.k8s_get_events", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_delete", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_list_services", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_scale_deployment" + ], + "hit1": "kubernetes.k8s_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "ranking": [ + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_exec", + "docker.docker_run", + "docker.docker_images", + "docker.docker_ps", + "docker.docker_build", + "kubernetes.k8s_pod_logs" + ], + "hit1": "docker.docker_stop", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "kind": "verb_product", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "ranking": [ + "docker.docker_build", + "docker.docker_run", + "docker.docker_images", + "docker.docker_ps", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_exec", + "filesystem.read_media_file" + ], + "hit1": "docker.docker_build", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "kind": "verb_product", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "ranking": [ + "playwright.browser_click", + "puppeteer.puppeteer_click", + "playwright.browser_type", + "playwright.browser_take_screenshot", + "playwright.browser_snapshot", + "clickup.clickup_create_task", + "clickup.clickup_list_tasks", + "clickup.clickup_get_task", + "clickup.clickup_update_task", + "playwright.browser_tab_list" + ], + "hit1": "playwright.browser_click", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 654, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "kind": "paraphrase", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "ranking": [ + "playwright.browser_type", + "playwright.browser_wait_for", + "puppeteer.puppeteer_fill", + "github.projects_list", + "playwright.browser_tab_list", + "filesystem.read_text_file", + "playwright.browser_navigate", + "playwright.browser_tab_new", + "playwright.browser_tab_close", + "mongodb.mongo_insert" + ], + "hit1": "playwright.browser_type", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_list_projects", + "github.projects_list", + "jira.jira_list_projects", + "linear.linear_list_projects", + "vercel.vercel_list_projects", + "sentry.sentry_list_projects", + "gitlab.gitlab_list_issues", + "github.projects_get", + "gitlab.gitlab_create_issue", + "gitlab.gitlab_list_merge_requests" + ], + "hit1": "gitlab.gitlab_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 255, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 609, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "kind": "verb_product", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "ranking": [ + "bitbucket.bitbucket_create_pr", + "bitbucket.bitbucket_list_prs", + "bitbucket.bitbucket_create_issue", + "github.pull_request_read", + "github.merge_pull_request", + "bitbucket.bitbucket_list_repos", + "github.update_pull_request_branch", + "github.update_pull_request", + "github.create_pull_request", + "github.pull_request_review_write" + ], + "hit1": "bitbucket.bitbucket_create_pr", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 508, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1091, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "kind": "verb_product", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "ranking": [ + "gmail.gmail_create_draft", + "gmail.gmail_list_labels", + "gmail.gmail_get_email", + "gmail.gmail_search_emails", + "gmail.gmail_trash_email", + "gmail.gmail_send_email", + "github.create_pull_request", + "gmail.gmail_modify_email", + "slack.create_file", + "linear.create_comment" + ], + "hit1": "gmail.gmail_create_draft", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 844, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "kind": "verb_product", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_share_file", + "google_drive.gdrive_update_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "slack.get_file", + "google_calendar.gcal_list_calendars", + "slack.delete_file", + "slack.update_file" + ], + "hit1": "google_drive.gdrive_share_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "kind": "verb_product", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_list_events", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_list_calendars", + "google_calendar.gcal_create_event", + "kubernetes.k8s_get_events", + "sentry.sentry_list_events", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "google_drive.gdrive_update_file" + ], + "hit1": "google_calendar.gcal_list_events", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "slack.delete_file", + "github.delete_repository", + "github.pull_request_review_write", + "linear.delete_comment", + "github.delete_milestone" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_projects", + "vercel.vercel_create_project", + "vercel.vercel_list_deployments", + "jira.jira_list_projects", + "vercel.vercel_get_deployment", + "gitlab.gitlab_list_projects", + "github.projects_list", + "linear.linear_list_projects", + "sentry.sentry_list_projects", + "github.projects_get" + ], + "hit1": "vercel.vercel_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 175, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 269, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "ranking": [ + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update", + "supabase.supabase_list_tables", + "supabase.supabase_invoke_fn", + "github.delete_file", + "github.create_branch", + "aws_s3.s3_get_object", + "gitlab.gitlab_get_file", + "github.get_tag" + ], + "hit1": "supabase.supabase_select", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 815, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "kind": "verb_product", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "figma.figma_get_file", + "slack.slack_post_message", + "figma.figma_get_comments", + "figma.figma_get_file_nodes", + "figma.figma_get_images", + "linear.get_comment", + "notion.get_comment", + "linear.delete_comment", + "linear.update_comment" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "kind": "verb_product", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "ranking": [ + "okta.okta_list_users", + "okta.okta_get_user", + "okta.okta_create_user", + "okta.okta_deactivate_user", + "linear.list_users", + "notion.notion_list_users", + "slack.slack_list_users", + "slack.slack_get_users", + "github.search_users", + "notion.list_users_tokens" + ], + "hit1": "okta.okta_list_users", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 306, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query_range", + "prometheus.prom_query", + "prometheus.prom_list_metrics", + "notion.notion_query_database", + "datadog.datadog_query_metrics", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "postgres.query", + "postgres.explain_query", + "mysql.mysql_query" + ], + "hit1": "prometheus.prom_query_range", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "kind": "verb_product", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "ranking": [ + "grafana.grafana_create_annotation", + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "slack.create_file", + "linear.create_comment", + "github.create_milestone", + "linear.create_user", + "notion.create_comment", + "gitlab.create_label", + "jira.create_filter" + ], + "hit1": "grafana.grafana_create_annotation", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 410, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 900, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "kind": "identifier", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "ranking": [ + "terraform.tf_apply", + "kubernetes.k8s_apply", + "terraform.tf_plan", + "terraform.tf_state_list" + ], + "hit1": "terraform.tf_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "kind": "verb_product", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "ranking": [ + "asana.asana_list_tasks", + "clickup.clickup_list_tasks", + "asana.asana_create_project", + "asana.asana_get_task", + "asana.asana_create_task", + "asana.asana_update_task", + "jira.jira_list_projects", + "slack.list_files", + "github.list_labels", + "linear.list_users" + ], + "hit1": "asana.asana_list_tasks", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 708, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "ranking": [ + "trello.trello_add_comment", + "trello.trello_create_card", + "trello.trello_create_board", + "trello.trello_list_cards", + "github.update_issue_comment", + "linear.get_comment", + "notion.get_comment", + "linear.delete_comment", + "linear.update_comment", + "notion.delete_comment" + ], + "hit1": "trello.trello_add_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 241, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 422, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 695, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_update_task", + "clickup.clickup_get_task", + "clickup.clickup_create_task", + "asana.asana_update_task", + "clickup.clickup_list_tasks", + "asana.asana_get_task", + "playwright.browser_click", + "puppeteer.puppeteer_click", + "asana.asana_create_task", + "slack.update_file" + ], + "hit1": "clickup.clickup_update_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "kind": "verb_product", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "ranking": [ + "confluence.confluence_search", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "confluence.confluence_create_page", + "github.search_users", + "github.search_issues", + "github.search_orgs", + "github.search_commits", + "slack.slack_search_messages", + "linear.linear_search_issues" + ], + "hit1": "confluence.confluence_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 677, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_deal", + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_search_contacts", + "github.create_repository", + "github.create_gist", + "github.create_branch", + "github.create_issue", + "github.create_or_update_file" + ], + "hit1": "hubspot.hubspot_create_deal", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 899, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_search", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_create_ticket", + "jira.jira_list_projects", + "slack.list_files", + "github.list_labels", + "linear.list_users", + "gitlab.list_labels" + ], + "hit1": "zendesk.zendesk_list_tickets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "kind": "verb_product", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "ranking": [ + "discord.discord_list_channels", + "slack.slack_list_channels", + "discord.discord_get_channel", + "discord.discord_list_members", + "discord.discord_send_message", + "discord.discord_add_reaction", + "jira.jira_list_projects", + "slack.list_files", + "github.list_labels", + "linear.list_users" + ], + "hit1": "discord.discord_list_channels", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 639, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "ranking": [ + "redis.redis_set", + "redis.redis_get", + "redis.redis_delete", + "redis.redis_keys", + "slack.slack_set_channel_topic", + "playwright.browser_press_key", + "jira.jira_get_issue", + "jira.jira_update_issue", + "aws_s3.s3_delete_object", + "figma.figma_get_file" + ], + "hit1": "redis.redis_set", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 595, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert", + "mongodb.mongo_update", + "mongodb.mongo_delete", + "mongodb.mongo_find", + "mongodb.mongo_list_collections", + "supabase.supabase_insert" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "kind": "verb_product", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "ranking": [ + "mysql.mysql_query", + "postgres.query", + "sqlite.sqlite_query", + "mysql.mysql_list_tables", + "postgres.explain_query", + "github.get_check_run", + "github.delete_check_run", + "github.update_check_run", + "salesforce.salesforce_query", + "github.create_check_run" + ], + "hit1": "mysql.mysql_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 619, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "elasticsearch.es_search", + "playwright.browser_tab_close", + "slack.get_file", + "slack.delete_file", + "linear.get_comment", + "github.get_milestone", + "linear.get_user", + "notion.get_comment" + ], + "hit1": "elasticsearch.es_index_doc", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 607, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_put_object", + "okta.okta_get_user", + "asana.asana_get_task", + "jira.jira_update_issue", + "github.create_issue", + "asana.asana_create_project", + "okta.okta_create_user", + "elasticsearch.es_search" + ], + "hit1": "aws_s3.s3_get_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones", + "cloudflare.cf_list_dns", + "cloudflare.cf_create_dns", + "cloudflare.cf_purge_cache", + "jira.jira_list_projects", + "slack.list_files", + "github.list_labels", + "linear.list_users", + "gitlab.list_labels", + "gitlab.gitlab_list_projects" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_create_product", + "stripe.stripe_create_customer", + "stripe.stripe_create_refund", + "stripe.stripe_create_price", + "stripe.stripe_list_customers", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_customer", + "stripe.stripe_list_invoices", + "stripe.stripe_list_subscriptions" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "kind": "verb_product", + "gold": [ + "github.actions_list" + ], + "alts": [], + "ranking": [ + "github.actions_list", + "github.actions_get", + "github.actions_get_job_logs", + "github.actions_run_trigger", + "github.list_tags", + "github.list_labels", + "github.projects_list", + "github.list_branches", + "git.git_branch", + "github.list_releases" + ], + "hit1": "github.actions_list", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 457, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 782, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "kind": "verb_product", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "ranking": [ + "github.actions_get_job_logs", + "gitlab.get_job", + "github.actions_list", + "github.actions_get", + "gitlab.delete_job", + "gitlab.update_job", + "gitlab.create_job", + "github.actions_run_trigger", + "kubernetes.k8s_pod_logs", + "docker.docker_logs" + ], + "hit1": "github.actions_get_job_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 822, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "kind": "verb_product", + "gold": [ + "github.create_repository" + ], + "alts": [], + "ranking": [ + "github.create_repository", + "github.delete_repository", + "github.star_repository", + "github.unstar_repository", + "github.create_branch", + "github.create_issue", + "github.fork_repository", + "github.list_repository_collaborators", + "github.list_labels", + "github.list_tags" + ], + "hit1": "github.create_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 406, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 837, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "kind": "verb_product", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "ranking": [ + "git.git_checkout", + "github.create_branch", + "git.git_create_branch", + "git.git_branch", + "github.update_pull_request_branch", + "gitlab.gitlab_create_merge_request", + "github.delete_file", + "github.list_commits", + "github.create_or_update_file", + "github.push_files" + ], + "hit1": "git.git_checkout", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 266, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1001, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "kind": "identifier", + "gold": [ + "git.git_log" + ], + "alts": [], + "ranking": [ + "git.git_log", + "git.git_show", + "git.git_status", + "git.git_diff_staged", + "git.git_diff_unstaged", + "git.git_diff", + "git.git_branch", + "git.git_checkout", + "git.git_commit", + "git.git_create_branch" + ], + "hit1": "git.git_log", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 619, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "kind": "verb_product", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "ranking": [ + "postgres.list_schemas", + "postgres.list_tables", + "postgres.query", + "postgres.explain_query", + "postgres.describe_table", + "jira.jira_list_projects", + "slack.list_files", + "github.list_labels", + "linear.list_users", + "gitlab.list_labels" + ], + "hit1": "postgres.list_schemas", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 158, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 604, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "kind": "verb_product", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "ranking": [ + "memory.open_nodes", + "memory.search_nodes", + "memory.read_graph", + "memory.create_entities", + "memory.delete_relations", + "memory.create_relations", + "memory.add_observations", + "memory.delete_entities", + "figma.figma_get_file_nodes", + "playwright.browser_tab_list" + ], + "hit1": "memory.open_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 260, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 531, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "kind": "paraphrase", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "ranking": [ + "filesystem.get_file_info", + "filesystem.read_file", + "filesystem.read_text_file", + "google_drive.gdrive_get_file", + "filesystem.read_media_file", + "gitlab.gitlab_get_file", + "slack.get_file", + "github.delete_file", + "filesystem.write_file", + "github.get_file_contents" + ], + "hit1": "filesystem.get_file_info", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 220, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 769, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "kind": "paraphrase", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "ranking": [ + "filesystem.write_file", + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "github.issue_write", + "filesystem.edit_file", + "playwright.browser_tab_new", + "brave_search.brave_local_search", + "slack.get_file", + "slack.delete_file" + ], + "hit1": "filesystem.write_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 752, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "kind": "verb_product", + "gold": [ + "github.search_issues" + ], + "alts": [], + "ranking": [ + "github.search_issues", + "linear.linear_search_issues", + "github.list_issues", + "gitlab.gitlab_list_issues", + "jira.jira_search_issues", + "github.search_users", + "github.search_orgs", + "github.search_commits", + "github.search_pull_requests", + "sentry.sentry_list_issues" + ], + "hit1": "github.search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 263, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 710, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "kind": "paraphrase", + "gold": [ + "github.get_me" + ], + "alts": [], + "ranking": [ + "github.get_teams", + "github.list_notifications", + "github.get_me", + "linear.get_user", + "okta.okta_get_user", + "slack.slack_get_user_profile", + "linear.delete_user", + "linear.update_user", + "linear.create_user", + "okta.okta_deactivate_user" + ], + "hit1": "github.get_teams", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 284, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "cost": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.3466666666666667, + "p95_calls": 1.0, + "success_within_1": 0.9733333333333334, + "success_within_2": 0.9866666666666667, + "success_within_3": 0.9933333333333333, + "mean_tokens": 318.5933333333333, + "p95_tokens": 336.9499999999998, + "total_tokens": 47789, + "tokens_per_successful_query": 318.5933333333333, + "total_tokens_successful": 47789 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.1933333333333334, + "p95_calls": 1.0, + "success_within_1": 0.9866666666666667, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 450.04, + "p95_tokens": 485.99999999999966, + "total_tokens": 67506, + "tokens_per_successful_query": 450.04, + "total_tokens_successful": 67506 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.0933333333333333, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 812.5, + "p95_tokens": 962.0, + "total_tokens": 121875, + "tokens_per_successful_query": 812.5, + "total_tokens_successful": 121875 + } + } +} diff --git a/experiments/mcp-tool-search/results/bm25_weighted.csv b/experiments/mcp-tool-search/results/bm25_weighted.csv new file mode 100644 index 00000000..a5f1591f --- /dev/null +++ b/experiments/mcp-tool-search/results/bm25_weighted.csv @@ -0,0 +1,151 @@ +id,query,kind,hit1,recall@1,recall@3,recall@5,mrr@10,ndcg@10,hit_rank,k3_calls,k3_tokens,k3_success1,k5_calls,k5_tokens,k5_success1,k10_calls,k10_tokens,k10_success1,ranking +q001,Create a GitHub issue,verb_product,github.create_issue,1.0,1.0,1.0,1.0000,0.9502,1,1,388,1.0,1,598,1.0,1,955,1.0,github.create_issue github.issue_read github.issue_write github.add_issue_comment github.update_issue_comment git.git_checkout git.git_show git.git_branch git.git_create_branch github.assign_copilot_to_issue +q002,Create a Jira ticket,verb_product,jira.jira_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,312,1.0,1,412,1.0,1,763,1.0,jira.jira_create_issue jira.jira_create_sprint jira.jira_assign_issue jira.jira_list_projects jira.jira_update_issue jira.jira_get_issue jira.jira_search_issues jira.jira_add_comment jira.jira_add_worklog jira.jira_transition_issue +q003,Send a Slack message,verb_product,slack.slack_post_message,1.0,1.0,1.0,1.0000,1.0000,1,1,232,1.0,1,356,1.0,1,707,1.0,slack.slack_post_message slack.slack_add_reaction slack.slack_join_channel slack.slack_leave_channel slack.slack_create_channel slack.slack_search_messages slack.slack_upload_file slack.slack_list_users slack.slack_get_users slack.slack_list_channels +q004,List open pull requests,resource,github.list_pull_requests,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,429,1.0,1,1035,1.0,github.list_pull_requests github.search_pull_requests gitlab.gitlab_list_merge_requests github.merge_pull_request github.pull_request_read github.create_pull_request github.update_pull_request github.update_pull_request_branch github.pull_request_review_write bitbucket.bitbucket_list_prs +q005,Open a pull request on GitHub,verb_product,github.create_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,368,1.0,1,653,1.0,1,1052,1.0,github.create_pull_request github.merge_pull_request github.pull_request_read github.update_pull_request github.pull_request_review_write github.update_pull_request_branch github.request_copilot_review github.search_pull_requests github.list_pull_requests git.git_checkout +q006,Run a SQL query against postgres,verb_product,postgres.query,1.0,1.0,1.0,1.0000,1.0000,1,1,167,1.0,1,280,1.0,1,590,1.0,postgres.query postgres.explain_query postgres.list_schemas postgres.list_tables postgres.describe_table mysql.mysql_query sqlite.sqlite_query salesforce.salesforce_query prometheus.prom_query docker.docker_run +q007,Read a local file from disk,paraphrase,filesystem.read_file,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,385,1.0,1,782,1.0,filesystem.read_file filesystem.read_text_file filesystem.read_media_file filesystem.write_file filesystem.edit_file filesystem.get_file_info filesystem.move_file brave_search.brave_local_search github.delete_file gitlab.gitlab_get_file +q008,What files are in this folder,paraphrase,filesystem.search_files,0.0,0.0,0.0,0.1667,0.4348,6,2,473,0.0,2,730,0.0,1,708,1.0,filesystem.search_files google_drive.gdrive_list_folder slack.list_files github.push_files filesystem.move_file filesystem.list_directory filesystem.list_directory_with_sizes filesystem.directory_tree filesystem.read_file filesystem.read_text_file +q009,Remember this fact in the knowledge graph,paraphrase,memory.read_graph,0.0,1.0,1.0,0.5000,0.5339,2,1,154,1.0,1,261,1.0,1,524,1.0,memory.read_graph memory.create_entities memory.delete_relations memory.create_relations memory.add_observations memory.open_nodes memory.search_nodes memory.delete_entities memory.delete_observations time.get_current_time +q010,Search the web with Brave,verb_product,brave_search.brave_web_search,1.0,1.0,1.0,1.0000,1.0000,1,1,245,1.0,1,412,1.0,1,726,1.0,brave_search.brave_web_search brave_search.brave_local_search filesystem.list_directory_with_sizes jira.jira_search_issues github.update_pull_request_branch memory.search_nodes puppeteer.puppeteer_screenshot puppeteer.puppeteer_navigate puppeteer.puppeteer_click puppeteer.puppeteer_fill +q011,Fetch the contents of a URL,verb_product,fetch.fetch,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,372,1.0,1,707,1.0,fetch.fetch github.get_file_contents playwright.browser_navigate filesystem.read_file puppeteer.puppeteer_navigate git.git_add slack.slack_set_channel_topic github.get_repository_tree git.git_status playwright.browser_snapshot +q012,Show git status,identifier,git.git_status,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,304,1.0,1,600,1.0,git.git_status git.git_show git.git_diff git.git_checkout git.git_branch git.git_commit git.git_add git.git_log git.git_diff_unstaged git.git_diff_staged +q013,Commit staged changes,paraphrase,git.git_commit,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,328,1.0,1,663,1.0,git.git_commit github.get_commit git.git_diff_staged git.git_reset git.git_log git.git_show github.push_files +q014,Create a Linear issue,verb_product,linear.linear_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,340,1.0,1,463,1.0,1,765,1.0,linear.linear_create_issue linear.linear_update_issue linear.linear_assign_issue linear.linear_get_issue linear.linear_create_comment linear.linear_create_project linear.linear_list_teams linear.linear_list_projects linear.linear_list_issues linear.linear_search_issues +q015,Make a task in my project tracker Linear,paraphrase,linear.linear_create_project,0.0,1.0,1.0,0.5000,0.6309,2,1,352,1.0,1,432,1.0,1,769,1.0,linear.linear_create_project linear.linear_create_issue linear.linear_update_issue linear.linear_list_teams linear.linear_list_projects linear.linear_list_issues linear.linear_search_issues linear.linear_create_comment linear.linear_assign_issue linear.linear_get_issue +q016,Post a comment on a GitHub issue,verb_product,github.update_issue_comment,0.0,1.0,1.0,0.5000,0.6309,2,1,333,1.0,1,601,1.0,1,938,1.0,github.update_issue_comment github.add_issue_comment github.issue_read github.issue_write github.create_issue github.assign_copilot_to_issue git.git_checkout git.git_show git.git_branch git.git_diff +q017,Merge the pull request,ambiguous,github.merge_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,283,1.0,1,523,1.0,1,1121,1.0,github.merge_pull_request gitlab.gitlab_merge_merge_request gitlab.gitlab_get_merge_request github.pull_request_read github.create_pull_request github.update_pull_request_branch github.update_pull_request gitlab.gitlab_create_merge_request github.pull_request_review_write github.request_copilot_review +q018,Trigger a GitHub Actions workflow,verb_product,github.actions_run_trigger,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,450,1.0,1,737,1.0,github.actions_run_trigger github.actions_get github.actions_list github.actions_get_job_logs git.git_checkout git.git_show git.git_branch git.git_diff git.git_commit git.git_status +q019,Get Dependabot alerts,resource,github.list_dependabot_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,287,1.0,1,445,1.0,1,798,1.0,github.list_dependabot_alerts github.get_dependabot_alert github.list_code_scanning_alerts github.list_secret_scanning_alerts kubernetes.k8s_get_pod redis.redis_get github.get_discussion github.projects_get github.get_label aws_s3.s3_get_object +q020,List Sentry issues,verb_product,sentry.sentry_list_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,396,1.0,1,796,1.0,sentry.sentry_list_issues sentry.sentry_list_projects sentry.sentry_list_events sentry.sentry_get_issue sentry.sentry_update_issue github.list_issues github.search_issues gitlab.gitlab_list_issues linear.linear_list_issues linear.linear_search_issues +q021,Create a Stripe customer,verb_product,stripe.stripe_get_customer,0.0,1.0,1.0,0.5000,0.6309,2,1,186,1.0,1,300,1.0,1,669,1.0,stripe.stripe_get_customer stripe.stripe_create_customer stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_list_subscriptions stripe.stripe_create_product stripe.stripe_create_price stripe.stripe_create_refund stripe.stripe_create_payment_intent stripe.stripe_list_customers +q022,Refund a Stripe payment,paraphrase,stripe.stripe_create_refund,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,352,1.0,1,662,1.0,stripe.stripe_create_refund stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_list_customers stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_create_product stripe.stripe_create_price +q023,Send an email with Gmail,verb_product,gmail.gmail_send_email,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,389,1.0,1,703,1.0,gmail.gmail_send_email gmail.gmail_get_email gmail.gmail_trash_email gmail.gmail_modify_email gmail.gmail_search_emails gmail.gmail_create_draft gmail.gmail_list_labels filesystem.list_directory_with_sizes twilio.twilio_send_sms puppeteer.puppeteer_click +q024,Search my Gmail inbox,verb_product,gmail.gmail_search_emails,1.0,1.0,1.0,1.0000,1.0000,1,1,205,1.0,1,352,1.0,1,714,1.0,gmail.gmail_search_emails gmail.gmail_create_draft gmail.gmail_list_labels gmail.gmail_send_email gmail.gmail_get_email gmail.gmail_trash_email gmail.gmail_modify_email brave_search.brave_web_search brave_search.brave_local_search mysql.mysql_list_tables +q025,Upload a file to Google Drive,paraphrase,google_drive.gdrive_update_file,0.0,0.0,1.0,0.2500,0.4307,4,2,496,0.0,1,418,1.0,1,791,1.0,google_drive.gdrive_update_file google_drive.gdrive_share_file google_drive.gdrive_get_file google_drive.gdrive_create_file google_drive.gdrive_search google_drive.gdrive_list_folder slack.slack_upload_file gitlab.gitlab_get_file slack.slack_reply_to_thread figma.figma_get_file +q026,Create a calendar event,resource,google_calendar.gcal_delete_event,0.0,1.0,1.0,0.5000,0.6309,2,1,300,1.0,1,427,1.0,1,895,1.0,google_calendar.gcal_delete_event google_calendar.gcal_create_event google_calendar.gcal_update_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars github.create_gist github.create_milestone github.create_hook github.create_environment github.create_package +q027,List Kubernetes pods,verb_product,kubernetes.k8s_list_pods,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,314,1.0,1,718,1.0,kubernetes.k8s_list_pods kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_get_pod kubernetes.k8s_apply kubernetes.k8s_delete kubernetes.k8s_scale_deployment kubernetes.k8s_pod_logs clickup.clickup_list_tasks +q028,Get logs from a k8s pod,verb_product,kubernetes.k8s_pod_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,262,1.0,1,455,1.0,1,834,1.0,kubernetes.k8s_pod_logs kubernetes.k8s_get_pod docker.docker_logs datadog.datadog_list_logs github.actions_get_job_logs git.git_log aws_s3.s3_get_object elasticsearch.es_get_doc gitlab.gitlab_get_file github.get_tag +q029,Run a docker container,verb_product,docker.docker_run,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,345,1.0,1,644,1.0,docker.docker_run docker.docker_stop docker.docker_rm docker.docker_exec docker.docker_logs docker.docker_images docker.docker_build docker.docker_ps github.create_check_run github.get_check_run +q030,Show docker container logs,verb_product,docker.docker_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,345,1.0,1,686,1.0,docker.docker_logs docker.docker_stop docker.docker_rm docker.docker_run docker.docker_exec docker.docker_images docker.docker_build docker.docker_ps kubernetes.k8s_pod_logs datadog.datadog_list_logs +q031,Navigate the browser to a URL,ambiguous,playwright.browser_navigate,1.0,1.0,1.0,1.0000,0.9502,1,1,180,1.0,1,265,1.0,1,536,1.0,playwright.browser_navigate playwright.browser_tab_new puppeteer.puppeteer_navigate playwright.browser_tab_close playwright.browser_tab_list playwright.browser_click playwright.browser_evaluate playwright.browser_snapshot playwright.browser_press_key playwright.browser_wait_for +q032,Take a screenshot of the page,ambiguous,playwright.browser_take_screenshot,1.0,1.0,1.0,1.0000,1.0000,1,1,226,1.0,1,352,1.0,1,711,1.0,playwright.browser_take_screenshot puppeteer.puppeteer_screenshot notion.notion_get_page notion.notion_update_page confluence.confluence_get_page confluence.confluence_create_page confluence.confluence_update_page notion.notion_create_page playwright.browser_snapshot playwright.browser_press_key +q033,Create a GitLab merge request,verb_product,gitlab.gitlab_merge_merge_request,0.0,1.0,1.0,0.3333,0.5000,3,1,297,1.0,1,475,1.0,1,792,1.0,gitlab.gitlab_merge_merge_request gitlab.gitlab_get_merge_request gitlab.gitlab_create_merge_request github.merge_pull_request gitlab.gitlab_list_merge_requests git.git_checkout git.git_show git.git_branch git.git_create_branch git.git_diff +q034,Search Notion pages,verb_product,notion.notion_search,1.0,1.0,1.0,1.0000,1.0000,1,1,195,1.0,1,366,1.0,1,686,1.0,notion.notion_search notion.notion_get_page notion.notion_get_block notion.notion_query_database notion.notion_create_database notion.notion_update_page notion.notion_list_users notion.notion_get_comments notion.notion_append_block notion.notion_create_page +q035,Create a Notion page,verb_product,notion.notion_get_page,0.0,1.0,1.0,0.3333,0.5000,3,1,230,1.0,1,371,1.0,1,687,1.0,notion.notion_get_page notion.notion_update_page notion.notion_create_page notion.notion_get_comments notion.notion_create_database notion.notion_search notion.notion_get_block notion.notion_query_database notion.notion_list_users notion.notion_append_block +q036,Query a Notion database,verb_product,notion.notion_query_database,1.0,1.0,1.0,1.0000,1.0000,1,1,279,1.0,1,401,1.0,1,687,1.0,notion.notion_query_database notion.notion_create_database notion.notion_create_page notion.notion_search notion.notion_get_page notion.notion_get_block notion.notion_update_page notion.notion_list_users notion.notion_get_comments notion.notion_append_block +q037,Create a Zendesk ticket,verb_product,zendesk.zendesk_create_ticket,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,368,1.0,1,856,1.0,zendesk.zendesk_create_ticket zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_search zendesk.zendesk_list_tickets jira.jira_create_issue github.create_gist github.create_milestone github.create_hook github.create_environment +q038,Create a HubSpot contact,verb_product,hubspot.hubspot_create_contact,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,904,1.0,hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_create_deal hubspot.hubspot_list_contacts hubspot.hubspot_search_contacts github.create_gist github.create_repository github.create_branch github.create_issue github.create_pull_request +q039,Create a Salesforce lead,verb_product,salesforce.salesforce_convert_lead,0.0,1.0,1.0,0.5000,0.6309,2,1,252,1.0,1,402,1.0,1,787,1.0,salesforce.salesforce_convert_lead salesforce.salesforce_create_lead salesforce.salesforce_create_record salesforce.salesforce_query salesforce.salesforce_update_record salesforce.salesforce_delete_record salesforce.salesforce_list_opportunities salesforce.salesforce_get_record github.create_gist github.create_milestone +q040,Run a SOQL query,identifier,salesforce.salesforce_query,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,291,1.0,1,724,1.0,salesforce.salesforce_query postgres.query sqlite.sqlite_query mysql.mysql_query prometheus.prom_query docker.docker_run postgres.explain_query prometheus.prom_query_range datadog.datadog_query_metrics notion.notion_query_database +q041,List Shopify orders,verb_product,shopify.shopify_list_orders,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,354,1.0,1,682,1.0,shopify.shopify_list_orders shopify.shopify_list_products shopify.shopify_list_customers shopify.shopify_get_product shopify.shopify_create_product shopify.shopify_update_product shopify.shopify_get_order shopify.shopify_create_order clickup.clickup_list_tasks trello.trello_list_cards +q042,Create a PagerDuty incident,verb_product,pagerduty.pagerduty_create_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,403,1.0,1,838,1.0,pagerduty.pagerduty_create_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls github.create_gist github.create_milestone github.create_hook github.create_environment +q043,Who is on call in PagerDuty,paraphrase,pagerduty.pagerduty_list_oncalls,1.0,1.0,1.0,1.0000,1.0000,1,1,227,1.0,1,401,1.0,1,721,1.0,pagerduty.pagerduty_list_oncalls pagerduty.pagerduty_list_incidents pagerduty.pagerduty_get_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident twilio.twilio_make_call filesystem.list_allowed_directories trello.trello_add_comment notion.notion_get_comments +q044,Query Datadog metrics,verb_product,datadog.datadog_query_metrics,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,458,1.0,1,728,1.0,datadog.datadog_query_metrics datadog.datadog_list_monitors datadog.datadog_create_monitor datadog.datadog_list_logs datadog.datadog_get_monitor prometheus.prom_list_metrics prometheus.prom_query postgres.query postgres.explain_query salesforce.salesforce_query +q045,Search Datadog logs,verb_product,datadog.datadog_list_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,458,1.0,1,877,1.0,datadog.datadog_list_logs datadog.datadog_list_monitors datadog.datadog_get_monitor datadog.datadog_create_monitor datadog.datadog_query_metrics docker.docker_logs kubernetes.k8s_pod_logs github.actions_get_job_logs brave_search.brave_web_search brave_search.brave_local_search +q046,List S3 buckets,verb_product,aws_s3.s3_list_buckets,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,318,1.0,1,667,1.0,aws_s3.s3_list_buckets aws_s3.s3_list_objects clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments kubernetes.k8s_list_services github.projects_list github.list_milestones github.list_hooks github.list_environments +q047,Upload an object to S3,paraphrase,aws_s3.s3_put_object,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,437,1.0,1,950,1.0,aws_s3.s3_put_object aws_s3.s3_delete_object aws_s3.s3_get_object slack.slack_upload_file github.assign_copilot_to_issue slack.slack_reply_to_thread github.add_comment_to_pending_review slack.slack_add_reaction twilio.twilio_send_sms gmail.gmail_send_email +q048,Purge Cloudflare cache,verb_product,cloudflare.cf_purge_cache,1.0,1.0,1.0,1.0000,1.0000,1,1,231,1.0,1,340,1.0,1,746,1.0,cloudflare.cf_purge_cache cloudflare.cf_list_zones cloudflare.cf_create_dns cloudflare.cf_list_dns +q049,Create a Cloudflare DNS record,verb_product,cloudflare.cf_create_dns,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,356,1.0,1,738,1.0,cloudflare.cf_create_dns cloudflare.cf_list_dns cloudflare.cf_list_zones cloudflare.cf_purge_cache salesforce.salesforce_create_record salesforce.salesforce_update_record salesforce.salesforce_delete_record salesforce.salesforce_get_record git.git_commit github.create_gist +q050,List Vercel deployments,verb_product,vercel.vercel_list_deployments,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,291,1.0,1,579,1.0,vercel.vercel_list_deployments vercel.vercel_list_projects vercel.vercel_get_deployment vercel.vercel_create_project kubernetes.k8s_list_deployments clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_services aws_s3.s3_list_buckets github.projects_list +q051,Insert a row into Supabase,verb_product,supabase.supabase_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,355,1.0,1,665,1.0,supabase.supabase_insert supabase.supabase_select supabase.supabase_update supabase.supabase_invoke_fn supabase.supabase_list_tables mongodb.mongo_insert playwright.browser_type +q052,Export images from Figma,paraphrase,figma.figma_get_images,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,368,1.0,1,658,1.0,figma.figma_get_images figma.figma_get_file figma.figma_get_file_nodes figma.figma_get_comments figma.figma_post_comment docker.docker_images elasticsearch.es_get_doc aws_s3.s3_get_object memory.delete_observations docker.docker_logs +q053,Send an SMS with Twilio,verb_product,twilio.twilio_send_sms,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,401,1.0,1,740,1.0,twilio.twilio_send_sms twilio.twilio_list_messages twilio.twilio_make_call filesystem.list_directory_with_sizes gmail.gmail_send_email puppeteer.puppeteer_click puppeteer.puppeteer_fill discord.discord_send_message puppeteer.puppeteer_screenshot jira.jira_search_issues +q054,Create an Okta user,verb_product,okta.okta_get_user,0.0,1.0,1.0,0.3333,0.5000,3,1,183,1.0,1,329,1.0,1,656,1.0,okta.okta_get_user okta.okta_deactivate_user okta.okta_create_user okta.okta_list_users linear.create_user linear.get_user linear.update_user linear.delete_user slack.slack_get_user_profile asana.asana_create_task +q055,Instant query Prometheus,verb_product,prometheus.prom_query,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,329,1.0,1,680,1.0,prometheus.prom_query prometheus.prom_query_range prometheus.prom_list_metrics postgres.query postgres.explain_query salesforce.salesforce_query sqlite.sqlite_query mysql.mysql_query datadog.datadog_query_metrics notion.notion_query_database +q056,Search Grafana dashboards,verb_product,grafana.grafana_search_dashboards,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,382,1.0,1,690,1.0,grafana.grafana_search_dashboards grafana.grafana_get_dashboard grafana.grafana_create_annotation brave_search.brave_web_search brave_search.brave_local_search confluence.confluence_search github.search_issues github.search_users github.search_orgs elasticsearch.es_search +q057,Trigger a CircleCI pipeline,verb_product,circleci.circleci_trigger_pipeline,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,390,1.0,1,755,1.0,circleci.circleci_trigger_pipeline circleci.circleci_get_pipeline circleci.circleci_list_pipelines gitlab.gitlab_create_pipeline github.actions_run_trigger gitlab.create_pipeline gitlab.get_pipeline gitlab.update_pipeline gitlab.delete_pipeline hubspot.hubspot_create_deal +q058,Run terraform plan,identifier,terraform.tf_plan,1.0,1.0,1.0,1.0000,1.0000,1,1,172,1.0,1,320,1.0,1,717,1.0,terraform.tf_plan terraform.tf_apply terraform.tf_state_list postgres.explain_query docker.docker_run github.create_check_run github.get_check_run github.update_check_run github.delete_check_run github.actions_run_trigger +q059,Create an Asana task,verb_product,asana.asana_get_task,0.0,1.0,1.0,0.5000,0.6309,2,1,267,1.0,1,405,1.0,1,825,1.0,asana.asana_get_task asana.asana_create_task asana.asana_update_task asana.asana_create_project asana.asana_list_tasks clickup.clickup_get_task clickup.clickup_create_task clickup.clickup_update_task okta.okta_create_user github.create_issue +q060,Create a Trello card,verb_product,trello.trello_create_card,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,394,1.0,1,884,1.0,trello.trello_create_card trello.trello_add_comment trello.trello_create_board trello.trello_list_cards github.create_gist github.create_milestone github.create_hook github.create_environment github.create_package github.create_artifact +q061,Create a ClickUp task,verb_product,clickup.clickup_get_task,0.0,1.0,1.0,0.5000,0.6309,2,1,257,1.0,1,364,1.0,1,751,1.0,clickup.clickup_get_task clickup.clickup_create_task clickup.clickup_update_task clickup.clickup_list_tasks puppeteer.puppeteer_click playwright.browser_click asana.asana_get_task asana.asana_create_task asana.asana_update_task github.create_gist +q062,Create a Confluence page,verb_product,confluence.confluence_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,367,1.0,1,703,1.0,confluence.confluence_create_page confluence.confluence_get_page confluence.confluence_update_page confluence.confluence_search notion.notion_get_page notion.notion_create_page notion.notion_update_page playwright.browser_take_screenshot notion.notion_get_comments playwright.browser_press_key +q063,Send a Discord message,verb_product,discord.discord_send_message,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,322,1.0,1,682,1.0,discord.discord_send_message discord.discord_add_reaction discord.discord_get_channel discord.discord_list_channels discord.discord_list_members slack.slack_post_message gmail.gmail_send_email twilio.twilio_send_sms gmail.gmail_get_email gmail.gmail_trash_email +q064,Search Elasticsearch,verb_product,elasticsearch.es_search,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,404,1.0,1,699,1.0,elasticsearch.es_search elasticsearch.es_index_doc elasticsearch.es_get_doc brave_search.brave_web_search brave_search.brave_local_search confluence.confluence_search github.search_issues github.search_users github.search_orgs github.search_commits +q065,Get a Redis key,verb_product,redis.redis_get,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,280,1.0,1,589,1.0,redis.redis_get redis.redis_set redis.redis_delete redis.redis_keys playwright.browser_press_key jira.jira_get_issue aws_s3.s3_get_object figma.figma_get_file figma.figma_get_comments figma.figma_get_file_nodes +q066,Find MongoDB documents,verb_product,mongodb.mongo_find,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,381,1.0,1,738,1.0,mongodb.mongo_find mongodb.mongo_insert mongodb.mongo_update mongodb.mongo_delete mongodb.mongo_list_collections +q067,List SQLite tables,verb_product,sqlite.sqlite_list_tables,1.0,1.0,1.0,1.0000,1.0000,1,1,151,1.0,1,240,1.0,1,514,1.0,sqlite.sqlite_list_tables postgres.list_tables mysql.mysql_list_tables supabase.supabase_list_tables sqlite.sqlite_describe_table sqlite.sqlite_query postgres.list_schemas clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments +q068,Describe a postgres table,verb_product,postgres.describe_table,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,272,1.0,1,597,1.0,postgres.describe_table sqlite.sqlite_describe_table postgres.query postgres.list_tables postgres.list_schemas postgres.explain_query supabase.supabase_select supabase.supabase_insert supabase.supabase_update +q069,github_create_issue,identifier,github.create_issue,1.0,1.0,1.0,1.0000,0.9502,1,1,386,1.0,1,596,1.0,1,1019,1.0,github.create_issue github.issue_read github.issue_write github.add_issue_comment github.update_issue_comment github.assign_copilot_to_issue bitbucket.bitbucket_create_issue sentry.sentry_get_issue gitlab.gitlab_create_issue jira.jira_assign_issue +q070,createIssue,identifier,github.create_issue,1.0,1.0,1.0,1.0000,0.7349,1,1,340,1.0,1,546,1.0,1,961,1.0,github.create_issue github.issue_read bitbucket.bitbucket_create_issue github.issue_write sentry.sentry_get_issue gitlab.gitlab_create_issue jira.jira_assign_issue jira.jira_get_issue linear.linear_create_issue gitlab.gitlab_get_issue +q071,list-directory,identifier,filesystem.create_directory,0.0,1.0,1.0,0.5000,0.6309,2,1,182,1.0,1,296,1.0,1,607,1.0,filesystem.create_directory filesystem.list_directory filesystem.directory_tree filesystem.list_directory_with_sizes filesystem.get_file_info filesystem.list_allowed_directories filesystem.read_file filesystem.read_text_file filesystem.read_media_file filesystem.write_file +q072,sequential thinking next thought,paraphrase,sequential_thinking.sequentialthinking,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,382,1.0,1,797,1.0,sequential_thinking.sequentialthinking +q073,What time is it in Tokyo,paraphrase,time.convert_time,1.0,1.0,1.0,1.0000,0.8597,1,1,217,1.0,1,329,1.0,1,783,1.0,time.convert_time time.get_current_time playwright.browser_wait_for filesystem.list_allowed_directories slack.slack_create_channel google_calendar.gcal_list_events grafana.grafana_create_annotation jira.jira_add_worklog elasticsearch.es_index_doc github.create_milestone +q074,Convert time between timezones,verb_product,time.convert_time,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,335,1.0,1,704,1.0,time.convert_time time.get_current_time salesforce.salesforce_convert_lead playwright.browser_wait_for git.git_diff memory.create_relations google_calendar.gcal_list_events grafana.grafana_create_annotation jira.jira_add_worklog +q075,Star a GitHub repo,verb_product,github.star_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,195,1.0,1,309,1.0,1,604,1.0,github.star_repository git.git_branch git.git_show git.git_checkout git.git_status git.git_diff git.git_commit git.git_add git.git_reset git.git_diff_unstaged +q076,Fork a repository,resource,github.fork_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,235,1.0,1,386,1.0,1,792,1.0,github.fork_repository github.delete_repository github.star_repository github.unstar_repository github.create_repository github.list_repository_collaborators github.get_repository_tree github.list_dependabot_alerts github.list_branches github.list_releases +q077,Get the latest GitHub release,verb_product,github.get_latest_release,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,366,1.0,1,647,1.0,github.get_latest_release github.get_release_by_tag git.git_commit git.git_status git.git_add git.git_checkout git.git_show git.git_branch git.git_diff git.git_reset +q078,Create a gist,resource,github.create_gist,1.0,1.0,1.0,1.0000,1.0000,1,1,229,1.0,1,429,1.0,1,912,1.0,github.create_gist github.update_gist github.get_gist github.create_milestone github.create_hook github.create_environment github.create_package github.create_artifact github.create_runner gitlab.create_pipeline +q079,Search GitHub code,verb_product,github.search_code,1.0,1.0,1.0,1.0000,1.0000,1,1,283,1.0,1,406,1.0,1,697,1.0,github.search_code github.get_code_scanning_alert github.list_code_scanning_alerts git.git_checkout git.git_show git.git_branch git.git_diff git.git_commit git.git_status git.git_add +q080,List GitHub notifications,verb_product,github.list_notifications,1.0,1.0,1.0,1.0000,1.0000,1,1,169,1.0,1,292,1.0,1,585,1.0,github.list_notifications github.mark_all_notifications_read git.git_branch git.git_checkout git.git_show git.git_diff git.git_commit git.git_status git.git_add git.git_reset +q081,Assign Copilot to an issue,verb_product,github.assign_copilot_to_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,434,1.0,1,928,1.0,github.assign_copilot_to_issue jira.jira_assign_issue linear.linear_assign_issue github.request_copilot_review jira.jira_update_issue jira.jira_transition_issue github.create_issue github.add_issue_comment slack.slack_reply_to_thread github.issue_read +q082,Get secret scanning alerts,resource,github.list_secret_scanning_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,295,1.0,1,470,1.0,1,830,1.0,github.list_secret_scanning_alerts github.get_secret_scanning_alert github.list_code_scanning_alerts github.list_dependabot_alerts github.get_code_scanning_alert kubernetes.k8s_get_pod redis.redis_get github.get_discussion github.projects_get github.get_label +q083,Create or update a file on GitHub,verb_product,github.create_or_update_file,1.0,1.0,1.0,1.0000,0.7602,1,1,347,1.0,1,519,1.0,1,869,1.0,github.create_or_update_file github.delete_file git.git_diff git.git_add github.get_file_contents git.git_checkout git.git_show git.git_branch github.get_file_blame git.git_create_branch +q084,List collaborators on a repo,resource,github.list_repository_collaborators,1.0,1.0,1.0,1.0000,1.0000,1,1,278,1.0,1,433,1.0,1,816,1.0,github.list_repository_collaborators github.list_commits trello.trello_list_cards pagerduty.pagerduty_list_oncalls github.get_discussion_comments github.update_issue_comment trello.trello_add_comment github.request_copilot_review notion.notion_get_comments playwright.browser_press_key +q085,Reply in a Slack thread,verb_product,slack.slack_reply_to_thread,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,358,1.0,1,717,1.0,slack.slack_reply_to_thread slack.slack_get_thread_replies slack.slack_list_channels slack.slack_join_channel slack.slack_leave_channel slack.slack_create_channel slack.slack_search_messages slack.slack_upload_file slack.slack_list_users slack.slack_post_message +q086,Add an emoji reaction on Slack,verb_product,slack.slack_add_reaction,1.0,1.0,1.0,1.0000,0.7602,1,1,216,1.0,1,357,1.0,1,709,1.0,slack.slack_add_reaction slack.slack_join_channel slack.slack_leave_channel slack.slack_create_channel slack.slack_search_messages slack.slack_upload_file slack.slack_list_users slack.slack_post_message slack.slack_get_users slack.slack_list_channels +q087,Search Slack messages,verb_product,slack.slack_search_messages,1.0,1.0,1.0,1.0000,1.0000,1,1,213,1.0,1,337,1.0,1,689,1.0,slack.slack_search_messages slack.slack_get_channel_history slack.slack_join_channel slack.slack_leave_channel slack.slack_create_channel slack.slack_upload_file slack.slack_list_users slack.slack_post_message slack.slack_get_users slack.slack_list_channels +q088,Create a Slack channel,verb_product,slack.slack_join_channel,0.0,1.0,1.0,0.3333,0.5000,3,1,196,1.0,1,342,1.0,1,719,1.0,slack.slack_join_channel slack.slack_leave_channel slack.slack_create_channel slack.slack_get_channel_history slack.slack_set_channel_topic slack.slack_post_message slack.slack_upload_file slack.slack_add_reaction slack.slack_search_messages slack.slack_list_users +q089,Transition a Jira issue,verb_product,jira.jira_transition_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,389,1.0,1,763,1.0,jira.jira_transition_issue jira.jira_assign_issue jira.jira_get_issue jira.jira_update_issue jira.jira_create_issue jira.jira_add_comment jira.jira_add_worklog jira.jira_list_projects jira.jira_create_sprint jira.jira_search_issues +q090,Search Jira with JQL,identifier,jira.jira_search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,194,1.0,1,370,1.0,1,764,1.0,jira.jira_search_issues jira.jira_assign_issue jira.jira_list_projects jira.jira_create_sprint jira.jira_update_issue jira.jira_get_issue jira.jira_add_comment jira.jira_add_worklog jira.jira_transition_issue jira.jira_create_issue +q091,Add a worklog to Jira,verb_product,jira.jira_add_worklog,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,350,1.0,1,765,1.0,jira.jira_add_worklog jira.jira_add_comment jira.jira_transition_issue jira.jira_assign_issue jira.jira_list_projects jira.jira_create_sprint jira.jira_update_issue jira.jira_get_issue jira.jira_search_issues jira.jira_create_issue +q092,List Linear teams,verb_product,linear.linear_list_teams,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,380,1.0,1,764,1.0,linear.linear_list_teams linear.linear_list_projects linear.linear_list_issues linear.linear_update_issue linear.linear_create_project linear.linear_create_issue linear.linear_search_issues linear.linear_create_comment linear.linear_assign_issue linear.linear_get_issue +q093,Comment on a Linear issue,verb_product,linear.linear_create_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,334,1.0,1,464,1.0,1,760,1.0,linear.linear_create_comment linear.linear_update_issue linear.linear_create_issue linear.linear_assign_issue linear.linear_get_issue linear.create_comment linear.get_comment linear.update_comment linear.delete_comment linear.linear_list_teams +q094,Move a local file,paraphrase,filesystem.move_file,1.0,1.0,1.0,1.0000,1.0000,1,1,214,1.0,1,354,1.0,1,699,1.0,filesystem.move_file brave_search.brave_local_search filesystem.read_file filesystem.read_text_file figma.figma_get_file filesystem.read_media_file filesystem.write_file slack.create_file slack.get_file slack.update_file +q095,Recursively search files by pattern,paraphrase,filesystem.search_files,1.0,1.0,1.0,1.0000,1.0000,1,1,280,1.0,1,433,1.0,1,705,1.0,filesystem.search_files slack.list_files github.push_files github.get_release_by_tag filesystem.move_file filesystem.list_directory filesystem.list_directory_with_sizes filesystem.directory_tree redis.redis_keys filesystem.read_file +q096,Edit a file in place with old and new string,paraphrase,filesystem.edit_file,1.0,1.0,1.0,1.0000,1.0000,1,1,227,1.0,1,394,1.0,1,682,1.0,filesystem.edit_file playwright.browser_tab_new filesystem.list_directory_with_sizes filesystem.write_file slack.create_file slack.get_file slack.update_file slack.delete_file figma.figma_get_file filesystem.move_file +q097,Read the whole knowledge graph,paraphrase,memory.read_graph,1.0,1.0,1.0,1.0000,1.0000,1,1,152,1.0,1,259,1.0,1,529,1.0,memory.read_graph memory.delete_relations memory.create_entities memory.create_relations memory.add_observations memory.delete_entities memory.open_nodes memory.search_nodes memory.delete_observations filesystem.read_file +q098,Search memory nodes,verb_product,memory.search_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,202,1.0,1,301,1.0,1,546,1.0,memory.search_nodes memory.open_nodes figma.figma_get_file_nodes memory.create_entities memory.create_relations memory.add_observations memory.delete_entities memory.delete_observations memory.delete_relations memory.read_graph +q099,Show unstaged git diff,verb_product,git.git_diff_unstaged,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,303,1.0,1,602,1.0,git.git_diff_unstaged git.git_diff git.git_diff_staged git.git_show git.git_status git.git_checkout git.git_branch git.git_commit git.git_add git.git_log +q100,Stage files with git add,identifier,filesystem.search_files,0.0,1.0,1.0,0.3333,0.5000,3,1,213,1.0,1,336,1.0,1,677,1.0,filesystem.search_files filesystem.list_directory_with_sizes git.git_add git.git_checkout git.git_show git.git_branch git.git_diff git.git_commit git.git_status github.push_files +q101,Create a Shopify product,verb_product,shopify.shopify_create_product,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,380,1.0,1,702,1.0,shopify.shopify_create_product shopify.shopify_get_product shopify.shopify_update_product shopify.shopify_create_order shopify.shopify_list_products shopify.shopify_list_orders shopify.shopify_get_order shopify.shopify_list_customers stripe.stripe_create_product stripe.stripe_create_price +q102,Cancel a Stripe subscription,verb_product,stripe.stripe_cancel_subscription,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,301,1.0,1,632,1.0,stripe.stripe_cancel_subscription stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_list_customers stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_create_product stripe.stripe_create_price stripe.stripe_list_subscriptions stripe.stripe_create_refund +q103,Create a payment intent,resource,stripe.stripe_create_payment_intent,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,417,1.0,1,897,1.0,stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent stripe.stripe_create_refund github.create_gist github.create_milestone github.create_hook github.create_environment github.create_package github.create_artifact github.create_runner +q104,Acknowledge a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,261,1.0,1,405,1.0,1,717,1.0,pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls +q105,Create a Datadog monitor,verb_product,datadog.datadog_get_monitor,0.0,1.0,1.0,0.5000,0.6309,2,1,259,1.0,1,459,1.0,1,927,1.0,datadog.datadog_get_monitor datadog.datadog_create_monitor datadog.datadog_list_monitors datadog.datadog_list_logs datadog.datadog_query_metrics github.create_gist github.create_milestone github.create_hook github.create_environment github.create_package +q106,Resolve a Sentry issue,paraphrase,sentry.sentry_get_issue,0.0,1.0,1.0,0.5000,0.6309,2,1,251,1.0,1,397,1.0,1,817,1.0,sentry.sentry_get_issue sentry.sentry_update_issue sentry.sentry_list_events sentry.sentry_list_projects sentry.sentry_list_issues pagerduty.pagerduty_resolve_incident github.issue_read jira.jira_assign_issue github.issue_write jira.jira_get_issue +q107,Scale a Kubernetes deployment,verb_product,kubernetes.k8s_scale_deployment,1.0,1.0,1.0,1.0000,1.0000,1,1,263,1.0,1,407,1.0,1,726,1.0,kubernetes.k8s_scale_deployment kubernetes.k8s_get_pod kubernetes.k8s_apply kubernetes.k8s_delete kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_list_pods kubernetes.k8s_pod_logs vercel.vercel_get_deployment +q108,Apply a Kubernetes manifest,verb_product,kubernetes.k8s_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,253,1.0,1,407,1.0,1,727,1.0,kubernetes.k8s_apply kubernetes.k8s_get_pod kubernetes.k8s_delete kubernetes.k8s_list_deployments kubernetes.k8s_scale_deployment kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_list_pods kubernetes.k8s_pod_logs terraform.tf_apply +q109,Stop a docker container,verb_product,docker.docker_stop,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,345,1.0,1,641,1.0,docker.docker_stop docker.docker_rm docker.docker_logs docker.docker_run docker.docker_exec docker.docker_images docker.docker_build docker.docker_ps kubernetes.k8s_pod_logs +q110,Build a docker image,verb_product,docker.docker_build,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,317,1.0,1,601,1.0,docker.docker_build docker.docker_run docker.docker_stop docker.docker_rm docker.docker_images docker.docker_ps docker.docker_logs docker.docker_exec filesystem.read_media_file +q111,Click an element in Playwright,verb_product,playwright.browser_click,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,329,1.0,1,566,1.0,playwright.browser_click puppeteer.puppeteer_click playwright.browser_type playwright.browser_take_screenshot playwright.browser_snapshot playwright.browser_navigate playwright.browser_fill_form playwright.browser_evaluate playwright.browser_tab_list playwright.browser_tab_new +q112,Type text into a browser input,paraphrase,playwright.browser_type,1.0,1.0,1.0,1.0000,0.7602,1,1,217,1.0,1,302,1.0,1,554,1.0,playwright.browser_type playwright.browser_navigate playwright.browser_wait_for playwright.browser_tab_close playwright.browser_tab_list playwright.browser_tab_new playwright.browser_snapshot playwright.browser_click playwright.browser_evaluate playwright.browser_fill_form +q113,List GitLab projects,verb_product,gitlab.gitlab_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,179,1.0,1,304,1.0,1,599,1.0,gitlab.gitlab_list_projects git.git_branch git.git_checkout git.git_show git.git_diff git.git_commit git.git_status git.git_add github.projects_list git.git_reset +q114,Create a Bitbucket pull request,verb_product,bitbucket.bitbucket_create_pr,1.0,1.0,1.0,1.0000,1.0000,1,1,313,1.0,1,471,1.0,1,1091,1.0,bitbucket.bitbucket_create_pr bitbucket.bitbucket_list_prs bitbucket.bitbucket_create_issue bitbucket.bitbucket_list_repos github.merge_pull_request github.pull_request_read github.create_pull_request github.update_pull_request github.update_pull_request_branch github.pull_request_review_write +q115,Create a Gmail draft,verb_product,gmail.gmail_create_draft,1.0,1.0,1.0,1.0000,1.0000,1,1,205,1.0,1,352,1.0,1,835,1.0,gmail.gmail_create_draft gmail.gmail_search_emails gmail.gmail_list_labels gmail.gmail_send_email gmail.gmail_get_email gmail.gmail_trash_email gmail.gmail_modify_email github.create_pull_request github.create_gist github.create_milestone +q116,Share a Google Drive file,verb_product,google_drive.gdrive_share_file,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,417,1.0,1,726,1.0,google_drive.gdrive_share_file google_drive.gdrive_update_file google_drive.gdrive_get_file google_drive.gdrive_create_file google_drive.gdrive_search google_drive.gdrive_list_folder gitlab.gitlab_get_file figma.figma_get_file google_calendar.gcal_list_calendars filesystem.write_file +q117,List Google Calendar events,verb_product,google_calendar.gcal_list_events,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,427,1.0,1,784,1.0,google_calendar.gcal_list_events google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event google_calendar.gcal_list_calendars kubernetes.k8s_get_events sentry.sentry_list_events google_drive.gdrive_list_folder google_drive.gdrive_update_file google_drive.gdrive_share_file +q118,Delete a calendar event,resource,google_calendar.gcal_delete_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,427,1.0,1,751,1.0,google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars kubernetes.k8s_delete redis.redis_delete aws_s3.s3_delete_object github.delete_repository github.delete_milestone +q119,List Vercel projects,verb_product,vercel.vercel_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,301,1.0,1,550,1.0,vercel.vercel_list_projects vercel.vercel_list_deployments vercel.vercel_get_deployment vercel.vercel_create_project github.projects_list linear.linear_list_projects jira.jira_list_projects sentry.sentry_list_projects gitlab.gitlab_list_projects github.projects_get +q120,Select rows from Supabase,verb_product,supabase.supabase_select,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,354,1.0,1,709,1.0,supabase.supabase_select supabase.supabase_insert supabase.supabase_update supabase.supabase_invoke_fn supabase.supabase_list_tables elasticsearch.es_get_doc aws_s3.s3_get_object memory.delete_observations docker.docker_logs kubernetes.k8s_pod_logs +q121,Post a Figma comment,verb_product,figma.figma_post_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,368,1.0,1,727,1.0,figma.figma_post_comment figma.figma_get_file figma.figma_get_comments figma.figma_get_images figma.figma_get_file_nodes linear.create_comment linear.get_comment linear.update_comment linear.delete_comment notion.create_comment +q122,List Okta users,verb_product,okta.okta_list_users,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,306,1.0,1,606,1.0,okta.okta_list_users okta.okta_get_user okta.okta_create_user okta.okta_deactivate_user linear.list_users github.search_users slack.slack_list_users notion.notion_list_users notion.list_users_tokens slack.slack_get_users +q123,Range query Prometheus,verb_product,prometheus.prom_query_range,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,329,1.0,1,680,1.0,prometheus.prom_query_range prometheus.prom_query prometheus.prom_list_metrics postgres.query postgres.explain_query salesforce.salesforce_query sqlite.sqlite_query mysql.mysql_query datadog.datadog_query_metrics notion.notion_query_database +q124,Create a Grafana annotation,verb_product,grafana.grafana_create_annotation,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,401,1.0,1,881,1.0,grafana.grafana_create_annotation grafana.grafana_search_dashboards grafana.grafana_get_dashboard github.create_gist github.create_milestone github.create_hook github.create_environment github.create_package github.create_artifact github.create_runner +q125,Apply terraform,identifier,terraform.tf_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,171,1.0,1,292,1.0,1,698,1.0,terraform.tf_apply terraform.tf_plan terraform.tf_state_list kubernetes.k8s_apply +q126,List Asana tasks,verb_product,asana.asana_list_tasks,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,404,1.0,1,682,1.0,asana.asana_list_tasks asana.asana_create_task asana.asana_get_task asana.asana_update_task asana.asana_create_project clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments kubernetes.k8s_list_services aws_s3.s3_list_buckets +q127,Comment on a Trello card,verb_product,trello.trello_add_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,422,1.0,1,757,1.0,trello.trello_add_comment trello.trello_create_card trello.trello_list_cards trello.trello_create_board github.update_issue_comment figma.figma_post_comment linear.create_comment linear.get_comment linear.update_comment linear.delete_comment +q128,Update a ClickUp task,verb_product,clickup.clickup_update_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,364,1.0,1,761,1.0,clickup.clickup_update_task clickup.clickup_get_task clickup.clickup_create_task clickup.clickup_list_tasks puppeteer.puppeteer_click playwright.browser_click asana.asana_update_task asana.asana_get_task asana.asana_create_task mongodb.mongo_update +q129,Search Confluence,verb_product,confluence.confluence_search,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,401,1.0,1,731,1.0,confluence.confluence_search confluence.confluence_get_page confluence.confluence_create_page confluence.confluence_update_page brave_search.brave_web_search brave_search.brave_local_search github.search_issues github.search_users github.search_orgs elasticsearch.es_search +q130,Create a HubSpot deal,verb_product,hubspot.hubspot_create_deal,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,904,1.0,hubspot.hubspot_create_deal hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_list_contacts hubspot.hubspot_search_contacts github.create_gist github.create_repository github.create_branch github.create_issue github.create_pull_request +q131,List Zendesk tickets,verb_product,zendesk.zendesk_list_tickets,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,367,1.0,1,645,1.0,zendesk.zendesk_list_tickets zendesk.zendesk_search zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_create_ticket clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments kubernetes.k8s_list_services aws_s3.s3_list_buckets +q132,List Discord channels,verb_product,discord.discord_list_channels,1.0,1.0,1.0,1.0000,1.0000,1,1,166,1.0,1,321,1.0,1,625,1.0,discord.discord_list_channels discord.discord_list_members discord.discord_get_channel discord.discord_send_message discord.discord_add_reaction slack.slack_list_channels clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments kubernetes.k8s_list_services +q133,Set a Redis key,verb_product,redis.redis_set,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,280,1.0,1,600,1.0,redis.redis_set redis.redis_get redis.redis_delete redis.redis_keys playwright.browser_press_key jira.jira_get_issue slack.slack_set_channel_topic aws_s3.s3_get_object aws_s3.s3_delete_object figma.figma_get_file +q134,Insert MongoDB documents,verb_product,mongodb.mongo_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,256,1.0,1,381,1.0,1,698,1.0,mongodb.mongo_insert mongodb.mongo_update mongodb.mongo_delete mongodb.mongo_find mongodb.mongo_list_collections supabase.supabase_insert +q135,Run a MySQL query,verb_product,postgres.query,0.0,1.0,1.0,0.3333,0.5000,3,1,182,1.0,1,268,1.0,1,543,1.0,postgres.query postgres.explain_query mysql.mysql_query postgres.list_schemas postgres.list_tables postgres.describe_table mysql.mysql_list_tables sqlite.sqlite_query salesforce.salesforce_query prometheus.prom_query +q136,Index a document in Elasticsearch,verb_product,elasticsearch.es_index_doc,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,396,1.0,1,753,1.0,elasticsearch.es_index_doc elasticsearch.es_search elasticsearch.es_get_doc playwright.browser_tab_close github.create_milestone github.get_milestone github.list_milestones github.update_milestone github.delete_milestone github.create_hook +q137,Get an S3 object,verb_product,aws_s3.s3_get_object,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,355,1.0,1,706,1.0,aws_s3.s3_get_object aws_s3.s3_delete_object aws_s3.s3_put_object okta.okta_get_user asana.asana_get_task aws_s3.s3_list_objects elasticsearch.es_search okta.okta_create_user okta.okta_deactivate_user asana.asana_create_task +q138,List Cloudflare zones,verb_product,cloudflare.cf_list_zones,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,338,1.0,1,629,1.0,cloudflare.cf_list_zones cloudflare.cf_purge_cache cloudflare.cf_list_dns cloudflare.cf_create_dns clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments kubernetes.k8s_list_services aws_s3.s3_list_buckets github.projects_list +q139,Create a Stripe invoice,verb_product,stripe.stripe_create_invoice,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,353,1.0,1,632,1.0,stripe.stripe_create_invoice stripe.stripe_create_customer stripe.stripe_create_product stripe.stripe_create_price stripe.stripe_create_refund stripe.stripe_get_customer stripe.stripe_list_customers stripe.stripe_list_invoices stripe.stripe_list_subscriptions stripe.stripe_cancel_subscription +q140,List GitHub Actions workflows,verb_product,github.actions_list,1.0,1.0,1.0,1.0000,1.0000,1,1,293,1.0,1,433,1.0,1,736,1.0,github.actions_list github.actions_get github.actions_run_trigger github.actions_get_job_logs git.git_branch git.git_checkout git.git_show git.git_diff git.git_commit git.git_status +q141,Get GitHub Actions job logs,verb_product,github.actions_get_job_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,450,1.0,1,747,1.0,github.actions_get_job_logs github.actions_get github.actions_list github.actions_run_trigger git.git_log git.git_checkout git.git_show git.git_branch git.git_diff git.git_commit +q142,Create a GitHub repository,verb_product,github.create_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,388,1.0,1,750,1.0,github.create_repository github.delete_repository github.fork_repository github.star_repository github.unstar_repository github.list_repository_collaborators github.get_repository_tree git.git_commit git.git_status git.git_checkout +q143,Checkout a git branch,verb_product,git.git_checkout,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,373,1.0,1,666,1.0,git.git_checkout git.git_create_branch git.git_branch git.git_show github.create_branch git.git_diff git.git_commit git.git_status git.git_add git.git_reset +q144,Show git log,identifier,git.git_log,1.0,1.0,1.0,1.0000,1.0000,1,1,206,1.0,1,320,1.0,1,600,1.0,git.git_log git.git_show git.git_diff git.git_status git.git_checkout git.git_branch git.git_commit git.git_add git.git_diff_unstaged git.git_diff_staged +q145,List postgres schemas,verb_product,postgres.list_schemas,1.0,1.0,1.0,1.0000,1.0000,1,1,158,1.0,1,277,1.0,1,555,1.0,postgres.list_schemas postgres.list_tables postgres.query postgres.describe_table postgres.explain_query clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments kubernetes.k8s_list_services aws_s3.s3_list_buckets +q146,Open nodes in the knowledge graph,verb_product,memory.open_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,163,1.0,1,260,1.0,1,549,1.0,memory.open_nodes memory.read_graph memory.search_nodes memory.create_entities memory.delete_relations memory.create_relations memory.add_observations memory.delete_entities memory.delete_observations figma.figma_get_file_nodes +q147,Get file metadata from the filesystem,paraphrase,filesystem.read_file,0.0,0.0,1.0,0.2500,0.4307,4,2,462,0.0,1,348,1.0,1,688,1.0,filesystem.read_file filesystem.read_text_file filesystem.read_media_file filesystem.get_file_info filesystem.write_file filesystem.edit_file filesystem.move_file filesystem.list_allowed_directories google_drive.gdrive_get_file gitlab.gitlab_get_file +q148,Write a new local file,paraphrase,playwright.browser_tab_new,0.0,1.0,1.0,0.5000,0.6309,2,1,217,1.0,1,366,1.0,1,714,1.0,playwright.browser_tab_new filesystem.write_file brave_search.brave_local_search filesystem.edit_file filesystem.read_file filesystem.read_text_file figma.figma_get_file filesystem.read_media_file slack.create_file slack.get_file +q149,Search GitHub issues,verb_product,github.search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,270,1.0,1,377,1.0,1,670,1.0,github.search_issues github.list_issues git.git_checkout git.git_show git.git_branch git.git_diff git.git_commit git.git_status git.git_add git.git_reset +q150,Get the authenticated GitHub user,paraphrase,git.git_commit,0.0,0.0,0.0,0.0000,0.0000,11,4,771,0.0,3,1050,0.0,2,1402,0.0,git.git_commit git.git_status git.git_add git.git_checkout git.git_show git.git_branch git.git_diff github.get_teams git.git_reset git.git_log diff --git a/experiments/mcp-tool-search/results/bm25_weighted.json b/experiments/mcp-tool-search/results/bm25_weighted.json new file mode 100644 index 00000000..bd245e4a --- /dev/null +++ b/experiments/mcp-tool-search/results/bm25_weighted.json @@ -0,0 +1,6969 @@ +{ + "summary": { + "n": 150, + "index_s": 0.00289805900013107, + "rss_mb": 0.0625, + "p50_ms": 0.4039610000177163, + "p95_ms": 0.6476736999275089, + "recall@1": 0.8533333333333334, + "recall@1_lo": 0.7933333333333333, + "recall@1_hi": 0.9066666666666666, + "recall@3": 0.9733333333333334, + "recall@3_lo": 0.9466666666666667, + "recall@3_hi": 0.9933333333333333, + "recall@5": 0.9866666666666667, + "recall@5_lo": 0.9666666666666667, + "recall@5_hi": 1.0, + "mrr@10": 0.9111111111111111, + "mrr@10_lo": 0.8744444444444444, + "mrr@10_hi": 0.946111111111111, + "ndcg@10": 0.9233077080171845, + "ndcg@10_lo": 0.894350781004159, + "ndcg@10_hi": 0.9506941957337339 + }, + "meta": { + "name": "bm25_weighted", + "needs_model": false, + "needs_api": false, + "notes": "Zed-like: field BM25 + ident split + verb downweight + name/namespace boosts" + }, + "rows": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "github.issue_read", + "github.issue_write", + "github.add_issue_comment", + "github.update_issue_comment", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "git.git_create_branch", + "github.assign_copilot_to_issue" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 388, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 598, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 955, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "kind": "verb_product", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_create_issue", + "jira.jira_create_sprint", + "jira.jira_assign_issue", + "jira.jira_list_projects", + "jira.jira_update_issue", + "jira.jira_get_issue", + "jira.jira_search_issues", + "jira.jira_add_comment", + "jira.jira_add_worklog", + "jira.jira_transition_issue" + ], + "hit1": "jira.jira_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 312, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "kind": "verb_product", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "ranking": [ + "slack.slack_post_message", + "slack.slack_add_reaction", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_create_channel", + "slack.slack_search_messages", + "slack.slack_upload_file", + "slack.slack_list_users", + "slack.slack_get_users", + "slack.slack_list_channels" + ], + "hit1": "slack.slack_post_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "kind": "resource", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "ranking": [ + "github.list_pull_requests", + "github.search_pull_requests", + "gitlab.gitlab_list_merge_requests", + "github.merge_pull_request", + "github.pull_request_read", + "github.create_pull_request", + "github.update_pull_request", + "github.update_pull_request_branch", + "github.pull_request_review_write", + "bitbucket.bitbucket_list_prs" + ], + "hit1": "github.list_pull_requests", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1035, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "ranking": [ + "github.create_pull_request", + "github.merge_pull_request", + "github.pull_request_read", + "github.update_pull_request", + "github.pull_request_review_write", + "github.update_pull_request_branch", + "github.request_copilot_review", + "github.search_pull_requests", + "github.list_pull_requests", + "git.git_checkout" + ], + "hit1": "github.create_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 368, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 653, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1052, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "kind": "verb_product", + "gold": [ + "postgres.query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "postgres.explain_query", + "postgres.list_schemas", + "postgres.list_tables", + "postgres.describe_table", + "mysql.mysql_query", + "sqlite.sqlite_query", + "salesforce.salesforce_query", + "prometheus.prom_query", + "docker.docker_run" + ], + "hit1": "postgres.query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 167, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 590, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "kind": "paraphrase", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "ranking": [ + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "filesystem.write_file", + "filesystem.edit_file", + "filesystem.get_file_info", + "filesystem.move_file", + "brave_search.brave_local_search", + "github.delete_file", + "gitlab.gitlab_get_file" + ], + "hit1": "filesystem.read_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 782, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "kind": "paraphrase", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "ranking": [ + "filesystem.search_files", + "google_drive.gdrive_list_folder", + "slack.list_files", + "github.push_files", + "filesystem.move_file", + "filesystem.list_directory", + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree", + "filesystem.read_file", + "filesystem.read_text_file" + ], + "hit1": "filesystem.search_files", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.16666666666666666, + "ndcg@10": 0.43476305489842704, + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 473, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 730, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 708, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "ranking": [ + "memory.read_graph", + "memory.create_entities", + "memory.delete_relations", + "memory.create_relations", + "memory.add_observations", + "memory.open_nodes", + "memory.search_nodes", + "memory.delete_entities", + "memory.delete_observations", + "time.get_current_time" + ], + "hit1": "memory.read_graph", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.5338931479009518, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 154, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 261, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 524, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "kind": "verb_product", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "ranking": [ + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "filesystem.list_directory_with_sizes", + "jira.jira_search_issues", + "github.update_pull_request_branch", + "memory.search_nodes", + "puppeteer.puppeteer_screenshot", + "puppeteer.puppeteer_navigate", + "puppeteer.puppeteer_click", + "puppeteer.puppeteer_fill" + ], + "hit1": "brave_search.brave_web_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "kind": "verb_product", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "ranking": [ + "fetch.fetch", + "github.get_file_contents", + "playwright.browser_navigate", + "filesystem.read_file", + "puppeteer.puppeteer_navigate", + "git.git_add", + "slack.slack_set_channel_topic", + "github.get_repository_tree", + "git.git_status", + "playwright.browser_snapshot" + ], + "hit1": "fetch.fetch", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 372, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "kind": "identifier", + "gold": [ + "git.git_status" + ], + "alts": [], + "ranking": [ + "git.git_status", + "git.git_show", + "git.git_diff", + "git.git_checkout", + "git.git_branch", + "git.git_commit", + "git.git_add", + "git.git_log", + "git.git_diff_unstaged", + "git.git_diff_staged" + ], + "hit1": "git.git_status", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 304, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "kind": "paraphrase", + "gold": [ + "git.git_commit" + ], + "alts": [], + "ranking": [ + "git.git_commit", + "github.get_commit", + "git.git_diff_staged", + "git.git_reset", + "git.git_log", + "git.git_show", + "github.push_files" + ], + "hit1": "git.git_commit", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_issue", + "linear.linear_update_issue", + "linear.linear_assign_issue", + "linear.linear_get_issue", + "linear.linear_create_comment", + "linear.linear_create_project", + "linear.linear_list_teams", + "linear.linear_list_projects", + "linear.linear_list_issues", + "linear.linear_search_issues" + ], + "hit1": "linear.linear_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 340, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "kind": "paraphrase", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_project", + "linear.linear_create_issue", + "linear.linear_update_issue", + "linear.linear_list_teams", + "linear.linear_list_projects", + "linear.linear_list_issues", + "linear.linear_search_issues", + "linear.linear_create_comment", + "linear.linear_assign_issue", + "linear.linear_get_issue" + ], + "hit1": "linear.linear_create_project", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 352, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 432, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 769, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "ranking": [ + "github.update_issue_comment", + "github.add_issue_comment", + "github.issue_read", + "github.issue_write", + "github.create_issue", + "github.assign_copilot_to_issue", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "git.git_diff" + ], + "hit1": "github.update_issue_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 601, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 938, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "kind": "ambiguous", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "ranking": [ + "github.merge_pull_request", + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_get_merge_request", + "github.pull_request_read", + "github.create_pull_request", + "github.update_pull_request_branch", + "github.update_pull_request", + "gitlab.gitlab_create_merge_request", + "github.pull_request_review_write", + "github.request_copilot_review" + ], + "hit1": "github.merge_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 523, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1121, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "kind": "verb_product", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "ranking": [ + "github.actions_run_trigger", + "github.actions_get", + "github.actions_list", + "github.actions_get_job_logs", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "git.git_diff", + "git.git_commit", + "git.git_status" + ], + "hit1": "github.actions_run_trigger", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 450, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "kind": "resource", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "ranking": [ + "github.list_dependabot_alerts", + "github.get_dependabot_alert", + "github.list_code_scanning_alerts", + "github.list_secret_scanning_alerts", + "kubernetes.k8s_get_pod", + "redis.redis_get", + "github.get_discussion", + "github.projects_get", + "github.get_label", + "aws_s3.s3_get_object" + ], + "hit1": "github.list_dependabot_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 287, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 445, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 798, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "kind": "verb_product", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_issues", + "sentry.sentry_list_projects", + "sentry.sentry_list_events", + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "github.list_issues", + "github.search_issues", + "gitlab.gitlab_list_issues", + "linear.linear_list_issues", + "linear.linear_search_issues" + ], + "hit1": "sentry.sentry_list_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "ranking": [ + "stripe.stripe_get_customer", + "stripe.stripe_create_customer", + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_list_subscriptions", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "stripe.stripe_create_refund", + "stripe.stripe_create_payment_intent", + "stripe.stripe_list_customers" + ], + "hit1": "stripe.stripe_get_customer", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "kind": "paraphrase", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_refund", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_list_customers", + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_create_product", + "stripe.stripe_create_price" + ], + "hit1": "stripe.stripe_create_refund", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "kind": "verb_product", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "ranking": [ + "gmail.gmail_send_email", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "gmail.gmail_modify_email", + "gmail.gmail_search_emails", + "gmail.gmail_create_draft", + "gmail.gmail_list_labels", + "filesystem.list_directory_with_sizes", + "twilio.twilio_send_sms", + "puppeteer.puppeteer_click" + ], + "hit1": "gmail.gmail_send_email", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "kind": "verb_product", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "ranking": [ + "gmail.gmail_search_emails", + "gmail.gmail_create_draft", + "gmail.gmail_list_labels", + "gmail.gmail_send_email", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "gmail.gmail_modify_email", + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "mysql.mysql_list_tables" + ], + "hit1": "gmail.gmail_search_emails", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "kind": "paraphrase", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_update_file", + "google_drive.gdrive_share_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "slack.slack_upload_file", + "gitlab.gitlab_get_file", + "slack.slack_reply_to_thread", + "figma.figma_get_file" + ], + "hit1": "google_drive.gdrive_update_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 496, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 791, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment", + "github.create_package" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 895, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_list_pods", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_apply", + "kubernetes.k8s_delete", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_pod_logs", + "clickup.clickup_list_tasks" + ], + "hit1": "kubernetes.k8s_list_pods", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_get_pod", + "docker.docker_logs", + "datadog.datadog_list_logs", + "github.actions_get_job_logs", + "git.git_log", + "aws_s3.s3_get_object", + "elasticsearch.es_get_doc", + "gitlab.gitlab_get_file", + "github.get_tag" + ], + "hit1": "kubernetes.k8s_pod_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 834, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "ranking": [ + "docker.docker_run", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_exec", + "docker.docker_logs", + "docker.docker_images", + "docker.docker_build", + "docker.docker_ps", + "github.create_check_run", + "github.get_check_run" + ], + "hit1": "docker.docker_run", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 644, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "kind": "verb_product", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "ranking": [ + "docker.docker_logs", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_run", + "docker.docker_exec", + "docker.docker_images", + "docker.docker_build", + "docker.docker_ps", + "kubernetes.k8s_pod_logs", + "datadog.datadog_list_logs" + ], + "hit1": "docker.docker_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "kind": "ambiguous", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "ranking": [ + "playwright.browser_navigate", + "playwright.browser_tab_new", + "puppeteer.puppeteer_navigate", + "playwright.browser_tab_close", + "playwright.browser_tab_list", + "playwright.browser_click", + "playwright.browser_evaluate", + "playwright.browser_snapshot", + "playwright.browser_press_key", + "playwright.browser_wait_for" + ], + "hit1": "playwright.browser_navigate", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 265, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 536, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "kind": "ambiguous", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "ranking": [ + "playwright.browser_take_screenshot", + "puppeteer.puppeteer_screenshot", + "notion.notion_get_page", + "notion.notion_update_page", + "confluence.confluence_get_page", + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "notion.notion_create_page", + "playwright.browser_snapshot", + "playwright.browser_press_key" + ], + "hit1": "playwright.browser_take_screenshot", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 711, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_get_merge_request", + "gitlab.gitlab_create_merge_request", + "github.merge_pull_request", + "gitlab.gitlab_list_merge_requests", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "git.git_create_branch", + "git.git_diff" + ], + "hit1": "gitlab.gitlab_merge_merge_request", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 792, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "kind": "verb_product", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "ranking": [ + "notion.notion_search", + "notion.notion_get_page", + "notion.notion_get_block", + "notion.notion_query_database", + "notion.notion_create_database", + "notion.notion_update_page", + "notion.notion_list_users", + "notion.notion_get_comments", + "notion.notion_append_block", + "notion.notion_create_page" + ], + "hit1": "notion.notion_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "kind": "verb_product", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "ranking": [ + "notion.notion_get_page", + "notion.notion_update_page", + "notion.notion_create_page", + "notion.notion_get_comments", + "notion.notion_create_database", + "notion.notion_search", + "notion.notion_get_block", + "notion.notion_query_database", + "notion.notion_list_users", + "notion.notion_append_block" + ], + "hit1": "notion.notion_get_page", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "kind": "verb_product", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "ranking": [ + "notion.notion_query_database", + "notion.notion_create_database", + "notion.notion_create_page", + "notion.notion_search", + "notion.notion_get_page", + "notion.notion_get_block", + "notion.notion_update_page", + "notion.notion_list_users", + "notion.notion_get_comments", + "notion.notion_append_block" + ], + "hit1": "notion.notion_query_database", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 279, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_search", + "zendesk.zendesk_list_tickets", + "jira.jira_create_issue", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment" + ], + "hit1": "zendesk.zendesk_create_ticket", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 856, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_create_deal", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_search_contacts", + "github.create_gist", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.create_pull_request" + ], + "hit1": "hubspot.hubspot_create_contact", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 904, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "kind": "verb_product", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_convert_lead", + "salesforce.salesforce_create_lead", + "salesforce.salesforce_create_record", + "salesforce.salesforce_query", + "salesforce.salesforce_update_record", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_list_opportunities", + "salesforce.salesforce_get_record", + "github.create_gist", + "github.create_milestone" + ], + "hit1": "salesforce.salesforce_convert_lead", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 787, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "kind": "identifier", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_query", + "postgres.query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "prometheus.prom_query", + "docker.docker_run", + "postgres.explain_query", + "prometheus.prom_query_range", + "datadog.datadog_query_metrics", + "notion.notion_query_database" + ], + "hit1": "salesforce.salesforce_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "kind": "verb_product", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "ranking": [ + "shopify.shopify_list_orders", + "shopify.shopify_list_products", + "shopify.shopify_list_customers", + "shopify.shopify_get_product", + "shopify.shopify_create_product", + "shopify.shopify_update_product", + "shopify.shopify_get_order", + "shopify.shopify_create_order", + "clickup.clickup_list_tasks", + "trello.trello_list_cards" + ], + "hit1": "shopify.shopify_list_orders", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment" + ], + "hit1": "pagerduty.pagerduty_create_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 838, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "kind": "paraphrase", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_list_oncalls", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "twilio.twilio_make_call", + "filesystem.list_allowed_directories", + "trello.trello_add_comment", + "notion.notion_get_comments" + ], + "hit1": "pagerduty.pagerduty_list_oncalls", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "kind": "verb_product", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "ranking": [ + "datadog.datadog_query_metrics", + "datadog.datadog_list_monitors", + "datadog.datadog_create_monitor", + "datadog.datadog_list_logs", + "datadog.datadog_get_monitor", + "prometheus.prom_list_metrics", + "prometheus.prom_query", + "postgres.query", + "postgres.explain_query", + "salesforce.salesforce_query" + ], + "hit1": "datadog.datadog_query_metrics", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "kind": "verb_product", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "ranking": [ + "datadog.datadog_list_logs", + "datadog.datadog_list_monitors", + "datadog.datadog_get_monitor", + "datadog.datadog_create_monitor", + "datadog.datadog_query_metrics", + "docker.docker_logs", + "kubernetes.k8s_pod_logs", + "github.actions_get_job_logs", + "brave_search.brave_web_search", + "brave_search.brave_local_search" + ], + "hit1": "datadog.datadog_list_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 877, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_list_buckets", + "aws_s3.s3_list_objects", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "github.projects_list", + "github.list_milestones", + "github.list_hooks", + "github.list_environments" + ], + "hit1": "aws_s3.s3_list_buckets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 318, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "kind": "paraphrase", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_put_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_get_object", + "slack.slack_upload_file", + "github.assign_copilot_to_issue", + "slack.slack_reply_to_thread", + "github.add_comment_to_pending_review", + "slack.slack_add_reaction", + "twilio.twilio_send_sms", + "gmail.gmail_send_email" + ], + "hit1": "aws_s3.s3_put_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 950, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns" + ], + "hit1": "cloudflare.cf_purge_cache", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "cloudflare.cf_list_zones", + "cloudflare.cf_purge_cache", + "salesforce.salesforce_create_record", + "salesforce.salesforce_update_record", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_get_record", + "git.git_commit", + "github.create_gist" + ], + "hit1": "cloudflare.cf_create_dns", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_deployments", + "vercel.vercel_list_projects", + "vercel.vercel_get_deployment", + "vercel.vercel_create_project", + "kubernetes.k8s_list_deployments", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_services", + "aws_s3.s3_list_buckets", + "github.projects_list" + ], + "hit1": "vercel.vercel_list_deployments", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 579, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "ranking": [ + "supabase.supabase_insert", + "supabase.supabase_select", + "supabase.supabase_update", + "supabase.supabase_invoke_fn", + "supabase.supabase_list_tables", + "mongodb.mongo_insert", + "playwright.browser_type" + ], + "hit1": "supabase.supabase_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "kind": "paraphrase", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "ranking": [ + "figma.figma_get_images", + "figma.figma_get_file", + "figma.figma_get_file_nodes", + "figma.figma_get_comments", + "figma.figma_post_comment", + "docker.docker_images", + "elasticsearch.es_get_doc", + "aws_s3.s3_get_object", + "memory.delete_observations", + "docker.docker_logs" + ], + "hit1": "figma.figma_get_images", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "kind": "verb_product", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "ranking": [ + "twilio.twilio_send_sms", + "twilio.twilio_list_messages", + "twilio.twilio_make_call", + "filesystem.list_directory_with_sizes", + "gmail.gmail_send_email", + "puppeteer.puppeteer_click", + "puppeteer.puppeteer_fill", + "discord.discord_send_message", + "puppeteer.puppeteer_screenshot", + "jira.jira_search_issues" + ], + "hit1": "twilio.twilio_send_sms", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "kind": "verb_product", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "ranking": [ + "okta.okta_get_user", + "okta.okta_deactivate_user", + "okta.okta_create_user", + "okta.okta_list_users", + "linear.create_user", + "linear.get_user", + "linear.update_user", + "linear.delete_user", + "slack.slack_get_user_profile", + "asana.asana_create_task" + ], + "hit1": "okta.okta_get_user", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 656, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query", + "prometheus.prom_query_range", + "prometheus.prom_list_metrics", + "postgres.query", + "postgres.explain_query", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "datadog.datadog_query_metrics", + "notion.notion_query_database" + ], + "hit1": "prometheus.prom_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "kind": "verb_product", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "ranking": [ + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "grafana.grafana_create_annotation", + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "confluence.confluence_search", + "github.search_issues", + "github.search_users", + "github.search_orgs", + "elasticsearch.es_search" + ], + "hit1": "grafana.grafana_search_dashboards", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 690, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "kind": "verb_product", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "ranking": [ + "circleci.circleci_trigger_pipeline", + "circleci.circleci_get_pipeline", + "circleci.circleci_list_pipelines", + "gitlab.gitlab_create_pipeline", + "github.actions_run_trigger", + "gitlab.create_pipeline", + "gitlab.get_pipeline", + "gitlab.update_pipeline", + "gitlab.delete_pipeline", + "hubspot.hubspot_create_deal" + ], + "hit1": "circleci.circleci_trigger_pipeline", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "kind": "identifier", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "ranking": [ + "terraform.tf_plan", + "terraform.tf_apply", + "terraform.tf_state_list", + "postgres.explain_query", + "docker.docker_run", + "github.create_check_run", + "github.get_check_run", + "github.update_check_run", + "github.delete_check_run", + "github.actions_run_trigger" + ], + "hit1": "terraform.tf_plan", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "kind": "verb_product", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "ranking": [ + "asana.asana_get_task", + "asana.asana_create_task", + "asana.asana_update_task", + "asana.asana_create_project", + "asana.asana_list_tasks", + "clickup.clickup_get_task", + "clickup.clickup_create_task", + "clickup.clickup_update_task", + "okta.okta_create_user", + "github.create_issue" + ], + "hit1": "asana.asana_get_task", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "ranking": [ + "trello.trello_create_card", + "trello.trello_add_comment", + "trello.trello_create_board", + "trello.trello_list_cards", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment", + "github.create_package", + "github.create_artifact" + ], + "hit1": "trello.trello_create_card", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 394, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 884, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_get_task", + "clickup.clickup_create_task", + "clickup.clickup_update_task", + "clickup.clickup_list_tasks", + "puppeteer.puppeteer_click", + "playwright.browser_click", + "asana.asana_get_task", + "asana.asana_create_task", + "asana.asana_update_task", + "github.create_gist" + ], + "hit1": "clickup.clickup_get_task", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "kind": "verb_product", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "ranking": [ + "confluence.confluence_create_page", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "confluence.confluence_search", + "notion.notion_get_page", + "notion.notion_create_page", + "notion.notion_update_page", + "playwright.browser_take_screenshot", + "notion.notion_get_comments", + "playwright.browser_press_key" + ], + "hit1": "confluence.confluence_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "kind": "verb_product", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "discord.discord_add_reaction", + "discord.discord_get_channel", + "discord.discord_list_channels", + "discord.discord_list_members", + "slack.slack_post_message", + "gmail.gmail_send_email", + "twilio.twilio_send_sms", + "gmail.gmail_get_email", + "gmail.gmail_trash_email" + ], + "hit1": "discord.discord_send_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_search", + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "confluence.confluence_search", + "github.search_issues", + "github.search_users", + "github.search_orgs", + "github.search_commits" + ], + "hit1": "elasticsearch.es_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "redis.redis_set", + "redis.redis_delete", + "redis.redis_keys", + "playwright.browser_press_key", + "jira.jira_get_issue", + "aws_s3.s3_get_object", + "figma.figma_get_file", + "figma.figma_get_comments", + "figma.figma_get_file_nodes" + ], + "hit1": "redis.redis_get", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 589, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_find", + "mongodb.mongo_insert", + "mongodb.mongo_update", + "mongodb.mongo_delete", + "mongodb.mongo_list_collections" + ], + "hit1": "mongodb.mongo_find", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "kind": "verb_product", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "ranking": [ + "sqlite.sqlite_list_tables", + "postgres.list_tables", + "mysql.mysql_list_tables", + "supabase.supabase_list_tables", + "sqlite.sqlite_describe_table", + "sqlite.sqlite_query", + "postgres.list_schemas", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments" + ], + "hit1": "sqlite.sqlite_list_tables", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 240, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 514, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "kind": "verb_product", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "ranking": [ + "postgres.describe_table", + "sqlite.sqlite_describe_table", + "postgres.query", + "postgres.list_tables", + "postgres.list_schemas", + "postgres.explain_query", + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update" + ], + "hit1": "postgres.describe_table", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 272, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "github.issue_read", + "github.issue_write", + "github.add_issue_comment", + "github.update_issue_comment", + "github.assign_copilot_to_issue", + "bitbucket.bitbucket_create_issue", + "sentry.sentry_get_issue", + "gitlab.gitlab_create_issue", + "jira.jira_assign_issue" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 386, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 596, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1019, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "ranking": [ + "github.create_issue", + "github.issue_read", + "bitbucket.bitbucket_create_issue", + "github.issue_write", + "sentry.sentry_get_issue", + "gitlab.gitlab_create_issue", + "jira.jira_assign_issue", + "jira.jira_get_issue", + "linear.linear_create_issue", + "gitlab.gitlab_get_issue" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7349350438281765, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 340, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 546, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 961, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "kind": "identifier", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "ranking": [ + "filesystem.create_directory", + "filesystem.list_directory", + "filesystem.directory_tree", + "filesystem.list_directory_with_sizes", + "filesystem.get_file_info", + "filesystem.list_allowed_directories", + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "filesystem.write_file" + ], + "hit1": "filesystem.create_directory", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 607, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "kind": "paraphrase", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "ranking": [ + "sequential_thinking.sequentialthinking" + ], + "hit1": "sequential_thinking.sequentialthinking", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "kind": "paraphrase", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "ranking": [ + "time.convert_time", + "time.get_current_time", + "playwright.browser_wait_for", + "filesystem.list_allowed_directories", + "slack.slack_create_channel", + "google_calendar.gcal_list_events", + "grafana.grafana_create_annotation", + "jira.jira_add_worklog", + "elasticsearch.es_index_doc", + "github.create_milestone" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8597186998521972, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 783, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "kind": "verb_product", + "gold": [ + "time.convert_time" + ], + "alts": [], + "ranking": [ + "time.convert_time", + "time.get_current_time", + "salesforce.salesforce_convert_lead", + "playwright.browser_wait_for", + "git.git_diff", + "memory.create_relations", + "google_calendar.gcal_list_events", + "grafana.grafana_create_annotation", + "jira.jira_add_worklog" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "kind": "verb_product", + "gold": [ + "github.star_repository" + ], + "alts": [], + "ranking": [ + "github.star_repository", + "git.git_branch", + "git.git_show", + "git.git_checkout", + "git.git_status", + "git.git_diff", + "git.git_commit", + "git.git_add", + "git.git_reset", + "git.git_diff_unstaged" + ], + "hit1": "github.star_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 309, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 604, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "kind": "resource", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "ranking": [ + "github.fork_repository", + "github.delete_repository", + "github.star_repository", + "github.unstar_repository", + "github.create_repository", + "github.list_repository_collaborators", + "github.get_repository_tree", + "github.list_dependabot_alerts", + "github.list_branches", + "github.list_releases" + ], + "hit1": "github.fork_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 792, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "kind": "verb_product", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "ranking": [ + "github.get_latest_release", + "github.get_release_by_tag", + "git.git_commit", + "git.git_status", + "git.git_add", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "git.git_diff", + "git.git_reset" + ], + "hit1": "github.get_latest_release", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "kind": "resource", + "gold": [ + "github.create_gist" + ], + "alts": [], + "ranking": [ + "github.create_gist", + "github.update_gist", + "github.get_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment", + "github.create_package", + "github.create_artifact", + "github.create_runner", + "gitlab.create_pipeline" + ], + "hit1": "github.create_gist", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 912, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "kind": "verb_product", + "gold": [ + "github.search_code" + ], + "alts": [], + "ranking": [ + "github.search_code", + "github.get_code_scanning_alert", + "github.list_code_scanning_alerts", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "git.git_diff", + "git.git_commit", + "git.git_status", + "git.git_add" + ], + "hit1": "github.search_code", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 406, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 697, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "kind": "verb_product", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "ranking": [ + "github.list_notifications", + "github.mark_all_notifications_read", + "git.git_branch", + "git.git_checkout", + "git.git_show", + "git.git_diff", + "git.git_commit", + "git.git_status", + "git.git_add", + "git.git_reset" + ], + "hit1": "github.list_notifications", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 169, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 585, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "kind": "verb_product", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "ranking": [ + "github.assign_copilot_to_issue", + "jira.jira_assign_issue", + "linear.linear_assign_issue", + "github.request_copilot_review", + "jira.jira_update_issue", + "jira.jira_transition_issue", + "github.create_issue", + "github.add_issue_comment", + "slack.slack_reply_to_thread", + "github.issue_read" + ], + "hit1": "github.assign_copilot_to_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 434, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 928, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "kind": "resource", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "ranking": [ + "github.list_secret_scanning_alerts", + "github.get_secret_scanning_alert", + "github.list_code_scanning_alerts", + "github.list_dependabot_alerts", + "github.get_code_scanning_alert", + "kubernetes.k8s_get_pod", + "redis.redis_get", + "github.get_discussion", + "github.projects_get", + "github.get_label" + ], + "hit1": "github.list_secret_scanning_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 830, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "ranking": [ + "github.create_or_update_file", + "github.delete_file", + "git.git_diff", + "git.git_add", + "github.get_file_contents", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "github.get_file_blame", + "git.git_create_branch" + ], + "hit1": "github.create_or_update_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 347, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 519, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 869, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "kind": "resource", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "ranking": [ + "github.list_repository_collaborators", + "github.list_commits", + "trello.trello_list_cards", + "pagerduty.pagerduty_list_oncalls", + "github.get_discussion_comments", + "github.update_issue_comment", + "trello.trello_add_comment", + "github.request_copilot_review", + "notion.notion_get_comments", + "playwright.browser_press_key" + ], + "hit1": "github.list_repository_collaborators", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 278, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 433, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 816, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "kind": "verb_product", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "ranking": [ + "slack.slack_reply_to_thread", + "slack.slack_get_thread_replies", + "slack.slack_list_channels", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_create_channel", + "slack.slack_search_messages", + "slack.slack_upload_file", + "slack.slack_list_users", + "slack.slack_post_message" + ], + "hit1": "slack.slack_reply_to_thread", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "kind": "verb_product", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "ranking": [ + "slack.slack_add_reaction", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_create_channel", + "slack.slack_search_messages", + "slack.slack_upload_file", + "slack.slack_list_users", + "slack.slack_post_message", + "slack.slack_get_users", + "slack.slack_list_channels" + ], + "hit1": "slack.slack_add_reaction", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 357, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "kind": "verb_product", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "ranking": [ + "slack.slack_search_messages", + "slack.slack_get_channel_history", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_create_channel", + "slack.slack_upload_file", + "slack.slack_list_users", + "slack.slack_post_message", + "slack.slack_get_users", + "slack.slack_list_channels" + ], + "hit1": "slack.slack_search_messages", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 689, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "kind": "verb_product", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "ranking": [ + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_create_channel", + "slack.slack_get_channel_history", + "slack.slack_set_channel_topic", + "slack.slack_post_message", + "slack.slack_upload_file", + "slack.slack_add_reaction", + "slack.slack_search_messages", + "slack.slack_list_users" + ], + "hit1": "slack.slack_join_channel", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 719, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "kind": "verb_product", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_transition_issue", + "jira.jira_assign_issue", + "jira.jira_get_issue", + "jira.jira_update_issue", + "jira.jira_create_issue", + "jira.jira_add_comment", + "jira.jira_add_worklog", + "jira.jira_list_projects", + "jira.jira_create_sprint", + "jira.jira_search_issues" + ], + "hit1": "jira.jira_transition_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "kind": "identifier", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "ranking": [ + "jira.jira_search_issues", + "jira.jira_assign_issue", + "jira.jira_list_projects", + "jira.jira_create_sprint", + "jira.jira_update_issue", + "jira.jira_get_issue", + "jira.jira_add_comment", + "jira.jira_add_worklog", + "jira.jira_transition_issue", + "jira.jira_create_issue" + ], + "hit1": "jira.jira_search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "kind": "verb_product", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "ranking": [ + "jira.jira_add_worklog", + "jira.jira_add_comment", + "jira.jira_transition_issue", + "jira.jira_assign_issue", + "jira.jira_list_projects", + "jira.jira_create_sprint", + "jira.jira_update_issue", + "jira.jira_get_issue", + "jira.jira_search_issues", + "jira.jira_create_issue" + ], + "hit1": "jira.jira_add_worklog", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "kind": "verb_product", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "ranking": [ + "linear.linear_list_teams", + "linear.linear_list_projects", + "linear.linear_list_issues", + "linear.linear_update_issue", + "linear.linear_create_project", + "linear.linear_create_issue", + "linear.linear_search_issues", + "linear.linear_create_comment", + "linear.linear_assign_issue", + "linear.linear_get_issue" + ], + "hit1": "linear.linear_list_teams", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "ranking": [ + "linear.linear_create_comment", + "linear.linear_update_issue", + "linear.linear_create_issue", + "linear.linear_assign_issue", + "linear.linear_get_issue", + "linear.create_comment", + "linear.get_comment", + "linear.update_comment", + "linear.delete_comment", + "linear.linear_list_teams" + ], + "hit1": "linear.linear_create_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 334, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 464, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 760, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "kind": "paraphrase", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "ranking": [ + "filesystem.move_file", + "brave_search.brave_local_search", + "filesystem.read_file", + "filesystem.read_text_file", + "figma.figma_get_file", + "filesystem.read_media_file", + "filesystem.write_file", + "slack.create_file", + "slack.get_file", + "slack.update_file" + ], + "hit1": "filesystem.move_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "kind": "paraphrase", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "ranking": [ + "filesystem.search_files", + "slack.list_files", + "github.push_files", + "github.get_release_by_tag", + "filesystem.move_file", + "filesystem.list_directory", + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree", + "redis.redis_keys", + "filesystem.read_file" + ], + "hit1": "filesystem.search_files", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 280, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 433, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "kind": "paraphrase", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "ranking": [ + "filesystem.edit_file", + "playwright.browser_tab_new", + "filesystem.list_directory_with_sizes", + "filesystem.write_file", + "slack.create_file", + "slack.get_file", + "slack.update_file", + "slack.delete_file", + "figma.figma_get_file", + "filesystem.move_file" + ], + "hit1": "filesystem.edit_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 394, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "ranking": [ + "memory.read_graph", + "memory.delete_relations", + "memory.create_entities", + "memory.create_relations", + "memory.add_observations", + "memory.delete_entities", + "memory.open_nodes", + "memory.search_nodes", + "memory.delete_observations", + "filesystem.read_file" + ], + "hit1": "memory.read_graph", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 259, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 529, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "kind": "verb_product", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "memory.open_nodes", + "figma.figma_get_file_nodes", + "memory.create_entities", + "memory.create_relations", + "memory.add_observations", + "memory.delete_entities", + "memory.delete_observations", + "memory.delete_relations", + "memory.read_graph" + ], + "hit1": "memory.search_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 546, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "kind": "verb_product", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "ranking": [ + "git.git_diff_unstaged", + "git.git_diff", + "git.git_diff_staged", + "git.git_show", + "git.git_status", + "git.git_checkout", + "git.git_branch", + "git.git_commit", + "git.git_add", + "git.git_log" + ], + "hit1": "git.git_diff_unstaged", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 602, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "kind": "identifier", + "gold": [ + "git.git_add" + ], + "alts": [], + "ranking": [ + "filesystem.search_files", + "filesystem.list_directory_with_sizes", + "git.git_add", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "git.git_diff", + "git.git_commit", + "git.git_status", + "github.push_files" + ], + "hit1": "filesystem.search_files", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 336, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 677, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "kind": "verb_product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "ranking": [ + "shopify.shopify_create_product", + "shopify.shopify_get_product", + "shopify.shopify_update_product", + "shopify.shopify_create_order", + "shopify.shopify_list_products", + "shopify.shopify_list_orders", + "shopify.shopify_get_order", + "shopify.shopify_list_customers", + "stripe.stripe_create_product", + "stripe.stripe_create_price" + ], + "hit1": "shopify.shopify_create_product", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "kind": "verb_product", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "ranking": [ + "stripe.stripe_cancel_subscription", + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_list_customers", + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "stripe.stripe_list_subscriptions", + "stripe.stripe_create_refund" + ], + "hit1": "stripe.stripe_cancel_subscription", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "kind": "resource", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_refund", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment", + "github.create_package", + "github.create_artifact", + "github.create_runner" + ], + "hit1": "stripe.stripe_create_payment_intent", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 897, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "kind": "verb_product", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "ranking": [ + "datadog.datadog_get_monitor", + "datadog.datadog_create_monitor", + "datadog.datadog_list_monitors", + "datadog.datadog_list_logs", + "datadog.datadog_query_metrics", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment", + "github.create_package" + ], + "hit1": "datadog.datadog_get_monitor", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 927, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "kind": "paraphrase", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "ranking": [ + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "sentry.sentry_list_events", + "sentry.sentry_list_projects", + "sentry.sentry_list_issues", + "pagerduty.pagerduty_resolve_incident", + "github.issue_read", + "jira.jira_assign_issue", + "github.issue_write", + "jira.jira_get_issue" + ], + "hit1": "sentry.sentry_get_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 817, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_apply", + "kubernetes.k8s_delete", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_pod_logs", + "vercel.vercel_get_deployment" + ], + "hit1": "kubernetes.k8s_scale_deployment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 263, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_apply", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_delete", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_pod_logs", + "terraform.tf_apply" + ], + "hit1": "kubernetes.k8s_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 253, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "ranking": [ + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_run", + "docker.docker_exec", + "docker.docker_images", + "docker.docker_build", + "docker.docker_ps", + "kubernetes.k8s_pod_logs" + ], + "hit1": "docker.docker_stop", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "kind": "verb_product", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "ranking": [ + "docker.docker_build", + "docker.docker_run", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_images", + "docker.docker_ps", + "docker.docker_logs", + "docker.docker_exec", + "filesystem.read_media_file" + ], + "hit1": "docker.docker_build", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "kind": "verb_product", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "ranking": [ + "playwright.browser_click", + "puppeteer.puppeteer_click", + "playwright.browser_type", + "playwright.browser_take_screenshot", + "playwright.browser_snapshot", + "playwright.browser_navigate", + "playwright.browser_fill_form", + "playwright.browser_evaluate", + "playwright.browser_tab_list", + "playwright.browser_tab_new" + ], + "hit1": "playwright.browser_click", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 566, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "kind": "paraphrase", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "ranking": [ + "playwright.browser_type", + "playwright.browser_navigate", + "playwright.browser_wait_for", + "playwright.browser_tab_close", + "playwright.browser_tab_list", + "playwright.browser_tab_new", + "playwright.browser_snapshot", + "playwright.browser_click", + "playwright.browser_evaluate", + "playwright.browser_fill_form" + ], + "hit1": "playwright.browser_type", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 554, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_list_projects", + "git.git_branch", + "git.git_checkout", + "git.git_show", + "git.git_diff", + "git.git_commit", + "git.git_status", + "git.git_add", + "github.projects_list", + "git.git_reset" + ], + "hit1": "gitlab.gitlab_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 304, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 599, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "kind": "verb_product", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "ranking": [ + "bitbucket.bitbucket_create_pr", + "bitbucket.bitbucket_list_prs", + "bitbucket.bitbucket_create_issue", + "bitbucket.bitbucket_list_repos", + "github.merge_pull_request", + "github.pull_request_read", + "github.create_pull_request", + "github.update_pull_request", + "github.update_pull_request_branch", + "github.pull_request_review_write" + ], + "hit1": "bitbucket.bitbucket_create_pr", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 471, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1091, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "kind": "verb_product", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "ranking": [ + "gmail.gmail_create_draft", + "gmail.gmail_search_emails", + "gmail.gmail_list_labels", + "gmail.gmail_send_email", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "gmail.gmail_modify_email", + "github.create_pull_request", + "github.create_gist", + "github.create_milestone" + ], + "hit1": "gmail.gmail_create_draft", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 835, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "kind": "verb_product", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_share_file", + "google_drive.gdrive_update_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "gitlab.gitlab_get_file", + "figma.figma_get_file", + "google_calendar.gcal_list_calendars", + "filesystem.write_file" + ], + "hit1": "google_drive.gdrive_share_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "kind": "verb_product", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_list_events", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_list_calendars", + "kubernetes.k8s_get_events", + "sentry.sentry_list_events", + "google_drive.gdrive_list_folder", + "google_drive.gdrive_update_file", + "google_drive.gdrive_share_file" + ], + "hit1": "google_calendar.gcal_list_events", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 784, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "kubernetes.k8s_delete", + "redis.redis_delete", + "aws_s3.s3_delete_object", + "github.delete_repository", + "github.delete_milestone" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_projects", + "vercel.vercel_list_deployments", + "vercel.vercel_get_deployment", + "vercel.vercel_create_project", + "github.projects_list", + "linear.linear_list_projects", + "jira.jira_list_projects", + "sentry.sentry_list_projects", + "gitlab.gitlab_list_projects", + "github.projects_get" + ], + "hit1": "vercel.vercel_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "ranking": [ + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update", + "supabase.supabase_invoke_fn", + "supabase.supabase_list_tables", + "elasticsearch.es_get_doc", + "aws_s3.s3_get_object", + "memory.delete_observations", + "docker.docker_logs", + "kubernetes.k8s_pod_logs" + ], + "hit1": "supabase.supabase_select", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "kind": "verb_product", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "figma.figma_get_file", + "figma.figma_get_comments", + "figma.figma_get_images", + "figma.figma_get_file_nodes", + "linear.create_comment", + "linear.get_comment", + "linear.update_comment", + "linear.delete_comment", + "notion.create_comment" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "kind": "verb_product", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "ranking": [ + "okta.okta_list_users", + "okta.okta_get_user", + "okta.okta_create_user", + "okta.okta_deactivate_user", + "linear.list_users", + "github.search_users", + "slack.slack_list_users", + "notion.notion_list_users", + "notion.list_users_tokens", + "slack.slack_get_users" + ], + "hit1": "okta.okta_list_users", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 306, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query_range", + "prometheus.prom_query", + "prometheus.prom_list_metrics", + "postgres.query", + "postgres.explain_query", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "datadog.datadog_query_metrics", + "notion.notion_query_database" + ], + "hit1": "prometheus.prom_query_range", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "kind": "verb_product", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "ranking": [ + "grafana.grafana_create_annotation", + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment", + "github.create_package", + "github.create_artifact", + "github.create_runner" + ], + "hit1": "grafana.grafana_create_annotation", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 881, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "kind": "identifier", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "ranking": [ + "terraform.tf_apply", + "terraform.tf_plan", + "terraform.tf_state_list", + "kubernetes.k8s_apply" + ], + "hit1": "terraform.tf_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "kind": "verb_product", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "ranking": [ + "asana.asana_list_tasks", + "asana.asana_create_task", + "asana.asana_get_task", + "asana.asana_update_task", + "asana.asana_create_project", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "aws_s3.s3_list_buckets" + ], + "hit1": "asana.asana_list_tasks", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "ranking": [ + "trello.trello_add_comment", + "trello.trello_create_card", + "trello.trello_list_cards", + "trello.trello_create_board", + "github.update_issue_comment", + "figma.figma_post_comment", + "linear.create_comment", + "linear.get_comment", + "linear.update_comment", + "linear.delete_comment" + ], + "hit1": "trello.trello_add_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 422, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_update_task", + "clickup.clickup_get_task", + "clickup.clickup_create_task", + "clickup.clickup_list_tasks", + "puppeteer.puppeteer_click", + "playwright.browser_click", + "asana.asana_update_task", + "asana.asana_get_task", + "asana.asana_create_task", + "mongodb.mongo_update" + ], + "hit1": "clickup.clickup_update_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 761, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "kind": "verb_product", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "ranking": [ + "confluence.confluence_search", + "confluence.confluence_get_page", + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "github.search_issues", + "github.search_users", + "github.search_orgs", + "elasticsearch.es_search" + ], + "hit1": "confluence.confluence_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 731, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_deal", + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_search_contacts", + "github.create_gist", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.create_pull_request" + ], + "hit1": "hubspot.hubspot_create_deal", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 904, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_search", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_create_ticket", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "aws_s3.s3_list_buckets" + ], + "hit1": "zendesk.zendesk_list_tickets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "kind": "verb_product", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "ranking": [ + "discord.discord_list_channels", + "discord.discord_list_members", + "discord.discord_get_channel", + "discord.discord_send_message", + "discord.discord_add_reaction", + "slack.slack_list_channels", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services" + ], + "hit1": "discord.discord_list_channels", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 166, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 625, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "ranking": [ + "redis.redis_set", + "redis.redis_get", + "redis.redis_delete", + "redis.redis_keys", + "playwright.browser_press_key", + "jira.jira_get_issue", + "slack.slack_set_channel_topic", + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "figma.figma_get_file" + ], + "hit1": "redis.redis_set", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert", + "mongodb.mongo_update", + "mongodb.mongo_delete", + "mongodb.mongo_find", + "mongodb.mongo_list_collections", + "supabase.supabase_insert" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "kind": "verb_product", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "postgres.explain_query", + "mysql.mysql_query", + "postgres.list_schemas", + "postgres.list_tables", + "postgres.describe_table", + "mysql.mysql_list_tables", + "sqlite.sqlite_query", + "salesforce.salesforce_query", + "prometheus.prom_query" + ], + "hit1": "postgres.query", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 268, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 543, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_index_doc", + "elasticsearch.es_search", + "elasticsearch.es_get_doc", + "playwright.browser_tab_close", + "github.create_milestone", + "github.get_milestone", + "github.list_milestones", + "github.update_milestone", + "github.delete_milestone", + "github.create_hook" + ], + "hit1": "elasticsearch.es_index_doc", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_put_object", + "okta.okta_get_user", + "asana.asana_get_task", + "aws_s3.s3_list_objects", + "elasticsearch.es_search", + "okta.okta_create_user", + "okta.okta_deactivate_user", + "asana.asana_create_task" + ], + "hit1": "aws_s3.s3_get_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones", + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_dns", + "cloudflare.cf_create_dns", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "aws_s3.s3_list_buckets", + "github.projects_list" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 629, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_create_customer", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "stripe.stripe_create_refund", + "stripe.stripe_get_customer", + "stripe.stripe_list_customers", + "stripe.stripe_list_invoices", + "stripe.stripe_list_subscriptions", + "stripe.stripe_cancel_subscription" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "kind": "verb_product", + "gold": [ + "github.actions_list" + ], + "alts": [], + "ranking": [ + "github.actions_list", + "github.actions_get", + "github.actions_run_trigger", + "github.actions_get_job_logs", + "git.git_branch", + "git.git_checkout", + "git.git_show", + "git.git_diff", + "git.git_commit", + "git.git_status" + ], + "hit1": "github.actions_list", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 433, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 736, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "kind": "verb_product", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "ranking": [ + "github.actions_get_job_logs", + "github.actions_get", + "github.actions_list", + "github.actions_run_trigger", + "git.git_log", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "git.git_diff", + "git.git_commit" + ], + "hit1": "github.actions_get_job_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 450, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 747, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "kind": "verb_product", + "gold": [ + "github.create_repository" + ], + "alts": [], + "ranking": [ + "github.create_repository", + "github.delete_repository", + "github.fork_repository", + "github.star_repository", + "github.unstar_repository", + "github.list_repository_collaborators", + "github.get_repository_tree", + "git.git_commit", + "git.git_status", + "git.git_checkout" + ], + "hit1": "github.create_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 750, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "kind": "verb_product", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "ranking": [ + "git.git_checkout", + "git.git_create_branch", + "git.git_branch", + "git.git_show", + "github.create_branch", + "git.git_diff", + "git.git_commit", + "git.git_status", + "git.git_add", + "git.git_reset" + ], + "hit1": "git.git_checkout", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 666, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "kind": "identifier", + "gold": [ + "git.git_log" + ], + "alts": [], + "ranking": [ + "git.git_log", + "git.git_show", + "git.git_diff", + "git.git_status", + "git.git_checkout", + "git.git_branch", + "git.git_commit", + "git.git_add", + "git.git_diff_unstaged", + "git.git_diff_staged" + ], + "hit1": "git.git_log", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "kind": "verb_product", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "ranking": [ + "postgres.list_schemas", + "postgres.list_tables", + "postgres.query", + "postgres.describe_table", + "postgres.explain_query", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "aws_s3.s3_list_buckets" + ], + "hit1": "postgres.list_schemas", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 158, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 555, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "kind": "verb_product", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "ranking": [ + "memory.open_nodes", + "memory.read_graph", + "memory.search_nodes", + "memory.create_entities", + "memory.delete_relations", + "memory.create_relations", + "memory.add_observations", + "memory.delete_entities", + "memory.delete_observations", + "figma.figma_get_file_nodes" + ], + "hit1": "memory.open_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 260, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 549, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "kind": "paraphrase", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "ranking": [ + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "filesystem.get_file_info", + "filesystem.write_file", + "filesystem.edit_file", + "filesystem.move_file", + "filesystem.list_allowed_directories", + "google_drive.gdrive_get_file", + "gitlab.gitlab_get_file" + ], + "hit1": "filesystem.read_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 462, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 688, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "kind": "paraphrase", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "ranking": [ + "playwright.browser_tab_new", + "filesystem.write_file", + "brave_search.brave_local_search", + "filesystem.edit_file", + "filesystem.read_file", + "filesystem.read_text_file", + "figma.figma_get_file", + "filesystem.read_media_file", + "slack.create_file", + "slack.get_file" + ], + "hit1": "playwright.browser_tab_new", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "kind": "verb_product", + "gold": [ + "github.search_issues" + ], + "alts": [], + "ranking": [ + "github.search_issues", + "github.list_issues", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "git.git_diff", + "git.git_commit", + "git.git_status", + "git.git_add", + "git.git_reset" + ], + "hit1": "github.search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 270, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 670, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "kind": "paraphrase", + "gold": [ + "github.get_me" + ], + "alts": [], + "ranking": [ + "git.git_commit", + "git.git_status", + "git.git_add", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "git.git_diff", + "github.get_teams", + "git.git_reset", + "git.git_log" + ], + "hit1": "git.git_commit", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 11, + "k3_calls": 4, + "k3_tokens": 771, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 3, + "k5_tokens": 1050, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 1.0, + "k10_calls": 2, + "k10_tokens": 1402, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "cost": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.04, + "p95_calls": 1.0, + "success_within_1": 0.9733333333333334, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 242.9, + "p95_tokens": 349.7499999999999, + "total_tokens": 36435, + "tokens_per_successful_query": 242.9, + "total_tokens_successful": 36435 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.02, + "p95_calls": 1.0, + "success_within_1": 0.9866666666666667, + "success_within_2": 0.9933333333333333, + "success_within_3": 1.0, + "mean_tokens": 382.8466666666667, + "p95_tokens": 521.1999999999999, + "total_tokens": 57427, + "tokens_per_successful_query": 382.8466666666667, + "total_tokens_successful": 57427 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.0066666666666666, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 733.46, + "p95_tokens": 952.75, + "total_tokens": 110019, + "tokens_per_successful_query": 733.46, + "total_tokens_successful": 110019 + } + } +} diff --git a/experiments/mcp-tool-search/results/boolean_and.csv b/experiments/mcp-tool-search/results/boolean_and.csv new file mode 100644 index 00000000..a5711613 --- /dev/null +++ b/experiments/mcp-tool-search/results/boolean_and.csv @@ -0,0 +1,151 @@ +id,query,kind,hit1,recall@1,recall@3,recall@5,mrr@10,ndcg@10,hit_rank,k3_calls,k3_tokens,k3_success1,k5_calls,k5_tokens,k5_success1,k10_calls,k10_tokens,k10_success1,ranking +q001,Create a GitHub issue,verb_product,github.create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,343,1.0,1,448,1.0,1,870,1.0,github.create_issue github.issue_write +q002,Create a Jira ticket,verb_product,jira.jira_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,227,1.0,1,385,1.0,1,800,1.0,jira.jira_create_issue +q003,Send a Slack message,verb_product,,0.0,0.0,0.0,0.0000,0.0000,121,41,10292,0.0,25,10122,0.0,13,10175,0.0, +q004,List open pull requests,resource,,0.0,0.0,0.0,0.0000,0.0000,37,13,3710,0.0,8,3752,0.0,4,3672,0.0, +q005,Open a pull request on GitHub,verb_product,github.create_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,265,1.0,1,423,1.0,1,838,1.0,github.create_pull_request +q006,Run a SQL query against postgres,verb_product,postgres.query,1.0,1.0,1.0,1.0000,1.0000,1,1,167,1.0,1,325,1.0,1,740,1.0,postgres.query +q007,Read a local file from disk,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,76,26,7282,0.0,16,7224,0.0,8,7048,0.0, +q008,What files are in this folder,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,82,28,7711,0.0,17,7530,0.0,9,7616,0.0, +q009,Remember this fact in the knowledge graph,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,89,30,8108,0.0,18,7832,0.0,9,7625,0.0, +q010,Search the web with Brave,verb_product,,0.0,0.0,0.0,0.0000,0.0000,119,40,10063,0.0,24,9727,0.0,12,9475,0.0, +q011,Fetch the contents of a URL,verb_product,,0.0,0.0,0.0,0.0000,0.0000,103,35,8982,0.0,21,8674,0.0,11,8746,0.0, +q012,Show git status,identifier,git.git_status,1.0,1.0,1.0,1.0000,1.0000,1,1,163,1.0,1,321,1.0,1,736,1.0,git.git_status +q013,Commit staged changes,paraphrase,git.git_commit,1.0,1.0,1.0,1.0000,1.0000,1,1,175,1.0,1,333,1.0,1,748,1.0,git.git_commit +q014,Create a Linear issue,verb_product,linear.linear_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,273,1.0,1,378,1.0,1,800,1.0,linear.linear_create_issue linear.linear_create_comment +q015,Make a task in my project tracker Linear,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,136,46,11712,0.0,28,11391,0.0,14,11055,0.0, +q016,Post a comment on a GitHub issue,verb_product,,0.0,0.0,0.0,0.0000,0.0000,34,12,3507,0.0,7,3237,0.0,4,3688,0.0, +q017,Merge the pull request,ambiguous,,0.0,0.0,0.0,0.0000,0.0000,41,14,4072,0.0,9,4373,0.0,5,4766,0.0, +q018,Trigger a GitHub Actions workflow,verb_product,github.actions_run_trigger,1.0,1.0,1.0,1.0000,1.0000,1,1,233,1.0,1,391,1.0,1,806,1.0,github.actions_run_trigger +q019,Get Dependabot alerts,resource,,0.0,0.0,0.0,0.0000,0.0000,51,17,5097,0.0,11,5225,0.0,6,5482,0.0, +q020,List Sentry issues,verb_product,sentry.sentry_list_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,207,1.0,1,365,1.0,1,780,1.0,sentry.sentry_list_issues +q021,Create a Stripe customer,verb_product,stripe.stripe_create_customer,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,315,1.0,1,751,1.0,stripe.stripe_create_customer stripe.stripe_create_payment_intent stripe.stripe_create_invoice +q022,Refund a Stripe payment,paraphrase,stripe.stripe_create_refund,1.0,1.0,1.0,1.0000,1.0000,1,1,181,1.0,1,339,1.0,1,754,1.0,stripe.stripe_create_refund +q023,Send an email with Gmail,verb_product,,0.0,0.0,0.0,0.0000,0.0000,244,82,19588,0.0,49,18841,0.0,25,18655,0.0, +q024,Search my Gmail inbox,verb_product,,0.0,0.0,0.0,0.0000,0.0000,245,82,19506,0.0,49,18792,0.0,25,18630,0.0, +q025,Upload a file to Google Drive,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,253,85,20366,0.0,51,19618,0.0,26,19523,0.0, +q026,Create a calendar event,resource,google_calendar.gcal_create_event,1.0,1.0,1.0,1.0000,1.0000,1,1,232,1.0,1,390,1.0,1,805,1.0,google_calendar.gcal_create_event +q027,List Kubernetes pods,verb_product,kubernetes.k8s_list_pods,1.0,1.0,1.0,1.0000,1.0000,1,1,168,1.0,1,326,1.0,1,741,1.0,kubernetes.k8s_list_pods +q028,Get logs from a k8s pod,verb_product,kubernetes.k8s_pod_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,214,1.0,1,372,1.0,1,787,1.0,kubernetes.k8s_pod_logs +q029,Run a docker container,verb_product,docker.docker_run,1.0,1.0,1.0,1.0000,1.0000,1,1,231,1.0,1,336,1.0,1,758,1.0,docker.docker_run docker.docker_exec +q030,Show docker container logs,verb_product,,0.0,0.0,0.0,0.0000,0.0000,184,62,14888,0.0,37,14342,0.0,19,14250,0.0, +q031,Navigate the browser to a URL,ambiguous,playwright.browser_navigate,1.0,1.0,1.0,1.0000,0.7602,1,1,165,1.0,1,323,1.0,1,738,1.0,playwright.browser_navigate +q032,Take a screenshot of the page,ambiguous,playwright.browser_take_screenshot,1.0,1.0,1.0,1.0000,0.7602,1,1,188,1.0,1,346,1.0,1,761,1.0,playwright.browser_take_screenshot +q033,Create a GitLab merge request,verb_product,gitlab.gitlab_create_merge_request,1.0,1.0,1.0,1.0000,1.0000,1,1,231,1.0,1,389,1.0,1,804,1.0,gitlab.gitlab_create_merge_request +q034,Search Notion pages,verb_product,notion.notion_search,1.0,1.0,1.0,1.0000,1.0000,1,1,181,1.0,1,339,1.0,1,754,1.0,notion.notion_search +q035,Create a Notion page,verb_product,notion.notion_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,358,1.0,1,773,1.0,notion.notion_create_page +q036,Query a Notion database,verb_product,notion.notion_query_database,1.0,1.0,1.0,1.0000,1.0000,1,1,198,1.0,1,356,1.0,1,771,1.0,notion.notion_query_database +q037,Create a Zendesk ticket,verb_product,zendesk.zendesk_create_ticket,1.0,1.0,1.0,1.0000,1.0000,1,1,201,1.0,1,359,1.0,1,774,1.0,zendesk.zendesk_create_ticket +q038,Create a HubSpot contact,verb_product,hubspot.hubspot_create_contact,1.0,1.0,1.0,1.0000,1.0000,1,1,198,1.0,1,356,1.0,1,771,1.0,hubspot.hubspot_create_contact +q039,Create a Salesforce lead,verb_product,salesforce.salesforce_create_lead,1.0,1.0,1.0,1.0000,1.0000,1,1,220,1.0,1,378,1.0,1,793,1.0,salesforce.salesforce_create_lead +q040,Run a SOQL query,identifier,salesforce.salesforce_query,1.0,1.0,1.0,1.0000,1.0000,1,1,173,1.0,1,331,1.0,1,746,1.0,salesforce.salesforce_query +q041,List Shopify orders,verb_product,shopify.shopify_list_orders,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,335,1.0,1,750,1.0,shopify.shopify_list_orders +q042,Create a PagerDuty incident,verb_product,pagerduty.pagerduty_create_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,375,1.0,1,790,1.0,pagerduty.pagerduty_create_incident +q043,Who is on call in PagerDuty,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,243,81,19511,0.0,49,18939,0.0,25,18705,0.0, +q044,Query Datadog metrics,verb_product,datadog.datadog_query_metrics,1.0,1.0,1.0,1.0000,1.0000,1,1,214,1.0,1,372,1.0,1,787,1.0,datadog.datadog_query_metrics +q045,Search Datadog logs,verb_product,datadog.datadog_list_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,370,1.0,1,785,1.0,datadog.datadog_list_logs +q046,List S3 buckets,verb_product,aws_s3.s3_list_buckets,1.0,1.0,1.0,1.0000,1.0000,1,1,153,1.0,1,311,1.0,1,726,1.0,aws_s3.s3_list_buckets +q047,Upload an object to S3,paraphrase,aws_s3.s3_put_object,1.0,1.0,1.0,1.0000,1.0000,1,1,201,1.0,1,359,1.0,1,774,1.0,aws_s3.s3_put_object +q048,Purge Cloudflare cache,verb_product,cloudflare.cf_purge_cache,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,341,1.0,1,756,1.0,cloudflare.cf_purge_cache +q049,Create a Cloudflare DNS record,verb_product,cloudflare.cf_create_dns,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,375,1.0,1,790,1.0,cloudflare.cf_create_dns +q050,List Vercel deployments,verb_product,vercel.vercel_list_deployments,1.0,1.0,1.0,1.0000,1.0000,1,1,171,1.0,1,329,1.0,1,744,1.0,vercel.vercel_list_deployments +q051,Insert a row into Supabase,verb_product,,0.0,0.0,0.0,0.0000,0.0000,308,103,24500,0.0,62,23675,0.0,31,22993,0.0, +q052,Export images from Figma,paraphrase,figma.figma_get_images,1.0,1.0,1.0,1.0000,1.0000,1,1,201,1.0,1,359,1.0,1,774,1.0,figma.figma_get_images +q053,Send an SMS with Twilio,verb_product,,0.0,0.0,0.0,0.0000,0.0000,316,106,25217,0.0,64,24398,0.0,32,23694,0.0, +q054,Create an Okta user,verb_product,okta.okta_create_user,1.0,1.0,1.0,1.0000,1.0000,1,1,165,1.0,1,323,1.0,1,738,1.0,okta.okta_create_user +q055,Instant query Prometheus,verb_product,prometheus.prom_query,1.0,1.0,1.0,1.0000,1.0000,1,1,164,1.0,1,322,1.0,1,737,1.0,prometheus.prom_query +q056,Search Grafana dashboards,verb_product,grafana.grafana_search_dashboards,1.0,1.0,1.0,1.0000,1.0000,1,1,171,1.0,1,329,1.0,1,744,1.0,grafana.grafana_search_dashboards +q057,Trigger a CircleCI pipeline,verb_product,circleci.circleci_trigger_pipeline,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,341,1.0,1,756,1.0,circleci.circleci_trigger_pipeline +q058,Run terraform plan,identifier,terraform.tf_plan,1.0,1.0,1.0,1.0000,1.0000,1,1,155,1.0,1,313,1.0,1,728,1.0,terraform.tf_plan +q059,Create an Asana task,verb_product,asana.asana_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,376,1.0,1,791,1.0,asana.asana_create_task +q060,Create a Trello card,verb_product,trello.trello_create_card,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,362,1.0,1,777,1.0,trello.trello_create_card +q061,Create a ClickUp task,verb_product,clickup.clickup_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,362,1.0,1,777,1.0,clickup.clickup_create_task +q062,Create a Confluence page,verb_product,confluence.confluence_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,362,1.0,1,777,1.0,confluence.confluence_create_page +q063,Send a Discord message,verb_product,discord.discord_send_message,1.0,1.0,1.0,1.0000,1.0000,1,1,179,1.0,1,337,1.0,1,752,1.0,discord.discord_send_message +q064,Search Elasticsearch,verb_product,elasticsearch.es_search,1.0,1.0,1.0,1.0000,1.0000,1,1,176,1.0,1,334,1.0,1,749,1.0,elasticsearch.es_search +q065,Get a Redis key,verb_product,redis.redis_get,1.0,1.0,1.0,1.0000,1.0000,1,1,157,1.0,1,315,1.0,1,730,1.0,redis.redis_get +q066,Find MongoDB documents,verb_product,mongodb.mongo_find,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,346,1.0,1,761,1.0,mongodb.mongo_find +q067,List SQLite tables,verb_product,sqlite.sqlite_list_tables,1.0,1.0,1.0,1.0000,1.0000,1,1,147,1.0,1,305,1.0,1,720,1.0,sqlite.sqlite_list_tables +q068,Describe a postgres table,verb_product,postgres.describe_table,1.0,1.0,1.0,1.0000,1.0000,1,1,176,1.0,1,334,1.0,1,749,1.0,postgres.describe_table +q069,github_create_issue,identifier,github.create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,341,1.0,1,446,1.0,1,868,1.0,github.create_issue github.issue_write +q070,createIssue,identifier,github.create_issue,1.0,1.0,1.0,1.0000,0.9220,1,1,418,1.0,1,600,1.0,1,941,1.0,github.create_issue github.issue_write linear.linear_create_issue linear.linear_create_comment jira.jira_create_issue gitlab.gitlab_create_issue bitbucket.bitbucket_create_issue +q071,list-directory,identifier,filesystem.list_directory,1.0,1.0,1.0,1.0000,1.0000,1,1,185,1.0,1,290,1.0,1,712,1.0,filesystem.list_directory filesystem.list_directory_with_sizes +q072,sequential thinking next thought,paraphrase,sequential_thinking.sequentialthinking,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,382,1.0,1,797,1.0,sequential_thinking.sequentialthinking +q073,What time is it in Tokyo,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,117,39,9838,0.0,24,9751,0.0,12,9487,0.0, +q074,Convert time between timezones,verb_product,time.convert_time,1.0,1.0,1.0,1.0000,1.0000,1,1,191,1.0,1,349,1.0,1,764,1.0,time.convert_time +q075,Star a GitHub repo,verb_product,github.star_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,338,1.0,1,753,1.0,github.star_repository +q076,Fork a repository,resource,github.fork_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,193,1.0,1,351,1.0,1,766,1.0,github.fork_repository +q077,Get the latest GitHub release,verb_product,github.get_latest_release,1.0,1.0,1.0,1.0000,1.0000,1,1,185,1.0,1,343,1.0,1,758,1.0,github.get_latest_release +q078,Create a gist,resource,github.create_gist,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,347,1.0,1,762,1.0,github.create_gist +q079,Search GitHub code,verb_product,github.search_code,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,338,1.0,1,782,1.0,github.search_code +q080,List GitHub notifications,verb_product,github.list_notifications,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,335,1.0,1,750,1.0,github.list_notifications +q081,Assign Copilot to an issue,verb_product,github.assign_copilot_to_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,376,1.0,1,791,1.0,github.assign_copilot_to_issue +q082,Get secret scanning alerts,resource,,0.0,0.0,0.0,0.0000,0.0000,49,17,5097,0.0,10,4866,0.0,5,4766,0.0, +q083,Create or update a file on GitHub,verb_product,,0.0,0.0,0.0,0.0000,0.0000,17,6,1593,0.0,4,1694,0.0,2,1646,0.0, +q084,List collaborators on a repo,resource,github.list_repository_collaborators,1.0,1.0,1.0,1.0000,1.0000,1,1,191,1.0,1,349,1.0,1,764,1.0,github.list_repository_collaborators +q085,Reply in a Slack thread,verb_product,,0.0,0.0,0.0,0.0000,0.0000,122,41,10333,0.0,25,10147,0.0,13,10188,0.0, +q086,Add an emoji reaction on Slack,verb_product,,0.0,0.0,0.0,0.0000,0.0000,123,41,10374,0.0,25,10172,0.0,13,10201,0.0, +q087,Search Slack messages,verb_product,slack.slack_search_messages,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,338,1.0,1,753,1.0,slack.slack_search_messages +q088,Create a Slack channel,verb_product,slack.slack_create_channel,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,342,1.0,1,757,1.0,slack.slack_create_channel +q089,Transition a Jira issue,verb_product,jira.jira_transition_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,181,1.0,1,339,1.0,1,754,1.0,jira.jira_transition_issue +q090,Search Jira with JQL,identifier,jira.jira_search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,185,1.0,1,343,1.0,1,758,1.0,jira.jira_search_issues +q091,Add a worklog to Jira,verb_product,jira.jira_add_worklog,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,362,1.0,1,777,1.0,jira.jira_add_worklog +q092,List Linear teams,verb_product,linear.linear_list_teams,1.0,1.0,1.0,1.0000,1.0000,1,1,145,1.0,1,303,1.0,1,718,1.0,linear.linear_list_teams +q093,Comment on a Linear issue,verb_product,,0.0,0.0,0.0,0.0000,0.0000,141,47,11773,0.0,29,11631,0.0,15,11710,0.0, +q094,Move a local file,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,85,29,7859,0.0,17,7496,0.0,9,7598,0.0, +q095,Recursively search files by pattern,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,86,29,7917,0.0,18,7814,0.0,9,7616,0.0, +q096,Edit a file in place with old and new string,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,80,27,7629,0.0,16,7288,0.0,8,7080,0.0, +q097,Read the whole knowledge graph,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,95,32,8392,0.0,19,8062,0.0,10,8146,0.0, +q098,Search memory nodes,verb_product,memory.search_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,168,1.0,1,326,1.0,1,741,1.0,memory.search_nodes +q099,Show unstaged git diff,verb_product,git.git_diff_unstaged,1.0,1.0,1.0,1.0000,1.0000,1,1,167,1.0,1,325,1.0,1,740,1.0,git.git_diff_unstaged +q100,Stage files with git add,identifier,,0.0,0.0,0.0,0.0000,0.0000,109,37,9343,0.0,22,8966,0.0,11,8735,0.0, +q101,Create a Shopify product,verb_product,shopify.shopify_create_product,1.0,1.0,1.0,1.0000,1.0000,1,1,202,1.0,1,360,1.0,1,775,1.0,shopify.shopify_create_product +q102,Cancel a Stripe subscription,verb_product,stripe.stripe_cancel_subscription,1.0,1.0,1.0,1.0000,1.0000,1,1,162,1.0,1,320,1.0,1,735,1.0,stripe.stripe_cancel_subscription +q103,Create a payment intent,resource,stripe.stripe_create_payment_intent,1.0,1.0,1.0,1.0000,1.0000,1,1,230,1.0,1,335,1.0,1,757,1.0,stripe.stripe_create_payment_intent stripe.stripe_create_refund +q104,Acknowledge a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,185,1.0,1,343,1.0,1,758,1.0,pagerduty.pagerduty_acknowledge_incident +q105,Create a Datadog monitor,verb_product,datadog.datadog_create_monitor,1.0,1.0,1.0,1.0000,1.0000,1,1,229,1.0,1,387,1.0,1,802,1.0,datadog.datadog_create_monitor +q106,Resolve a Sentry issue,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,231,77,18333,0.0,47,17977,0.0,24,17954,0.0, +q107,Scale a Kubernetes deployment,verb_product,kubernetes.k8s_scale_deployment,1.0,1.0,1.0,1.0000,1.0000,1,1,210,1.0,1,368,1.0,1,783,1.0,kubernetes.k8s_scale_deployment +q108,Apply a Kubernetes manifest,verb_product,kubernetes.k8s_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,179,1.0,1,337,1.0,1,752,1.0,kubernetes.k8s_apply +q109,Stop a docker container,verb_product,docker.docker_stop,1.0,1.0,1.0,1.0000,1.0000,1,1,158,1.0,1,316,1.0,1,731,1.0,docker.docker_stop +q110,Build a docker image,verb_product,docker.docker_build,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,335,1.0,1,750,1.0,docker.docker_build +q111,Click an element in Playwright,verb_product,,0.0,0.0,0.0,0.0000,0.0000,168,56,13816,0.0,34,13458,0.0,17,13084,0.0, +q112,Type text into a browser input,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,169,57,14038,0.0,34,13458,0.0,17,13084,0.0, +q113,List GitLab projects,verb_product,gitlab.gitlab_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,163,1.0,1,321,1.0,1,736,1.0,gitlab.gitlab_list_projects +q114,Create a Bitbucket pull request,verb_product,bitbucket.bitbucket_create_pr,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,396,1.0,1,811,1.0,bitbucket.bitbucket_create_pr +q115,Create a Gmail draft,verb_product,gmail.gmail_create_draft,1.0,1.0,1.0,1.0000,1.0000,1,1,193,1.0,1,351,1.0,1,766,1.0,gmail.gmail_create_draft +q116,Share a Google Drive file,verb_product,google_drive.gdrive_share_file,1.0,1.0,1.0,1.0000,1.0000,1,1,199,1.0,1,357,1.0,1,772,1.0,google_drive.gdrive_share_file +q117,List Google Calendar events,verb_product,google_calendar.gcal_list_events,1.0,1.0,1.0,1.0000,1.0000,1,1,201,1.0,1,359,1.0,1,774,1.0,google_calendar.gcal_list_events +q118,Delete a calendar event,resource,google_calendar.gcal_delete_event,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,342,1.0,1,757,1.0,google_calendar.gcal_delete_event +q119,List Vercel projects,verb_product,vercel.vercel_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,151,1.0,1,309,1.0,1,724,1.0,vercel.vercel_list_projects +q120,Select rows from Supabase,verb_product,supabase.supabase_select,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,338,1.0,1,753,1.0,supabase.supabase_select +q121,Post a Figma comment,verb_product,figma.figma_post_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,344,1.0,1,759,1.0,figma.figma_post_comment +q122,List Okta users,verb_product,okta.okta_list_users,1.0,1.0,1.0,1.0000,1.0000,1,1,163,1.0,1,321,1.0,1,736,1.0,okta.okta_list_users +q123,Range query Prometheus,verb_product,prometheus.prom_query_range,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,369,1.0,1,784,1.0,prometheus.prom_query_range +q124,Create a Grafana annotation,verb_product,grafana.grafana_create_annotation,1.0,1.0,1.0,1.0000,1.0000,1,1,198,1.0,1,356,1.0,1,771,1.0,grafana.grafana_create_annotation +q125,Apply terraform,identifier,terraform.tf_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,171,1.0,1,329,1.0,1,744,1.0,terraform.tf_apply +q126,List Asana tasks,verb_product,asana.asana_list_tasks,1.0,1.0,1.0,1.0000,1.0000,1,1,179,1.0,1,337,1.0,1,752,1.0,asana.asana_list_tasks +q127,Comment on a Trello card,verb_product,trello.trello_add_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,185,1.0,1,343,1.0,1,758,1.0,trello.trello_add_comment +q128,Update a ClickUp task,verb_product,clickup.clickup_update_task,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,362,1.0,1,777,1.0,clickup.clickup_update_task +q129,Search Confluence,verb_product,confluence.confluence_search,1.0,1.0,1.0,1.0000,1.0000,1,1,165,1.0,1,323,1.0,1,738,1.0,confluence.confluence_search +q130,Create a HubSpot deal,verb_product,hubspot.hubspot_create_deal,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,362,1.0,1,777,1.0,hubspot.hubspot_create_deal +q131,List Zendesk tickets,verb_product,zendesk.zendesk_list_tickets,1.0,1.0,1.0,1.0000,1.0000,1,1,166,1.0,1,324,1.0,1,739,1.0,zendesk.zendesk_list_tickets +q132,List Discord channels,verb_product,discord.discord_list_channels,1.0,1.0,1.0,1.0000,1.0000,1,1,160,1.0,1,318,1.0,1,733,1.0,discord.discord_list_channels +q133,Set a Redis key,verb_product,redis.redis_set,1.0,1.0,1.0,1.0000,1.0000,1,1,185,1.0,1,343,1.0,1,758,1.0,redis.redis_set +q134,Insert MongoDB documents,verb_product,mongodb.mongo_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,344,1.0,1,759,1.0,mongodb.mongo_insert +q135,Run a MySQL query,verb_product,mysql.mysql_query,1.0,1.0,1.0,1.0000,1.0000,1,1,166,1.0,1,324,1.0,1,739,1.0,mysql.mysql_query +q136,Index a document in Elasticsearch,verb_product,elasticsearch.es_index_doc,1.0,1.0,1.0,1.0000,1.0000,1,1,198,1.0,1,356,1.0,1,771,1.0,elasticsearch.es_index_doc +q137,Get an S3 object,verb_product,aws_s3.s3_get_object,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,342,1.0,1,757,1.0,aws_s3.s3_get_object +q138,List Cloudflare zones,verb_product,cloudflare.cf_list_zones,1.0,1.0,1.0,1.0000,1.0000,1,1,149,1.0,1,307,1.0,1,722,1.0,cloudflare.cf_list_zones +q139,Create a Stripe invoice,verb_product,stripe.stripe_create_invoice,1.0,1.0,1.0,1.0000,1.0000,1,1,160,1.0,1,318,1.0,1,733,1.0,stripe.stripe_create_invoice +q140,List GitHub Actions workflows,verb_product,github.actions_list,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,338,1.0,1,753,1.0,github.actions_list +q141,Get GitHub Actions job logs,verb_product,github.actions_get_job_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,366,1.0,1,781,1.0,github.actions_get_job_logs +q142,Create a GitHub repository,verb_product,github.create_or_update_file,0.0,1.0,1.0,0.5000,0.6309,2,1,346,1.0,1,532,1.0,1,938,1.0,github.create_or_update_file github.create_repository github.create_branch github.create_issue +q143,Checkout a git branch,verb_product,git.git_checkout,1.0,1.0,1.0,1.0000,1.0000,1,1,174,1.0,1,332,1.0,1,747,1.0,git.git_checkout +q144,Show git log,identifier,git.git_log,1.0,1.0,1.0,1.0000,1.0000,1,1,173,1.0,1,331,1.0,1,746,1.0,git.git_log +q145,List postgres schemas,verb_product,postgres.list_schemas,1.0,1.0,1.0,1.0000,1.0000,1,1,148,1.0,1,306,1.0,1,721,1.0,postgres.list_schemas +q146,Open nodes in the knowledge graph,verb_product,memory.open_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,164,1.0,1,322,1.0,1,737,1.0,memory.open_nodes +q147,Get file metadata from the filesystem,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,87,29,7917,0.0,18,7814,0.0,9,7616,0.0, +q148,Write a new local file,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,79,27,7494,0.0,16,7208,0.0,8,7040,0.0, +q149,Search GitHub issues,verb_product,github.search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,198,1.0,1,356,1.0,1,771,1.0,github.search_issues +q150,Get the authenticated GitHub user,paraphrase,github.get_me,1.0,1.0,1.0,1.0000,1.0000,1,1,181,1.0,1,379,1.0,1,784,1.0,github.get_me github.get_teams diff --git a/experiments/mcp-tool-search/results/boolean_and.json b/experiments/mcp-tool-search/results/boolean_and.json new file mode 100644 index 00000000..164b29ca --- /dev/null +++ b/experiments/mcp-tool-search/results/boolean_and.json @@ -0,0 +1,5613 @@ +{ + "summary": { + "n": 150, + "index_s": 0.0, + "rss_mb": 0.0234375, + "p50_ms": 0.24348450006073108, + "p95_ms": 0.27677745001142284, + "recall@1": 0.7666666666666667, + "recall@1_lo": 0.7, + "recall@1_hi": 0.8333333333333334, + "recall@3": 0.7733333333333333, + "recall@3_lo": 0.7066666666666667, + "recall@3_hi": 0.84, + "recall@5": 0.7733333333333333, + "recall@5_lo": 0.7066666666666667, + "recall@5_hi": 0.84, + "mrr@10": 0.77, + "mrr@10_lo": 0.7033333333333334, + "mrr@10_hi": 0.84, + "ndcg@10": 0.7671556541472715, + "ndcg@10_lo": 0.7010483146131327, + "ndcg@10_hi": 0.8352656594616827 + }, + "meta": { + "name": "boolean_and", + "needs_model": false, + "needs_api": false, + "notes": "all query tokens must match" + }, + "rows": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "github.issue_write" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 343, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 448, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 870, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "kind": "verb_product", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_create_issue" + ], + "hit1": "jira.jira_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 800, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "kind": "verb_product", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 121, + "k3_calls": 41, + "k3_tokens": 10292, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 25, + "k5_tokens": 10122, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 13, + "k10_tokens": 10175, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "kind": "resource", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 37, + "k3_calls": 13, + "k3_tokens": 3710, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 8, + "k5_tokens": 3752, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3672, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "ranking": [ + "github.create_pull_request" + ], + "hit1": "github.create_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 265, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 423, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 838, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "kind": "verb_product", + "gold": [ + "postgres.query" + ], + "alts": [], + "ranking": [ + "postgres.query" + ], + "hit1": "postgres.query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 167, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 325, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "kind": "paraphrase", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 76, + "k3_calls": 26, + "k3_tokens": 7282, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 16, + "k5_tokens": 7224, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 8, + "k10_tokens": 7048, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "kind": "paraphrase", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 82, + "k3_calls": 28, + "k3_tokens": 7711, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 17, + "k5_tokens": 7530, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7616, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 89, + "k3_calls": 30, + "k3_tokens": 8108, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7832, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7625, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "kind": "verb_product", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 119, + "k3_calls": 40, + "k3_tokens": 10063, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 24, + "k5_tokens": 9727, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9475, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "kind": "verb_product", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 103, + "k3_calls": 35, + "k3_tokens": 8982, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 21, + "k5_tokens": 8674, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 11, + "k10_tokens": 8746, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q012", + "query": "Show git status", + "kind": "identifier", + "gold": [ + "git.git_status" + ], + "alts": [], + "ranking": [ + "git.git_status" + ], + "hit1": "git.git_status", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 736, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "kind": "paraphrase", + "gold": [ + "git.git_commit" + ], + "alts": [], + "ranking": [ + "git.git_commit" + ], + "hit1": "git.git_commit", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 175, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 748, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_issue", + "linear.linear_create_comment" + ], + "hit1": "linear.linear_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 273, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 800, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "kind": "paraphrase", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 136, + "k3_calls": 46, + "k3_tokens": 11712, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 28, + "k5_tokens": 11391, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 14, + "k10_tokens": 11055, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 34, + "k3_calls": 12, + "k3_tokens": 3507, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 7, + "k5_tokens": 3237, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3688, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "kind": "ambiguous", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 41, + "k3_calls": 14, + "k3_tokens": 4072, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 9, + "k5_tokens": 4373, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 5, + "k10_tokens": 4766, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "kind": "verb_product", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "ranking": [ + "github.actions_run_trigger" + ], + "hit1": "github.actions_run_trigger", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 233, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 806, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "kind": "resource", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 51, + "k3_calls": 17, + "k3_tokens": 5097, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 11, + "k5_tokens": 5225, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 6, + "k10_tokens": 5482, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "kind": "verb_product", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_issues" + ], + "hit1": "sentry.sentry_list_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 207, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 365, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 780, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_customer", + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_invoice" + ], + "hit1": "stripe.stripe_create_customer", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "kind": "paraphrase", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_refund" + ], + "hit1": "stripe.stripe_create_refund", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 181, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 754, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "kind": "verb_product", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 244, + "k3_calls": 82, + "k3_tokens": 19588, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 49, + "k5_tokens": 18841, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 25, + "k10_tokens": 18655, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "kind": "verb_product", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 245, + "k3_calls": 82, + "k3_tokens": 19506, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 49, + "k5_tokens": 18792, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 25, + "k10_tokens": 18630, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "kind": "paraphrase", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 253, + "k3_calls": 85, + "k3_tokens": 20366, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 51, + "k5_tokens": 19618, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 26, + "k10_tokens": 19523, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_create_event" + ], + "hit1": "google_calendar.gcal_create_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 805, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_list_pods" + ], + "hit1": "kubernetes.k8s_list_pods", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 168, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 326, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 741, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_pod_logs" + ], + "hit1": "kubernetes.k8s_pod_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 372, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 787, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "ranking": [ + "docker.docker_run", + "docker.docker_exec" + ], + "hit1": "docker.docker_run", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 336, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "kind": "verb_product", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 184, + "k3_calls": 62, + "k3_tokens": 14888, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 37, + "k5_tokens": 14342, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 19, + "k10_tokens": 14250, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "kind": "ambiguous", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "ranking": [ + "playwright.browser_navigate" + ], + "hit1": "playwright.browser_navigate", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 165, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 323, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "kind": "ambiguous", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "ranking": [ + "playwright.browser_take_screenshot" + ], + "hit1": "playwright.browser_take_screenshot", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 346, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 761, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_create_merge_request" + ], + "hit1": "gitlab.gitlab_create_merge_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 804, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "kind": "verb_product", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "ranking": [ + "notion.notion_search" + ], + "hit1": "notion.notion_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 181, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 754, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "kind": "verb_product", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "ranking": [ + "notion.notion_create_page" + ], + "hit1": "notion.notion_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 773, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "kind": "verb_product", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "ranking": [ + "notion.notion_query_database" + ], + "hit1": "notion.notion_query_database", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 198, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 771, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_create_ticket" + ], + "hit1": "zendesk.zendesk_create_ticket", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 201, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 359, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_contact" + ], + "hit1": "hubspot.hubspot_create_contact", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 198, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 771, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "kind": "verb_product", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_create_lead" + ], + "hit1": "salesforce.salesforce_create_lead", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 220, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 793, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "kind": "identifier", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_query" + ], + "hit1": "salesforce.salesforce_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 173, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "kind": "verb_product", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "ranking": [ + "shopify.shopify_list_orders" + ], + "hit1": "shopify.shopify_list_orders", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 750, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_create_incident" + ], + "hit1": "pagerduty.pagerduty_create_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 790, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "kind": "paraphrase", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 243, + "k3_calls": 81, + "k3_tokens": 19511, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 49, + "k5_tokens": 18939, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 25, + "k10_tokens": 18705, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "kind": "verb_product", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "ranking": [ + "datadog.datadog_query_metrics" + ], + "hit1": "datadog.datadog_query_metrics", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 372, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 787, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "kind": "verb_product", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "ranking": [ + "datadog.datadog_list_logs" + ], + "hit1": "datadog.datadog_list_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 785, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_list_buckets" + ], + "hit1": "aws_s3.s3_list_buckets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 153, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 311, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "kind": "paraphrase", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_put_object" + ], + "hit1": "aws_s3.s3_put_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 201, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 359, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_purge_cache" + ], + "hit1": "cloudflare.cf_purge_cache", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 756, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_create_dns" + ], + "hit1": "cloudflare.cf_create_dns", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 790, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_deployments" + ], + "hit1": "vercel.vercel_list_deployments", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 744, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 308, + "k3_calls": 103, + "k3_tokens": 24500, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 62, + "k5_tokens": 23675, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 31, + "k10_tokens": 22993, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "kind": "paraphrase", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "ranking": [ + "figma.figma_get_images" + ], + "hit1": "figma.figma_get_images", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 201, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 359, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "kind": "verb_product", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 316, + "k3_calls": 106, + "k3_tokens": 25217, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 64, + "k5_tokens": 24398, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 32, + "k10_tokens": 23694, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "kind": "verb_product", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "ranking": [ + "okta.okta_create_user" + ], + "hit1": "okta.okta_create_user", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 165, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 323, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query" + ], + "hit1": "prometheus.prom_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 164, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "kind": "verb_product", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "ranking": [ + "grafana.grafana_search_dashboards" + ], + "hit1": "grafana.grafana_search_dashboards", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 744, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "kind": "verb_product", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "ranking": [ + "circleci.circleci_trigger_pipeline" + ], + "hit1": "circleci.circleci_trigger_pipeline", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 756, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "kind": "identifier", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "ranking": [ + "terraform.tf_plan" + ], + "hit1": "terraform.tf_plan", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 155, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 313, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "kind": "verb_product", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "ranking": [ + "asana.asana_create_task" + ], + "hit1": "asana.asana_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 376, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 791, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "ranking": [ + "trello.trello_create_card" + ], + "hit1": "trello.trello_create_card", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_create_task" + ], + "hit1": "clickup.clickup_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "kind": "verb_product", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "ranking": [ + "confluence.confluence_create_page" + ], + "hit1": "confluence.confluence_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "kind": "verb_product", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message" + ], + "hit1": "discord.discord_send_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 752, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_search" + ], + "hit1": "elasticsearch.es_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 176, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 749, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "ranking": [ + "redis.redis_get" + ], + "hit1": "redis.redis_get", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 157, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 730, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_find" + ], + "hit1": "mongodb.mongo_find", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 346, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 761, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "kind": "verb_product", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "ranking": [ + "sqlite.sqlite_list_tables" + ], + "hit1": "sqlite.sqlite_list_tables", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 147, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 305, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 720, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "kind": "verb_product", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "ranking": [ + "postgres.describe_table" + ], + "hit1": "postgres.describe_table", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 176, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 749, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "github.issue_write" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 341, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 446, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 868, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "ranking": [ + "github.create_issue", + "github.issue_write", + "linear.linear_create_issue", + "linear.linear_create_comment", + "jira.jira_create_issue", + "gitlab.gitlab_create_issue", + "bitbucket.bitbucket_create_issue" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9220433016555235, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 418, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 600, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 941, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "kind": "identifier", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "ranking": [ + "filesystem.list_directory", + "filesystem.list_directory_with_sizes" + ], + "hit1": "filesystem.list_directory", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 290, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 712, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "kind": "paraphrase", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "ranking": [ + "sequential_thinking.sequentialthinking" + ], + "hit1": "sequential_thinking.sequentialthinking", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "kind": "paraphrase", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 117, + "k3_calls": 39, + "k3_tokens": 9838, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 24, + "k5_tokens": 9751, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9487, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "kind": "verb_product", + "gold": [ + "time.convert_time" + ], + "alts": [], + "ranking": [ + "time.convert_time" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 191, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "kind": "verb_product", + "gold": [ + "github.star_repository" + ], + "alts": [], + "ranking": [ + "github.star_repository" + ], + "hit1": "github.star_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "kind": "resource", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "ranking": [ + "github.fork_repository" + ], + "hit1": "github.fork_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 351, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 766, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "kind": "verb_product", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "ranking": [ + "github.get_latest_release" + ], + "hit1": "github.get_latest_release", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "kind": "resource", + "gold": [ + "github.create_gist" + ], + "alts": [], + "ranking": [ + "github.create_gist" + ], + "hit1": "github.create_gist", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 347, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 762, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "kind": "verb_product", + "gold": [ + "github.search_code" + ], + "alts": [], + "ranking": [ + "github.search_code" + ], + "hit1": "github.search_code", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 782, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "kind": "verb_product", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "ranking": [ + "github.list_notifications" + ], + "hit1": "github.list_notifications", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 750, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "kind": "verb_product", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "ranking": [ + "github.assign_copilot_to_issue" + ], + "hit1": "github.assign_copilot_to_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 376, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 791, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "kind": "resource", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 49, + "k3_calls": 17, + "k3_tokens": 5097, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 10, + "k5_tokens": 4866, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 5, + "k10_tokens": 4766, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 17, + "k3_calls": 6, + "k3_tokens": 1593, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 4, + "k5_tokens": 1694, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 2, + "k10_tokens": 1646, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "kind": "resource", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "ranking": [ + "github.list_repository_collaborators" + ], + "hit1": "github.list_repository_collaborators", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 191, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "kind": "verb_product", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 122, + "k3_calls": 41, + "k3_tokens": 10333, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 25, + "k5_tokens": 10147, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 13, + "k10_tokens": 10188, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "kind": "verb_product", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 123, + "k3_calls": 41, + "k3_tokens": 10374, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 25, + "k5_tokens": 10172, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 13, + "k10_tokens": 10201, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "kind": "verb_product", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "ranking": [ + "slack.slack_search_messages" + ], + "hit1": "slack.slack_search_messages", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "kind": "verb_product", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "ranking": [ + "slack.slack_create_channel" + ], + "hit1": "slack.slack_create_channel", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "kind": "verb_product", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_transition_issue" + ], + "hit1": "jira.jira_transition_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 181, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 754, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "kind": "identifier", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "ranking": [ + "jira.jira_search_issues" + ], + "hit1": "jira.jira_search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "kind": "verb_product", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "ranking": [ + "jira.jira_add_worklog" + ], + "hit1": "jira.jira_add_worklog", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "kind": "verb_product", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "ranking": [ + "linear.linear_list_teams" + ], + "hit1": "linear.linear_list_teams", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 145, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 141, + "k3_calls": 47, + "k3_tokens": 11773, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 29, + "k5_tokens": 11631, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11710, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q094", + "query": "Move a local file", + "kind": "paraphrase", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 85, + "k3_calls": 29, + "k3_tokens": 7859, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 17, + "k5_tokens": 7496, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7598, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "kind": "paraphrase", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 86, + "k3_calls": 29, + "k3_tokens": 7917, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7814, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7616, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "kind": "paraphrase", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 80, + "k3_calls": 27, + "k3_tokens": 7629, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 16, + "k5_tokens": 7288, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 8, + "k10_tokens": 7080, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 95, + "k3_calls": 32, + "k3_tokens": 8392, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 19, + "k5_tokens": 8062, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 10, + "k10_tokens": 8146, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "kind": "verb_product", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes" + ], + "hit1": "memory.search_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 168, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 326, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 741, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "kind": "verb_product", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "ranking": [ + "git.git_diff_unstaged" + ], + "hit1": "git.git_diff_unstaged", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 167, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 325, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "kind": "identifier", + "gold": [ + "git.git_add" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 109, + "k3_calls": 37, + "k3_tokens": 9343, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 22, + "k5_tokens": 8966, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 11, + "k10_tokens": 8735, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "kind": "verb_product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "ranking": [ + "shopify.shopify_create_product" + ], + "hit1": "shopify.shopify_create_product", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 775, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "kind": "verb_product", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "ranking": [ + "stripe.stripe_cancel_subscription" + ], + "hit1": "stripe.stripe_cancel_subscription", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 162, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 735, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "kind": "resource", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_refund" + ], + "hit1": "stripe.stripe_create_payment_intent", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "kind": "verb_product", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "ranking": [ + "datadog.datadog_create_monitor" + ], + "hit1": "datadog.datadog_create_monitor", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 387, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 802, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "kind": "paraphrase", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 231, + "k3_calls": 77, + "k3_tokens": 18333, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 47, + "k5_tokens": 17977, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 24, + "k10_tokens": 17954, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_scale_deployment" + ], + "hit1": "kubernetes.k8s_scale_deployment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 210, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 783, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_apply" + ], + "hit1": "kubernetes.k8s_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 752, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "ranking": [ + "docker.docker_stop" + ], + "hit1": "docker.docker_stop", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 158, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 731, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "kind": "verb_product", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "ranking": [ + "docker.docker_build" + ], + "hit1": "docker.docker_build", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 750, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "kind": "verb_product", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 168, + "k3_calls": 56, + "k3_tokens": 13816, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 34, + "k5_tokens": 13458, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 17, + "k10_tokens": 13084, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "kind": "paraphrase", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 169, + "k3_calls": 57, + "k3_tokens": 14038, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 34, + "k5_tokens": 13458, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 17, + "k10_tokens": 13084, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_list_projects" + ], + "hit1": "gitlab.gitlab_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 736, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "kind": "verb_product", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "ranking": [ + "bitbucket.bitbucket_create_pr" + ], + "hit1": "bitbucket.bitbucket_create_pr", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 811, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "kind": "verb_product", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "ranking": [ + "gmail.gmail_create_draft" + ], + "hit1": "gmail.gmail_create_draft", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 351, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 766, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "kind": "verb_product", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_share_file" + ], + "hit1": "google_drive.gdrive_share_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 199, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 357, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 772, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "kind": "verb_product", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_list_events" + ], + "hit1": "google_calendar.gcal_list_events", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 201, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 359, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_projects" + ], + "hit1": "vercel.vercel_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 309, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "ranking": [ + "supabase.supabase_select" + ], + "hit1": "supabase.supabase_select", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "kind": "verb_product", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "kind": "verb_product", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "ranking": [ + "okta.okta_list_users" + ], + "hit1": "okta.okta_list_users", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 736, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query_range" + ], + "hit1": "prometheus.prom_query_range", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 369, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 784, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "kind": "verb_product", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "ranking": [ + "grafana.grafana_create_annotation" + ], + "hit1": "grafana.grafana_create_annotation", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 198, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 771, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "kind": "identifier", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "ranking": [ + "terraform.tf_apply" + ], + "hit1": "terraform.tf_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 744, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "kind": "verb_product", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "ranking": [ + "asana.asana_list_tasks" + ], + "hit1": "asana.asana_list_tasks", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 752, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "ranking": [ + "trello.trello_add_comment" + ], + "hit1": "trello.trello_add_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_update_task" + ], + "hit1": "clickup.clickup_update_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "kind": "verb_product", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "ranking": [ + "confluence.confluence_search" + ], + "hit1": "confluence.confluence_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 165, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 323, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_deal" + ], + "hit1": "hubspot.hubspot_create_deal", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_list_tickets" + ], + "hit1": "zendesk.zendesk_list_tickets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 166, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 739, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "kind": "verb_product", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "ranking": [ + "discord.discord_list_channels" + ], + "hit1": "discord.discord_list_channels", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 160, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 318, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "ranking": [ + "redis.redis_set" + ], + "hit1": "redis.redis_set", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "kind": "verb_product", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "ranking": [ + "mysql.mysql_query" + ], + "hit1": "mysql.mysql_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 166, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 739, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_index_doc" + ], + "hit1": "elasticsearch.es_index_doc", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 198, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 771, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_get_object" + ], + "hit1": "aws_s3.s3_get_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 149, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 307, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 722, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 160, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 318, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "kind": "verb_product", + "gold": [ + "github.actions_list" + ], + "alts": [], + "ranking": [ + "github.actions_list" + ], + "hit1": "github.actions_list", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "kind": "verb_product", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "ranking": [ + "github.actions_get_job_logs" + ], + "hit1": "github.actions_get_job_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "kind": "verb_product", + "gold": [ + "github.create_repository" + ], + "alts": [], + "ranking": [ + "github.create_or_update_file", + "github.create_repository", + "github.create_branch", + "github.create_issue" + ], + "hit1": "github.create_or_update_file", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 346, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 532, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 938, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "kind": "verb_product", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "ranking": [ + "git.git_checkout" + ], + "hit1": "git.git_checkout", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 174, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 332, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 747, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "kind": "identifier", + "gold": [ + "git.git_log" + ], + "alts": [], + "ranking": [ + "git.git_log" + ], + "hit1": "git.git_log", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 173, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "kind": "verb_product", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "ranking": [ + "postgres.list_schemas" + ], + "hit1": "postgres.list_schemas", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 148, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 306, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "kind": "verb_product", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "ranking": [ + "memory.open_nodes" + ], + "hit1": "memory.open_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 164, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "kind": "paraphrase", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 87, + "k3_calls": 29, + "k3_tokens": 7917, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7814, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7616, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "kind": "paraphrase", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 79, + "k3_calls": 27, + "k3_tokens": 7494, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 16, + "k5_tokens": 7208, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 8, + "k10_tokens": 7040, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "kind": "verb_product", + "gold": [ + "github.search_issues" + ], + "alts": [], + "ranking": [ + "github.search_issues" + ], + "hit1": "github.search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 198, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 771, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "kind": "paraphrase", + "gold": [ + "github.get_me" + ], + "alts": [], + "ranking": [ + "github.get_me", + "github.get_teams" + ], + "hit1": "github.get_me", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 181, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 784, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "cost": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 10.733333333333333, + "p95_calls": 59.749999999999915, + "success_within_1": 0.7733333333333333, + "success_within_2": 0.7733333333333333, + "success_within_3": 0.7733333333333333, + "mean_tokens": 2656.4733333333334, + "p95_tokens": 14505.499999999985, + "total_tokens": 398471, + "tokens_per_successful_query": 2656.4733333333334, + "total_tokens_successful": 398471 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 6.793333333333333, + "p95_calls": 35.64999999999995, + "success_within_1": 0.7733333333333333, + "success_within_2": 0.7733333333333333, + "success_within_3": 0.7733333333333333, + "mean_tokens": 2707.62, + "p95_tokens": 13944.199999999986, + "total_tokens": 406143, + "tokens_per_successful_query": 2707.62, + "total_tokens_successful": 406143 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 3.84, + "p95_calls": 18.099999999999966, + "success_within_1": 0.7733333333333333, + "success_within_2": 0.78, + "success_within_3": 0.78, + "mean_tokens": 3003.58, + "p95_tokens": 13725.299999999981, + "total_tokens": 450537, + "tokens_per_successful_query": 3003.58, + "total_tokens_successful": 450537 + } + } +} diff --git a/experiments/mcp-tool-search/results/boolean_or.csv b/experiments/mcp-tool-search/results/boolean_or.csv new file mode 100644 index 00000000..076ebfa1 --- /dev/null +++ b/experiments/mcp-tool-search/results/boolean_or.csv @@ -0,0 +1,151 @@ +id,query,kind,hit1,recall@1,recall@3,recall@5,mrr@10,ndcg@10,hit_rank,k3_calls,k3_tokens,k3_success1,k5_calls,k5_tokens,k5_success1,k10_calls,k10_tokens,k10_success1,ranking +q001,Create a GitHub issue,verb_product,github.create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,434,1.0,1,614,1.0,1,1216,1.0,github.create_issue github.issue_write github.create_or_update_file github.create_repository github.create_branch github.issue_read github.add_issue_comment github.update_issue_comment github.create_pull_request github.pull_request_review_write +q002,Create a Jira ticket,verb_product,jira.jira_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,333,1.0,1,528,1.0,1,1058,1.0,jira.jira_create_issue jira.jira_create_sprint zendesk.zendesk_create_ticket jira.create_board jira.create_sprint jira.create_filter jira.create_webhook jira.create_attachment jira.create_version github.create_or_update_file +q003,Send a Slack message,verb_product,slack.slack_post_message,1.0,1.0,1.0,1.0000,1.0000,1,1,247,1.0,1,507,1.0,1,924,1.0,slack.slack_post_message slack.slack_add_reaction discord.discord_send_message github.create_or_update_file github.push_files github.delete_file git.git_commit slack.slack_reply_to_thread slack.slack_get_channel_history slack.slack_get_thread_replies +q004,List open pull requests,resource,github.list_pull_requests,1.0,1.0,1.0,1.0000,0.9239,1,1,334,1.0,1,423,1.0,1,808,1.0,github.list_pull_requests bitbucket.bitbucket_list_prs github.create_pull_request github.search_pull_requests playwright.browser_tab_list gitlab.gitlab_list_merge_requests github.get_team_members github.list_commits github.list_branches github.list_tags +q005,Open a pull request on GitHub,verb_product,github.create_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,364,1.0,1,611,1.0,1,1194,1.0,github.create_pull_request github.request_copilot_review github.pull_request_read github.update_pull_request github.merge_pull_request github.pull_request_review_write github.add_comment_to_pending_review github.list_commits github.list_repository_collaborators github.update_issue_comment +q006,Run a SQL query against postgres,verb_product,postgres.query,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,295,1.0,1,604,1.0,postgres.query sqlite.sqlite_query mysql.mysql_query postgres.explain_query salesforce.salesforce_query postgres.list_tables postgres.describe_table postgres.list_schemas github.search_repositories github.search_code +q007,Read a local file from disk,paraphrase,filesystem.read_file,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,444,1.0,1,881,1.0,filesystem.read_file filesystem.read_text_file filesystem.read_media_file github.get_file_contents github.delete_file gitlab.gitlab_get_file figma.figma_get_file_nodes figma.figma_get_images github.get_file_blame github.get_tag +q008,What files are in this folder,paraphrase,google_drive.gdrive_list_folder,0.0,0.0,0.0,0.0000,0.0000,85,29,7919,0.0,17,7541,0.0,9,7663,0.0,google_drive.gdrive_list_folder github.push_files google_drive.gdrive_search slack.list_files github.get_file_blame github.list_branches github.list_tags github.create_or_update_file github.create_branch github.create_issue +q009,Remember this fact in the knowledge graph,paraphrase,memory.create_entities,1.0,1.0,1.0,1.0000,0.7602,1,1,178,1.0,1,285,1.0,1,506,1.0,memory.create_entities memory.create_relations memory.add_observations memory.search_nodes memory.open_nodes memory.delete_entities memory.delete_relations memory.read_graph postgres.list_tables postgres.list_schemas +q010,Search the web with Brave,verb_product,brave_search.brave_web_search,1.0,1.0,1.0,1.0000,1.0000,1,1,262,1.0,1,410,1.0,1,714,1.0,brave_search.brave_web_search github.update_pull_request_branch memory.search_nodes brave_search.brave_local_search jira.jira_search_issues github.get_me github.get_teams github.search_repositories github.search_code github.search_commits +q011,Fetch the contents of a URL,verb_product,fetch.fetch,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,369,1.0,1,671,1.0,fetch.fetch filesystem.read_file github.get_me github.get_repository_tree git.git_status git.git_add slack.slack_set_channel_topic playwright.browser_navigate playwright.browser_snapshot playwright.browser_take_screenshot +q012,Show git status,identifier,git.git_status,1.0,1.0,1.0,1.0000,1.0000,1,1,176,1.0,1,302,1.0,1,613,1.0,git.git_status git.git_diff_unstaged git.git_diff_staged git.git_diff git.git_log git.git_show github.get_me github.get_teams github.get_team_members github.search_repositories +q013,Commit staged changes,paraphrase,git.git_commit,1.0,1.0,1.0,1.0000,1.0000,1,1,215,1.0,1,381,1.0,1,663,1.0,git.git_commit git.git_reset github.get_commit github.push_files git.git_diff_staged git.git_log git.git_show +q014,Create a Linear issue,verb_product,linear.linear_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,351,1.0,1,601,1.0,1,1019,1.0,linear.linear_create_issue linear.linear_create_comment github.create_issue github.issue_write linear.linear_update_issue linear.linear_get_issue linear.linear_create_project linear.linear_assign_issue jira.jira_create_issue gitlab.gitlab_create_issue +q015,Make a task in my project tracker Linear,paraphrase,linear.linear_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,353,1.0,1,488,1.0,1,823,1.0,linear.linear_create_issue linear.linear_create_project jira.jira_create_issue supabase.supabase_list_tables linear.create_cycle linear.get_cycle linear.list_cycles linear.update_cycle linear.delete_cycle linear.create_label +q016,Post a comment on a GitHub issue,verb_product,github.update_issue_comment,0.0,1.0,1.0,0.5000,0.6309,2,1,338,1.0,1,519,1.0,1,1190,1.0,github.update_issue_comment github.add_issue_comment github.assign_copilot_to_issue github.list_commits github.list_repository_collaborators github.create_issue github.issue_write github.issue_read github.create_pull_request github.add_comment_to_pending_review +q017,Merge the pull request,ambiguous,github.merge_pull_request,1.0,1.0,1.0,1.0000,0.7602,1,1,370,1.0,1,602,1.0,1,1196,1.0,github.merge_pull_request github.update_pull_request_branch github.create_pull_request github.pull_request_read github.update_pull_request github.pull_request_review_write github.add_comment_to_pending_review github.request_copilot_review gitlab.gitlab_create_merge_request gitlab.gitlab_get_merge_request +q018,Trigger a GitHub Actions workflow,verb_product,github.actions_run_trigger,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,442,1.0,1,848,1.0,github.actions_run_trigger github.actions_get github.actions_list github.actions_get_job_logs github.get_me github.get_teams github.get_team_members github.search_repositories github.get_file_contents github.list_commits +q019,Get Dependabot alerts,resource,github.get_dependabot_alert,1.0,1.0,1.0,1.0000,0.8597,1,1,243,1.0,1,348,1.0,1,792,1.0,github.get_dependabot_alert github.list_dependabot_alerts github.get_me github.get_teams github.get_team_members github.get_file_contents github.get_commit github.get_file_blame github.get_tag github.get_latest_release +q020,List Sentry issues,verb_product,sentry.sentry_list_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,302,1.0,1,425,1.0,1,809,1.0,sentry.sentry_list_issues github.list_issues linear.linear_list_issues sentry.sentry_list_events sentry.sentry_list_projects gitlab.gitlab_list_issues github.get_team_members github.list_commits github.list_branches github.list_tags +q021,Create a Stripe customer,verb_product,stripe.stripe_create_customer,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,322,1.0,1,749,1.0,stripe.stripe_create_customer stripe.stripe_create_payment_intent stripe.stripe_create_invoice stripe.stripe_get_customer stripe.stripe_list_invoices stripe.stripe_create_product stripe.stripe_create_price stripe.stripe_list_subscriptions stripe.stripe_create_refund github.create_or_update_file +q022,Refund a Stripe payment,paraphrase,stripe.stripe_create_refund,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,352,1.0,1,662,1.0,stripe.stripe_create_refund stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_list_customers stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_create_product stripe.stripe_create_price +q023,Send an email with Gmail,verb_product,gmail.gmail_send_email,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,385,1.0,1,859,1.0,gmail.gmail_send_email puppeteer.puppeteer_click puppeteer.puppeteer_fill gmail.gmail_get_email gmail.gmail_modify_email gmail.gmail_trash_email twilio.twilio_send_sms github.create_issue github.update_pull_request_branch github.assign_copilot_to_issue +q024,Search my Gmail inbox,verb_product,gmail.gmail_search_emails,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,389,1.0,1,669,1.0,gmail.gmail_search_emails github.search_repositories github.search_code github.search_commits github.search_issues github.search_pull_requests github.search_users github.search_orgs filesystem.search_files memory.search_nodes +q025,Upload a file to Google Drive,paraphrase,slack.slack_upload_file,0.0,1.0,1.0,0.3333,0.5000,3,1,269,1.0,1,446,1.0,1,801,1.0,slack.slack_upload_file google_drive.gdrive_get_file google_drive.gdrive_create_file google_drive.gdrive_update_file google_drive.gdrive_share_file filesystem.edit_file git.git_add google_drive.gdrive_search google_drive.gdrive_list_folder aws_s3.s3_put_object +q026,Create a calendar event,resource,google_calendar.gcal_create_event,1.0,1.0,1.0,1.0000,1.0000,1,1,371,1.0,1,588,1.0,1,1186,1.0,google_calendar.gcal_create_event github.pull_request_review_write google_calendar.gcal_update_event google_calendar.gcal_delete_event github.create_or_update_file github.create_repository github.create_branch github.create_issue github.issue_write github.create_pull_request +q027,List Kubernetes pods,verb_product,kubernetes.k8s_list_pods,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,303,1.0,1,675,1.0,kubernetes.k8s_list_pods kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_get_events github.get_team_members github.list_commits github.list_branches github.list_tags github.list_releases github.list_starred_repositories +q028,Get logs from a k8s pod,verb_product,kubernetes.k8s_pod_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,314,1.0,1,481,1.0,1,881,1.0,kubernetes.k8s_pod_logs github.get_file_contents github.get_tag github.actions_get_job_logs slack.slack_get_channel_history docker.docker_logs kubernetes.k8s_get_pod datadog.datadog_list_logs gitlab.gitlab_get_file elasticsearch.es_get_doc +q029,Run a docker container,verb_product,docker.docker_run,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,345,1.0,1,682,1.0,docker.docker_run docker.docker_exec docker.docker_logs docker.docker_stop docker.docker_rm github.actions_get github.actions_run_trigger postgres.query docker.docker_ps docker.docker_images +q030,Show docker container logs,verb_product,docker.docker_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,343,1.0,1,705,1.0,docker.docker_logs git.git_log docker.docker_run docker.docker_stop docker.docker_rm docker.docker_exec kubernetes.k8s_pod_logs github.actions_get_job_logs git.git_status git.git_diff_unstaged +q031,Navigate the browser to a URL,ambiguous,playwright.browser_navigate,1.0,1.0,1.0,1.0000,1.0000,1,1,170,1.0,1,304,1.0,1,609,1.0,playwright.browser_navigate puppeteer.puppeteer_navigate filesystem.list_allowed_directories memory.add_observations fetch.fetch git.git_commit git.git_add playwright.browser_snapshot playwright.browser_click playwright.browser_take_screenshot +q032,Take a screenshot of the page,ambiguous,playwright.browser_take_screenshot,1.0,1.0,1.0,1.0000,0.7602,1,1,189,1.0,1,352,1.0,1,642,1.0,playwright.browser_take_screenshot playwright.browser_snapshot github.get_me github.get_repository_tree filesystem.read_file git.git_status slack.slack_set_channel_topic playwright.browser_click playwright.browser_evaluate playwright.browser_press_key +q033,Create a GitLab merge request,verb_product,gitlab.gitlab_create_merge_request,1.0,1.0,1.0,1.0000,1.0000,1,1,297,1.0,1,552,1.0,1,1075,1.0,gitlab.gitlab_create_merge_request gitlab.gitlab_get_merge_request gitlab.gitlab_merge_merge_request github.create_pull_request github.merge_pull_request github.pull_request_review_write gitlab.gitlab_create_issue gitlab.gitlab_list_merge_requests gitlab.gitlab_create_pipeline github.create_or_update_file +q034,Search Notion pages,verb_product,notion.notion_search,1.0,1.0,1.0,1.0000,1.0000,1,1,248,1.0,1,387,1.0,1,667,1.0,notion.notion_search github.search_repositories github.search_code github.search_commits github.search_issues github.search_pull_requests github.search_users github.search_orgs filesystem.search_files memory.search_nodes +q035,Create a Notion page,verb_product,notion.notion_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,230,1.0,1,371,1.0,1,856,1.0,notion.notion_create_page notion.notion_get_page notion.notion_update_page notion.notion_create_database notion.notion_get_comments confluence.confluence_create_page notion.create_comment notion.create_users_token notion.create_data_source github.search_repositories +q036,Query a Notion database,verb_product,notion.notion_query_database,1.0,1.0,1.0,1.0000,1.0000,1,1,262,1.0,1,439,1.0,1,749,1.0,notion.notion_query_database notion.notion_search notion.notion_create_page notion.notion_create_database github.search_repositories github.search_code github.search_commits github.search_issues github.search_pull_requests github.search_users +q037,Create a Zendesk ticket,verb_product,zendesk.zendesk_create_ticket,1.0,1.0,1.0,1.0000,1.0000,1,1,282,1.0,1,517,1.0,1,1115,1.0,zendesk.zendesk_create_ticket jira.jira_create_issue zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket github.create_or_update_file github.create_repository github.create_branch github.create_issue github.issue_write github.create_pull_request +q038,Create a HubSpot contact,verb_product,hubspot.hubspot_create_contact,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,480,1.0,1,1141,1.0,hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_create_deal github.create_or_update_file github.create_repository github.create_branch github.create_issue github.issue_write github.create_pull_request github.pull_request_review_write +q039,Create a Salesforce lead,verb_product,salesforce.salesforce_create_lead,1.0,1.0,1.0,1.0000,1.0000,1,1,252,1.0,1,477,1.0,1,1138,1.0,salesforce.salesforce_create_lead salesforce.salesforce_create_record salesforce.salesforce_convert_lead github.create_or_update_file github.create_repository github.create_branch github.create_issue github.issue_write github.create_pull_request github.pull_request_review_write +q040,Run a SOQL query,identifier,salesforce.salesforce_query,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,330,1.0,1,676,1.0,salesforce.salesforce_query postgres.query sqlite.sqlite_query mysql.mysql_query github.search_repositories github.search_code github.search_commits github.search_issues github.search_pull_requests github.actions_get +q041,List Shopify orders,verb_product,shopify.shopify_list_orders,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,378,1.0,1,725,1.0,shopify.shopify_list_orders shopify.shopify_list_products shopify.shopify_list_customers github.get_team_members github.list_commits github.list_branches github.list_tags github.list_releases github.list_starred_repositories github.list_repository_collaborators +q042,Create a PagerDuty incident,verb_product,pagerduty.pagerduty_create_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,403,1.0,1,922,1.0,pagerduty.pagerduty_create_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls github.create_or_update_file github.create_repository github.create_branch github.create_issue +q043,Who is on call in PagerDuty,paraphrase,pagerduty.pagerduty_list_oncalls,1.0,1.0,1.0,1.0000,1.0000,1,1,220,1.0,1,388,1.0,1,796,1.0,pagerduty.pagerduty_list_oncalls memory.search_nodes pagerduty.pagerduty_list_incidents pagerduty.pagerduty_get_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident github.list_commits github.get_file_blame github.list_branches +q044,Query Datadog metrics,verb_product,datadog.datadog_query_metrics,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,490,1.0,1,800,1.0,datadog.datadog_query_metrics datadog.datadog_list_monitors datadog.datadog_create_monitor datadog.datadog_list_logs github.search_repositories github.search_code github.search_commits github.search_issues github.search_pull_requests github.search_users +q045,Search Datadog logs,verb_product,datadog.datadog_list_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,279,1.0,1,418,1.0,1,735,1.0,datadog.datadog_list_logs github.search_repositories github.search_code github.search_commits github.search_issues github.search_pull_requests github.actions_get_job_logs github.search_users github.search_orgs filesystem.search_files +q046,List S3 buckets,verb_product,aws_s3.s3_list_buckets,1.0,1.0,1.0,1.0000,1.0000,1,1,231,1.0,1,373,1.0,1,768,1.0,aws_s3.s3_list_buckets github.get_team_members github.list_commits github.list_branches github.list_tags github.list_releases github.list_starred_repositories github.list_repository_collaborators github.list_issues github.list_pull_requests +q047,Upload an object to S3,paraphrase,aws_s3.s3_put_object,1.0,1.0,1.0,1.0000,1.0000,1,1,303,1.0,1,479,1.0,1,954,1.0,aws_s3.s3_put_object github.assign_copilot_to_issue slack.slack_add_reaction slack.slack_upload_file gmail.gmail_send_email aws_s3.s3_get_object aws_s3.s3_delete_object twilio.twilio_send_sms github.push_files github.create_issue +q048,Purge Cloudflare cache,verb_product,cloudflare.cf_purge_cache,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,340,1.0,1,746,1.0,cloudflare.cf_purge_cache cloudflare.cf_list_zones cloudflare.cf_list_dns cloudflare.cf_create_dns +q049,Create a Cloudflare DNS record,verb_product,cloudflare.cf_create_dns,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,474,1.0,1,1135,1.0,cloudflare.cf_create_dns salesforce.salesforce_create_record cloudflare.cf_list_dns github.create_or_update_file github.create_repository github.create_branch github.create_issue github.issue_write github.create_pull_request github.pull_request_review_write +q050,List Vercel deployments,verb_product,vercel.vercel_list_deployments,1.0,1.0,1.0,1.0000,1.0000,1,1,173,1.0,1,343,1.0,1,690,1.0,vercel.vercel_list_deployments kubernetes.k8s_list_deployments vercel.vercel_list_projects github.get_team_members github.list_commits github.list_branches github.list_tags github.list_releases github.list_starred_repositories github.list_repository_collaborators +q051,Insert a row into Supabase,verb_product,supabase.supabase_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,347,1.0,1,665,1.0,supabase.supabase_insert mongodb.mongo_insert playwright.browser_type supabase.supabase_list_tables supabase.supabase_select supabase.supabase_update supabase.supabase_invoke_fn +q052,Export images from Figma,paraphrase,figma.figma_get_images,1.0,1.0,1.0,1.0000,1.0000,1,1,292,1.0,1,477,1.0,1,844,1.0,figma.figma_get_images figma.figma_get_file_nodes github.get_file_contents github.get_tag github.create_branch github.delete_file filesystem.read_file filesystem.read_text_file filesystem.read_media_file memory.delete_entities +q053,Send an SMS with Twilio,verb_product,twilio.twilio_send_sms,1.0,1.0,1.0,1.0000,1.0000,1,1,232,1.0,1,377,1.0,1,836,1.0,twilio.twilio_send_sms puppeteer.puppeteer_click puppeteer.puppeteer_fill gmail.gmail_send_email twilio.twilio_list_messages github.create_issue github.update_pull_request_branch github.assign_copilot_to_issue filesystem.read_media_file filesystem.write_file +q054,Create an Okta user,verb_product,okta.okta_create_user,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,385,1.0,1,748,1.0,okta.okta_create_user okta.okta_get_user okta.okta_deactivate_user github.create_issue filesystem.write_file asana.asana_create_task asana.asana_create_project linear.create_user github.get_me github.get_teams +q055,Instant query Prometheus,verb_product,prometheus.prom_query,1.0,1.0,1.0,1.0000,1.0000,1,1,262,1.0,1,383,1.0,1,681,1.0,prometheus.prom_query prometheus.prom_query_range github.search_repositories github.search_code github.search_commits github.search_issues github.search_pull_requests github.search_users github.search_orgs memory.search_nodes +q056,Search Grafana dashboards,verb_product,grafana.grafana_search_dashboards,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,377,1.0,1,657,1.0,grafana.grafana_search_dashboards github.search_repositories github.search_code github.search_commits github.search_issues github.search_pull_requests github.search_users github.search_orgs filesystem.search_files memory.search_nodes +q057,Trigger a CircleCI pipeline,verb_product,circleci.circleci_trigger_pipeline,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,390,1.0,1,755,1.0,circleci.circleci_trigger_pipeline circleci.circleci_get_pipeline gitlab.gitlab_create_pipeline circleci.circleci_list_pipelines github.actions_run_trigger hubspot.hubspot_create_deal gitlab.create_pipeline gitlab.get_pipeline gitlab.update_pipeline gitlab.delete_pipeline +q058,Run terraform plan,identifier,terraform.tf_plan,1.0,1.0,1.0,1.0000,1.0000,1,1,210,1.0,1,382,1.0,1,706,1.0,terraform.tf_plan terraform.tf_apply github.actions_get github.actions_run_trigger postgres.query postgres.explain_query docker.docker_run docker.docker_exec salesforce.salesforce_query sqlite.sqlite_query +q059,Create an Asana task,verb_product,asana.asana_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,470,1.0,1,908,1.0,asana.asana_create_task asana.asana_get_task asana.asana_update_task asana.asana_create_project github.create_issue filesystem.write_file okta.okta_create_user clickup.clickup_create_task github.create_or_update_file github.create_repository +q060,Create a Trello card,verb_product,trello.trello_create_card,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,465,1.0,1,1126,1.0,trello.trello_create_card trello.trello_create_board trello.trello_add_comment github.create_or_update_file github.create_repository github.create_branch github.create_issue github.issue_write github.create_pull_request github.pull_request_review_write +q061,Create a ClickUp task,verb_product,clickup.clickup_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,415,1.0,1,1008,1.0,clickup.clickup_create_task clickup.clickup_get_task clickup.clickup_update_task asana.asana_create_task clickup.clickup_list_tasks github.create_or_update_file github.create_repository github.create_branch github.create_issue github.issue_write +q062,Create a Confluence page,verb_product,confluence.confluence_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,256,1.0,1,438,1.0,1,934,1.0,confluence.confluence_create_page notion.notion_create_page confluence.confluence_get_page confluence.confluence_update_page github.search_repositories github.list_commits github.search_code github.create_or_update_file github.create_repository github.create_branch +q063,Send a Discord message,verb_product,discord.discord_send_message,1.0,1.0,1.0,1.0000,1.0000,1,1,320,1.0,1,549,1.0,1,981,1.0,discord.discord_send_message discord.discord_add_reaction github.create_or_update_file github.push_files github.delete_file git.git_commit slack.slack_post_message slack.slack_add_reaction datadog.datadog_create_monitor gmail.gmail_send_email +q064,Search Elasticsearch,verb_product,elasticsearch.es_search,1.0,1.0,1.0,1.0000,1.0000,1,1,243,1.0,1,382,1.0,1,662,1.0,elasticsearch.es_search github.search_repositories github.search_code github.search_commits github.search_issues github.search_pull_requests github.search_users github.search_orgs filesystem.search_files memory.search_nodes +q065,Get a Redis key,verb_product,redis.redis_get,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,315,1.0,1,627,1.0,redis.redis_get jira.jira_get_issue redis.redis_set aws_s3.s3_get_object figma.figma_get_file figma.figma_get_file_nodes figma.figma_get_comments figma.figma_get_images github.get_me github.get_teams +q066,Find MongoDB documents,verb_product,mongodb.mongo_find,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,381,1.0,1,738,1.0,mongodb.mongo_find mongodb.mongo_insert mongodb.mongo_update mongodb.mongo_delete mongodb.mongo_list_collections +q067,List SQLite tables,verb_product,sqlite.sqlite_list_tables,1.0,1.0,1.0,1.0000,1.0000,1,1,156,1.0,1,252,1.0,1,604,1.0,sqlite.sqlite_list_tables postgres.list_tables sqlite.sqlite_query sqlite.sqlite_describe_table mysql.mysql_list_tables supabase.supabase_list_tables github.get_team_members github.list_commits github.list_branches github.list_tags +q068,Describe a postgres table,verb_product,postgres.describe_table,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,272,1.0,1,597,1.0,postgres.describe_table sqlite.sqlite_describe_table postgres.query postgres.list_tables postgres.list_schemas postgres.explain_query supabase.supabase_select supabase.supabase_insert supabase.supabase_update +q069,github_create_issue,identifier,github.create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,432,1.0,1,612,1.0,1,1214,1.0,github.create_issue github.issue_write github.create_or_update_file github.create_repository github.create_branch github.issue_read github.add_issue_comment github.update_issue_comment github.create_pull_request github.pull_request_review_write +q070,createIssue,identifier,github.create_issue,1.0,1.0,1.0,1.0000,0.9220,1,1,418,1.0,1,600,1.0,1,1107,1.0,github.create_issue github.issue_write linear.linear_create_issue linear.linear_create_comment jira.jira_create_issue gitlab.gitlab_create_issue bitbucket.bitbucket_create_issue github.create_or_update_file github.create_repository github.create_branch +q071,list-directory,identifier,filesystem.list_directory,1.0,1.0,1.0,1.0000,1.0000,1,1,198,1.0,1,408,1.0,1,755,1.0,filesystem.list_directory filesystem.list_directory_with_sizes github.get_team_members github.get_file_contents github.list_commits github.list_branches github.list_tags github.list_releases github.list_starred_repositories github.list_repository_collaborators +q072,sequential thinking next thought,paraphrase,sequential_thinking.sequentialthinking,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,382,1.0,1,797,1.0,sequential_thinking.sequentialthinking +q073,What time is it in Tokyo,paraphrase,time.get_current_time,1.0,1.0,1.0,1.0000,0.7602,1,1,241,1.0,1,456,1.0,1,993,1.0,time.get_current_time github.get_file_blame github.list_branches github.list_tags github.create_or_update_file github.create_branch github.push_files github.create_issue github.list_issues github.list_pull_requests +q074,Convert time between timezones,verb_product,time.convert_time,1.0,1.0,1.0,1.0000,1.0000,1,1,214,1.0,1,354,1.0,1,704,1.0,time.convert_time memory.create_relations git.git_diff time.get_current_time jira.jira_add_worklog playwright.browser_wait_for salesforce.salesforce_convert_lead google_calendar.gcal_list_events grafana.grafana_create_annotation +q075,Star a GitHub repo,verb_product,github.star_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,298,1.0,1,479,1.0,1,849,1.0,github.star_repository github.get_file_contents github.list_commits github.get_commit github.get_file_blame github.list_branches github.list_tags github.get_tag github.list_releases github.get_latest_release +q076,Fork a repository,resource,github.fork_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,311,1.0,1,480,1.0,1,932,1.0,github.fork_repository github.get_file_contents github.list_commits github.get_file_blame github.list_branches github.list_tags github.get_tag github.list_releases github.create_or_update_file github.create_repository +q077,Get the latest GitHub release,verb_product,github.get_latest_release,1.0,1.0,1.0,1.0000,1.0000,1,1,185,1.0,1,383,1.0,1,823,1.0,github.get_latest_release github.get_me github.get_teams github.get_release_by_tag github.get_repository_tree github.get_team_members github.get_file_contents github.get_commit github.get_file_blame github.get_tag +q078,Create a gist,resource,github.create_gist,1.0,1.0,1.0,1.0000,1.0000,1,1,322,1.0,1,554,1.0,1,1115,1.0,github.create_gist github.create_or_update_file github.create_repository github.create_branch github.create_issue github.issue_write github.create_pull_request github.pull_request_review_write github.get_gist github.update_gist +q079,Search GitHub code,verb_product,github.search_code,1.0,1.0,1.0,1.0000,1.0000,1,1,232,1.0,1,371,1.0,1,722,1.0,github.search_code github.search_repositories github.search_commits github.search_issues github.search_pull_requests github.get_code_scanning_alert github.list_code_scanning_alerts github.search_users github.search_orgs github.get_me +q080,List GitHub notifications,verb_product,github.list_notifications,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,397,1.0,1,792,1.0,github.list_notifications github.get_team_members github.list_commits github.list_branches github.list_tags github.list_releases github.list_starred_repositories github.list_repository_collaborators github.list_issues github.list_pull_requests +q081,Assign Copilot to an issue,verb_product,github.assign_copilot_to_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,332,1.0,1,527,1.0,1,864,1.0,github.assign_copilot_to_issue linear.linear_assign_issue github.create_issue github.add_issue_comment slack.slack_add_reaction linear.linear_create_comment jira.jira_update_issue jira.jira_add_comment jira.jira_transition_issue jira.jira_assign_issue +q082,Get secret scanning alerts,resource,github.get_secret_scanning_alert,1.0,1.0,1.0,1.0000,0.8597,1,1,294,1.0,1,446,1.0,1,839,1.0,github.get_secret_scanning_alert github.list_secret_scanning_alerts github.get_code_scanning_alert github.list_code_scanning_alerts github.get_me github.get_teams github.get_team_members github.get_file_contents github.get_commit github.get_file_blame +q083,Create or update a file on GitHub,verb_product,github.create_or_update_file,1.0,1.0,1.0,1.0000,0.7602,1,1,389,1.0,1,612,1.0,1,1243,1.0,github.create_or_update_file github.issue_write github.projects_write github.label_write github.get_file_contents github.update_issue_comment github.create_pull_request github.update_pull_request github.pull_request_review_write github.search_repositories +q084,List collaborators on a repo,resource,github.list_repository_collaborators,1.0,1.0,1.0,1.0000,1.0000,1,1,274,1.0,1,416,1.0,1,965,1.0,github.list_repository_collaborators github.list_commits github.list_branches github.list_tags github.list_releases github.list_issues github.update_issue_comment github.create_pull_request github.list_pull_requests github.assign_copilot_to_issue +q085,Reply in a Slack thread,verb_product,slack.slack_reply_to_thread,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,411,1.0,1,781,1.0,slack.slack_reply_to_thread slack.slack_get_thread_replies slack.slack_list_channels slack.create_reminder slack.get_reminder slack.list_reminders slack.update_reminder slack.delete_reminder slack.create_canvase slack.get_canvase +q086,Add an emoji reaction on Slack,verb_product,slack.slack_add_reaction,1.0,1.0,1.0,1.0000,1.0000,1,1,304,1.0,1,461,1.0,1,950,1.0,slack.slack_add_reaction discord.discord_add_reaction github.assign_copilot_to_issue playwright.browser_click gmail.gmail_modify_email trello.trello_add_comment github.list_commits github.list_repository_collaborators github.create_issue github.add_issue_comment +q087,Search Slack messages,verb_product,slack.slack_search_messages,1.0,1.0,1.0,1.0000,1.0000,1,1,232,1.0,1,391,1.0,1,685,1.0,slack.slack_search_messages slack.slack_get_channel_history gmail.gmail_search_emails github.search_repositories github.search_code github.search_commits github.search_issues github.search_pull_requests github.search_users github.search_orgs +q088,Create a Slack channel,verb_product,slack.slack_create_channel,1.0,1.0,1.0,1.0000,1.0000,1,1,251,1.0,1,414,1.0,1,751,1.0,slack.slack_create_channel slack.slack_post_message slack.slack_reply_to_thread slack.slack_add_reaction slack.slack_get_channel_history slack.slack_get_thread_replies slack.slack_join_channel slack.slack_leave_channel slack.slack_set_channel_topic slack.slack_upload_file +q089,Transition a Jira issue,verb_product,jira.jira_transition_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,270,1.0,1,390,1.0,1,914,1.0,jira.jira_transition_issue jira.jira_create_issue jira.jira_update_issue jira.jira_get_issue jira.jira_add_comment jira.jira_assign_issue jira.jira_add_worklog github.create_issue github.issue_write github.issue_read +q090,Search Jira with JQL,identifier,jira.jira_search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,252,1.0,1,391,1.0,1,700,1.0,jira.jira_search_issues github.search_repositories github.search_code github.search_commits github.search_issues github.search_pull_requests github.update_pull_request_branch github.search_users github.search_orgs filesystem.list_directory_with_sizes +q091,Add a worklog to Jira,verb_product,jira.jira_add_worklog,1.0,1.0,1.0,1.0000,1.0000,1,1,285,1.0,1,495,1.0,1,874,1.0,jira.jira_add_worklog jira.jira_add_comment github.add_issue_comment github.add_comment_to_pending_review memory.add_observations git.git_add slack.slack_add_reaction linear.linear_create_comment jira.jira_transition_issue discord.discord_add_reaction +q092,List Linear teams,verb_product,linear.linear_list_teams,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,330,1.0,1,658,1.0,linear.linear_list_teams linear.linear_list_issues linear.linear_list_projects linear.list_cycles linear.list_labels linear.list_users linear.list_comments linear.list_attachments github.get_teams github.get_team_members +q093,Comment on a Linear issue,verb_product,github.update_issue_comment,0.0,1.0,1.0,0.5000,0.6309,2,1,298,1.0,1,534,1.0,1,936,1.0,github.update_issue_comment linear.linear_create_comment github.add_issue_comment github.assign_copilot_to_issue linear.linear_create_issue linear.linear_update_issue linear.linear_get_issue linear.linear_assign_issue jira.jira_add_comment jira.jira_add_worklog +q094,Move a local file,paraphrase,filesystem.read_file,0.0,0.0,1.0,0.2500,0.4307,4,2,506,0.0,1,389,1.0,1,907,1.0,filesystem.read_file filesystem.read_text_file filesystem.read_media_file filesystem.move_file github.get_file_contents github.get_file_blame github.create_or_update_file github.delete_file filesystem.write_file filesystem.edit_file +q095,Recursively search files by pattern,paraphrase,filesystem.search_files,1.0,1.0,1.0,1.0000,1.0000,1,1,236,1.0,1,359,1.0,1,758,1.0,filesystem.search_files github.search_repositories linear.linear_search_issues google_drive.gdrive_search github.search_code github.search_commits github.get_commit github.get_release_by_tag github.push_files github.list_starred_repositories +q096,Edit a file in place with old and new string,paraphrase,filesystem.edit_file,1.0,1.0,1.0,1.0000,1.0000,1,1,360,1.0,1,489,1.0,1,875,1.0,filesystem.edit_file github.get_file_blame github.create_or_update_file filesystem.write_file filesystem.list_directory_with_sizes filesystem.move_file google_drive.gdrive_create_file slack.create_file slack.get_file slack.update_file +q097,Read the whole knowledge graph,paraphrase,memory.read_graph,1.0,1.0,1.0,1.0000,1.0000,1,1,154,1.0,1,262,1.0,1,561,1.0,memory.read_graph memory.create_entities memory.create_relations memory.add_observations memory.delete_entities memory.delete_relations memory.search_nodes memory.open_nodes filesystem.read_file filesystem.read_text_file +q098,Search memory nodes,verb_product,memory.search_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,338,1.0,1,647,1.0,memory.search_nodes memory.open_nodes github.search_repositories github.search_code github.search_commits github.search_issues github.search_pull_requests github.search_users github.search_orgs filesystem.search_files +q099,Show unstaged git diff,verb_product,git.git_diff_unstaged,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,304,1.0,1,615,1.0,git.git_diff_unstaged git.git_diff_staged git.git_diff git.git_status git.git_log git.git_show github.get_me github.get_teams github.get_team_members github.search_repositories +q100,Stage files with git add,identifier,git.git_add,1.0,1.0,1.0,1.0000,1.0000,1,1,306,1.0,1,538,1.0,1,836,1.0,git.git_add github.push_files github.add_issue_comment github.add_comment_to_pending_review github.create_gist github.update_gist filesystem.list_directory_with_sizes github.get_me github.get_teams github.get_team_members +q101,Create a Shopify product,verb_product,shopify.shopify_create_product,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,388,1.0,1,914,1.0,shopify.shopify_create_product stripe.stripe_create_product stripe.stripe_create_price shopify.shopify_get_product shopify.shopify_update_product shopify.shopify_create_order github.create_or_update_file github.create_repository github.create_branch github.create_issue +q102,Cancel a Stripe subscription,verb_product,stripe.stripe_cancel_subscription,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,340,1.0,1,643,1.0,stripe.stripe_cancel_subscription stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_list_customers stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_create_product stripe.stripe_create_price +q103,Create a payment intent,resource,stripe.stripe_create_payment_intent,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,459,1.0,1,1120,1.0,stripe.stripe_create_payment_intent stripe.stripe_create_refund stripe.stripe_get_payment_intent github.create_or_update_file github.create_repository github.create_branch github.create_issue github.issue_write github.create_pull_request github.pull_request_review_write +q104,Acknowledge a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,261,1.0,1,405,1.0,1,717,1.0,pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls +q105,Create a Datadog monitor,verb_product,datadog.datadog_create_monitor,1.0,1.0,1.0,1.0000,1.0000,1,1,341,1.0,1,521,1.0,1,1160,1.0,datadog.datadog_create_monitor datadog.datadog_get_monitor github.create_or_update_file github.create_repository github.create_branch github.create_issue github.issue_write github.create_pull_request github.pull_request_review_write github.create_gist +q106,Resolve a Sentry issue,paraphrase,sentry.sentry_get_issue,0.0,1.0,1.0,0.3333,0.5000,3,1,251,1.0,1,519,1.0,1,1064,1.0,sentry.sentry_get_issue sentry.sentry_list_events sentry.sentry_update_issue github.create_issue github.issue_write github.issue_read github.add_issue_comment github.update_issue_comment github.assign_copilot_to_issue linear.linear_create_issue +q107,Scale a Kubernetes deployment,verb_product,kubernetes.k8s_scale_deployment,1.0,1.0,1.0,1.0000,1.0000,1,1,253,1.0,1,418,1.0,1,726,1.0,kubernetes.k8s_scale_deployment kubernetes.k8s_list_pods kubernetes.k8s_get_pod kubernetes.k8s_pod_logs kubernetes.k8s_apply kubernetes.k8s_delete kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_get_events vercel.vercel_get_deployment +q108,Apply a Kubernetes manifest,verb_product,kubernetes.k8s_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,222,1.0,1,408,1.0,1,727,1.0,kubernetes.k8s_apply kubernetes.k8s_list_pods kubernetes.k8s_get_pod kubernetes.k8s_pod_logs kubernetes.k8s_delete kubernetes.k8s_list_deployments kubernetes.k8s_scale_deployment kubernetes.k8s_list_services kubernetes.k8s_get_events terraform.tf_apply +q109,Stop a docker container,verb_product,docker.docker_stop,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,345,1.0,1,641,1.0,docker.docker_stop docker.docker_logs docker.docker_run docker.docker_rm docker.docker_exec docker.docker_ps docker.docker_images docker.docker_build kubernetes.k8s_pod_logs +q110,Build a docker image,verb_product,docker.docker_build,1.0,1.0,1.0,1.0000,1.0000,1,1,236,1.0,1,344,1.0,1,601,1.0,docker.docker_build docker.docker_run filesystem.read_media_file docker.docker_ps docker.docker_logs docker.docker_stop docker.docker_rm docker.docker_images docker.docker_exec +q111,Click an element in Playwright,verb_product,playwright.browser_click,1.0,1.0,1.0,1.0000,1.0000,1,1,236,1.0,1,422,1.0,1,772,1.0,playwright.browser_click playwright.browser_type playwright.browser_take_screenshot puppeteer.puppeteer_click github.create_issue playwright.browser_snapshot aws_s3.s3_list_objects github.get_file_blame github.list_branches github.list_tags +q112,Type text into a browser input,paraphrase,playwright.browser_type,1.0,1.0,1.0,1.0000,0.7602,1,1,232,1.0,1,410,1.0,1,714,1.0,playwright.browser_type playwright.browser_wait_for github.projects_list filesystem.read_text_file filesystem.edit_file slack.slack_post_message slack.slack_reply_to_thread linear.linear_search_issues playwright.browser_navigate playwright.browser_snapshot +q113,List GitLab projects,verb_product,gitlab.gitlab_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,209,1.0,1,349,1.0,1,721,1.0,gitlab.gitlab_list_projects github.projects_list gitlab.gitlab_list_issues gitlab.gitlab_list_merge_requests github.get_team_members github.list_commits github.list_branches github.list_tags github.list_releases github.list_starred_repositories +q114,Create a Bitbucket pull request,verb_product,bitbucket.bitbucket_create_pr,1.0,1.0,1.0,1.0000,1.0000,1,1,439,1.0,1,671,1.0,1,1243,1.0,bitbucket.bitbucket_create_pr github.create_pull_request github.pull_request_review_write github.pull_request_read github.update_pull_request github.merge_pull_request github.update_pull_request_branch github.add_comment_to_pending_review github.request_copilot_review gitlab.gitlab_create_merge_request +q115,Create a Gmail draft,verb_product,gmail.gmail_create_draft,1.0,1.0,1.0,1.0000,1.0000,1,1,396,1.0,1,576,1.0,1,1149,1.0,gmail.gmail_create_draft github.create_pull_request github.create_or_update_file github.create_repository github.create_branch github.create_issue github.issue_write github.pull_request_review_write github.create_gist github.projects_write +q116,Share a Google Drive file,verb_product,google_drive.gdrive_share_file,1.0,1.0,1.0,1.0000,1.0000,1,1,271,1.0,1,417,1.0,1,936,1.0,google_drive.gdrive_share_file google_drive.gdrive_get_file google_drive.gdrive_create_file google_drive.gdrive_update_file google_drive.gdrive_search google_drive.gdrive_list_folder github.get_file_contents github.get_file_blame github.create_or_update_file github.delete_file +q117,List Google Calendar events,verb_product,google_calendar.gcal_list_events,1.0,1.0,1.0,1.0000,1.0000,1,1,199,1.0,1,327,1.0,1,777,1.0,google_calendar.gcal_list_events google_calendar.gcal_list_calendars kubernetes.k8s_get_events sentry.sentry_list_events google_drive.gdrive_list_folder google_calendar.gcal_create_event google_calendar.gcal_update_event google_calendar.gcal_delete_event github.get_team_members github.list_commits +q118,Delete a calendar event,resource,google_calendar.gcal_delete_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,481,1.0,1,895,1.0,google_calendar.gcal_delete_event google_calendar.gcal_create_event google_calendar.gcal_update_event github.delete_repository github.delete_file github.pull_request_review_write github.label_write memory.delete_entities memory.delete_observations memory.delete_relations +q119,List Vercel projects,verb_product,vercel.vercel_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,171,1.0,1,256,1.0,1,607,1.0,vercel.vercel_list_projects github.projects_list linear.linear_list_projects jira.jira_list_projects sentry.sentry_list_projects gitlab.gitlab_list_projects vercel.vercel_list_deployments github.get_team_members github.list_commits github.list_branches +q120,Select rows from Supabase,verb_product,supabase.supabase_select,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,429,1.0,1,844,1.0,supabase.supabase_select supabase.supabase_insert supabase.supabase_update github.get_file_contents github.get_tag github.create_branch github.delete_file filesystem.read_file filesystem.read_text_file filesystem.read_media_file +q121,Post a Figma comment,verb_product,figma.figma_post_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,304,1.0,1,528,1.0,1,883,1.0,figma.figma_post_comment github.add_issue_comment github.update_issue_comment github.add_comment_to_pending_review slack.slack_post_message linear.linear_create_comment jira.jira_add_comment jira.jira_add_worklog figma.figma_get_file figma.figma_get_file_nodes +q122,List Okta users,verb_product,okta.okta_list_users,1.0,1.0,1.0,1.0000,1.0000,1,1,203,1.0,1,303,1.0,1,698,1.0,okta.okta_list_users slack.slack_get_users slack.slack_list_users notion.notion_list_users pagerduty.pagerduty_list_oncalls linear.list_users notion.list_users_tokens github.get_team_members github.list_commits github.list_branches +q123,Range query Prometheus,verb_product,prometheus.prom_query_range,1.0,1.0,1.0,1.0000,1.0000,1,1,262,1.0,1,383,1.0,1,681,1.0,prometheus.prom_query_range prometheus.prom_query github.search_repositories github.search_code github.search_commits github.search_issues github.search_pull_requests github.search_users github.search_orgs memory.search_nodes +q124,Create a Grafana annotation,verb_product,grafana.grafana_create_annotation,1.0,1.0,1.0,1.0000,1.0000,1,1,331,1.0,1,563,1.0,1,1154,1.0,grafana.grafana_create_annotation github.create_or_update_file github.create_repository github.create_branch github.create_issue github.issue_write github.create_pull_request github.pull_request_review_write github.create_gist github.projects_write +q125,Apply terraform,identifier,terraform.tf_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,292,1.0,1,698,1.0,terraform.tf_apply kubernetes.k8s_apply terraform.tf_plan terraform.tf_state_list +q126,List Asana tasks,verb_product,asana.asana_list_tasks,1.0,1.0,1.0,1.0000,1.0000,1,1,207,1.0,1,382,1.0,1,759,1.0,asana.asana_list_tasks clickup.clickup_list_tasks github.get_team_members github.list_commits github.list_branches github.list_tags github.list_releases github.list_starred_repositories github.list_repository_collaborators github.list_issues +q127,Comment on a Trello card,verb_product,trello.trello_add_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,271,1.0,1,438,1.0,1,1028,1.0,trello.trello_add_comment github.update_issue_comment figma.figma_post_comment trello.trello_create_card trello.trello_list_cards github.list_commits github.list_repository_collaborators github.add_issue_comment github.create_pull_request github.add_comment_to_pending_review +q128,Update a ClickUp task,verb_product,clickup.clickup_update_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,399,1.0,1,1022,1.0,clickup.clickup_update_task clickup.clickup_create_task clickup.clickup_get_task asana.asana_update_task clickup.clickup_list_tasks github.create_or_update_file github.issue_write github.update_issue_comment github.update_pull_request github.update_pull_request_branch +q129,Search Confluence,verb_product,confluence.confluence_search,1.0,1.0,1.0,1.0000,1.0000,1,1,232,1.0,1,371,1.0,1,651,1.0,confluence.confluence_search github.search_repositories github.search_code github.search_commits github.search_issues github.search_pull_requests github.search_users github.search_orgs filesystem.search_files memory.search_nodes +q130,Create a HubSpot deal,verb_product,hubspot.hubspot_create_deal,1.0,1.0,1.0,1.0000,1.0000,1,1,342,1.0,1,522,1.0,1,1161,1.0,hubspot.hubspot_create_deal hubspot.hubspot_create_contact github.create_or_update_file github.create_repository github.create_branch github.create_issue github.issue_write github.create_pull_request github.pull_request_review_write github.create_gist +q131,List Zendesk tickets,verb_product,zendesk.zendesk_list_tickets,1.0,1.0,1.0,1.0000,1.0000,1,1,198,1.0,1,373,1.0,1,750,1.0,zendesk.zendesk_list_tickets zendesk.zendesk_search github.get_team_members github.list_commits github.list_branches github.list_tags github.list_releases github.list_starred_repositories github.list_repository_collaborators github.list_issues +q132,List Discord channels,verb_product,discord.discord_list_channels,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,354,1.0,1,701,1.0,discord.discord_list_channels slack.slack_list_channels discord.discord_list_members github.get_team_members github.list_commits github.list_branches github.list_tags github.list_releases github.list_starred_repositories github.list_repository_collaborators +q133,Set a Redis key,verb_product,redis.redis_set,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,392,1.0,1,738,1.0,redis.redis_set redis.redis_get slack.slack_set_channel_topic jira.jira_create_issue jira.jira_update_issue jira.jira_get_issue jira.jira_add_comment jira.jira_transition_issue jira.jira_assign_issue jira.jira_add_worklog +q134,Insert MongoDB documents,verb_product,mongodb.mongo_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,381,1.0,1,698,1.0,mongodb.mongo_insert mongodb.mongo_find mongodb.mongo_update mongodb.mongo_delete mongodb.mongo_list_collections supabase.supabase_insert +q135,Run a MySQL query,verb_product,mysql.mysql_query,1.0,1.0,1.0,1.0000,1.0000,1,1,179,1.0,1,294,1.0,1,639,1.0,mysql.mysql_query postgres.query sqlite.sqlite_query postgres.explain_query salesforce.salesforce_query mysql.mysql_list_tables github.search_repositories github.search_code github.search_commits github.search_issues +q136,Index a document in Elasticsearch,verb_product,elasticsearch.es_index_doc,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,411,1.0,1,973,1.0,elasticsearch.es_index_doc elasticsearch.es_get_doc elasticsearch.es_search github.get_file_blame github.list_branches github.list_tags github.create_or_update_file github.create_branch github.push_files github.create_issue +q137,Get an S3 object,verb_product,aws_s3.s3_get_object,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,355,1.0,1,705,1.0,aws_s3.s3_get_object aws_s3.s3_put_object aws_s3.s3_delete_object okta.okta_get_user asana.asana_get_task github.get_me github.get_teams github.get_team_members github.get_file_contents github.get_commit +q138,List Cloudflare zones,verb_product,cloudflare.cf_list_zones,1.0,1.0,1.0,1.0000,1.0000,1,1,179,1.0,1,354,1.0,1,731,1.0,cloudflare.cf_list_zones cloudflare.cf_list_dns github.get_team_members github.list_commits github.list_branches github.list_tags github.list_releases github.list_starred_repositories github.list_repository_collaborators github.list_issues +q139,Create a Stripe invoice,verb_product,stripe.stripe_create_invoice,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,371,1.0,1,897,1.0,stripe.stripe_create_invoice stripe.stripe_create_customer stripe.stripe_create_payment_intent stripe.stripe_create_product stripe.stripe_create_price stripe.stripe_create_refund github.create_or_update_file github.create_repository github.create_branch github.create_issue +q140,List GitHub Actions workflows,verb_product,github.actions_list,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,400,1.0,1,795,1.0,github.actions_list github.get_team_members github.list_commits github.list_branches github.list_tags github.list_releases github.list_starred_repositories github.list_repository_collaborators github.list_issues github.list_pull_requests +q141,Get GitHub Actions job logs,verb_product,github.actions_get_job_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,256,1.0,1,361,1.0,1,805,1.0,github.actions_get_job_logs github.actions_get github.get_me github.get_teams github.get_team_members github.get_file_contents github.get_commit github.get_file_blame github.get_tag github.get_latest_release +q142,Create a GitHub repository,verb_product,github.create_or_update_file,0.0,1.0,1.0,0.5000,0.6309,2,1,346,1.0,1,585,1.0,1,1012,1.0,github.create_or_update_file github.create_repository github.create_branch github.create_issue github.get_file_contents github.list_commits github.get_file_blame github.list_branches github.list_tags github.get_tag +q143,Checkout a git branch,verb_product,git.git_checkout,1.0,1.0,1.0,1.0000,1.0000,1,1,330,1.0,1,545,1.0,1,960,1.0,git.git_checkout github.list_commits github.create_or_update_file github.create_branch github.push_files github.delete_file git.git_create_branch git.git_branch gitlab.gitlab_create_merge_request github.get_me +q144,Show git log,identifier,git.git_log,1.0,1.0,1.0,1.0000,1.0000,1,1,187,1.0,1,302,1.0,1,613,1.0,git.git_log git.git_status git.git_diff_unstaged git.git_diff_staged git.git_diff git.git_show github.get_me github.get_teams github.get_team_members github.search_repositories +q145,List postgres schemas,verb_product,postgres.list_schemas,1.0,1.0,1.0,1.0000,1.0000,1,1,172,1.0,1,347,1.0,1,724,1.0,postgres.list_schemas postgres.list_tables github.get_team_members github.list_commits github.list_branches github.list_tags github.list_releases github.list_starred_repositories github.list_repository_collaborators github.list_issues +q146,Open nodes in the knowledge graph,verb_product,memory.open_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,284,1.0,1,505,1.0,memory.open_nodes memory.search_nodes memory.create_entities memory.create_relations memory.add_observations memory.delete_entities memory.delete_relations memory.read_graph postgres.list_tables postgres.list_schemas +q147,Get file metadata from the filesystem,paraphrase,filesystem.read_file,0.0,0.0,1.0,0.2500,0.4307,4,2,463,0.0,1,385,1.0,1,750,1.0,filesystem.read_file filesystem.read_text_file filesystem.read_media_file filesystem.get_file_info github.get_file_contents google_drive.gdrive_get_file gitlab.gitlab_get_file figma.figma_get_file_nodes figma.figma_get_images github.get_me +q148,Write a new local file,paraphrase,filesystem.write_file,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,384,1.0,1,927,1.0,filesystem.write_file filesystem.read_file filesystem.read_text_file filesystem.read_media_file filesystem.edit_file github.get_file_contents github.get_file_blame github.create_or_update_file github.create_repository github.delete_file +q149,Search GitHub issues,verb_product,github.search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,265,1.0,1,420,1.0,1,667,1.0,github.search_issues github.search_repositories github.search_code github.search_commits github.list_issues github.search_pull_requests github.search_users github.search_orgs github.get_me github.get_teams +q150,Get the authenticated GitHub user,paraphrase,github.get_me,1.0,1.0,1.0,1.0000,1.0000,1,1,179,1.0,1,356,1.0,1,796,1.0,github.get_me github.get_teams github.list_notifications github.get_latest_release github.get_repository_tree github.get_team_members github.get_file_contents github.get_commit github.get_file_blame github.get_tag diff --git a/experiments/mcp-tool-search/results/boolean_or.json b/experiments/mcp-tool-search/results/boolean_or.json new file mode 100644 index 00000000..98f664f5 --- /dev/null +++ b/experiments/mcp-tool-search/results/boolean_or.json @@ -0,0 +1,6969 @@ +{ + "summary": { + "n": 150, + "index_s": 0.0, + "rss_mb": 0.08984375, + "p50_ms": 0.302081000029375, + "p95_ms": 0.3668339500336514, + "recall@1": 0.9466666666666667, + "recall@1_lo": 0.9066666666666666, + "recall@1_hi": 0.98, + "recall@3": 0.98, + "recall@3_lo": 0.9533333333333334, + "recall@3_hi": 1.0, + "recall@5": 0.9933333333333333, + "recall@5_lo": 0.98, + "recall@5_hi": 1.0, + "mrr@10": 0.9644444444444443, + "mrr@10_lo": 0.9366666666666666, + "mrr@10_hi": 0.9872222222222223, + "ndcg@10": 0.9592042219162568, + "ndcg@10_lo": 0.9358180619000194, + "ndcg@10_hi": 0.9779941382833974 + }, + "meta": { + "name": "boolean_or", + "needs_model": false, + "needs_api": false, + "notes": "token overlap count" + }, + "rows": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "github.issue_write", + "github.create_or_update_file", + "github.create_repository", + "github.create_branch", + "github.issue_read", + "github.add_issue_comment", + "github.update_issue_comment", + "github.create_pull_request", + "github.pull_request_review_write" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 434, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 614, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1216, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "kind": "verb_product", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_create_issue", + "jira.jira_create_sprint", + "zendesk.zendesk_create_ticket", + "jira.create_board", + "jira.create_sprint", + "jira.create_filter", + "jira.create_webhook", + "jira.create_attachment", + "jira.create_version", + "github.create_or_update_file" + ], + "hit1": "jira.jira_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 528, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1058, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "kind": "verb_product", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "ranking": [ + "slack.slack_post_message", + "slack.slack_add_reaction", + "discord.discord_send_message", + "github.create_or_update_file", + "github.push_files", + "github.delete_file", + "git.git_commit", + "slack.slack_reply_to_thread", + "slack.slack_get_channel_history", + "slack.slack_get_thread_replies" + ], + "hit1": "slack.slack_post_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 507, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 924, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "kind": "resource", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "ranking": [ + "github.list_pull_requests", + "bitbucket.bitbucket_list_prs", + "github.create_pull_request", + "github.search_pull_requests", + "playwright.browser_tab_list", + "gitlab.gitlab_list_merge_requests", + "github.get_team_members", + "github.list_commits", + "github.list_branches", + "github.list_tags" + ], + "hit1": "github.list_pull_requests", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9238850086262381, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 334, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 423, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 808, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "ranking": [ + "github.create_pull_request", + "github.request_copilot_review", + "github.pull_request_read", + "github.update_pull_request", + "github.merge_pull_request", + "github.pull_request_review_write", + "github.add_comment_to_pending_review", + "github.list_commits", + "github.list_repository_collaborators", + "github.update_issue_comment" + ], + "hit1": "github.create_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 364, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 611, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1194, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "kind": "verb_product", + "gold": [ + "postgres.query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "postgres.explain_query", + "salesforce.salesforce_query", + "postgres.list_tables", + "postgres.describe_table", + "postgres.list_schemas", + "github.search_repositories", + "github.search_code" + ], + "hit1": "postgres.query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 604, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "kind": "paraphrase", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "ranking": [ + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "github.get_file_contents", + "github.delete_file", + "gitlab.gitlab_get_file", + "figma.figma_get_file_nodes", + "figma.figma_get_images", + "github.get_file_blame", + "github.get_tag" + ], + "hit1": "filesystem.read_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 444, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 881, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "kind": "paraphrase", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "ranking": [ + "google_drive.gdrive_list_folder", + "github.push_files", + "google_drive.gdrive_search", + "slack.list_files", + "github.get_file_blame", + "github.list_branches", + "github.list_tags", + "github.create_or_update_file", + "github.create_branch", + "github.create_issue" + ], + "hit1": "google_drive.gdrive_list_folder", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 85, + "k3_calls": 29, + "k3_tokens": 7919, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 17, + "k5_tokens": 7541, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7663, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "ranking": [ + "memory.create_entities", + "memory.create_relations", + "memory.add_observations", + "memory.search_nodes", + "memory.open_nodes", + "memory.delete_entities", + "memory.delete_relations", + "memory.read_graph", + "postgres.list_tables", + "postgres.list_schemas" + ], + "hit1": "memory.create_entities", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 285, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 506, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "kind": "verb_product", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "ranking": [ + "brave_search.brave_web_search", + "github.update_pull_request_branch", + "memory.search_nodes", + "brave_search.brave_local_search", + "jira.jira_search_issues", + "github.get_me", + "github.get_teams", + "github.search_repositories", + "github.search_code", + "github.search_commits" + ], + "hit1": "brave_search.brave_web_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 410, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "kind": "verb_product", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "ranking": [ + "fetch.fetch", + "filesystem.read_file", + "github.get_me", + "github.get_repository_tree", + "git.git_status", + "git.git_add", + "slack.slack_set_channel_topic", + "playwright.browser_navigate", + "playwright.browser_snapshot", + "playwright.browser_take_screenshot" + ], + "hit1": "fetch.fetch", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 369, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 671, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "kind": "identifier", + "gold": [ + "git.git_status" + ], + "alts": [], + "ranking": [ + "git.git_status", + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_diff", + "git.git_log", + "git.git_show", + "github.get_me", + "github.get_teams", + "github.get_team_members", + "github.search_repositories" + ], + "hit1": "git.git_status", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 176, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 613, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "kind": "paraphrase", + "gold": [ + "git.git_commit" + ], + "alts": [], + "ranking": [ + "git.git_commit", + "git.git_reset", + "github.get_commit", + "github.push_files", + "git.git_diff_staged", + "git.git_log", + "git.git_show" + ], + "hit1": "git.git_commit", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_issue", + "linear.linear_create_comment", + "github.create_issue", + "github.issue_write", + "linear.linear_update_issue", + "linear.linear_get_issue", + "linear.linear_create_project", + "linear.linear_assign_issue", + "jira.jira_create_issue", + "gitlab.gitlab_create_issue" + ], + "hit1": "linear.linear_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 351, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 601, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1019, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "kind": "paraphrase", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_issue", + "linear.linear_create_project", + "jira.jira_create_issue", + "supabase.supabase_list_tables", + "linear.create_cycle", + "linear.get_cycle", + "linear.list_cycles", + "linear.update_cycle", + "linear.delete_cycle", + "linear.create_label" + ], + "hit1": "linear.linear_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 353, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 488, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 823, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "ranking": [ + "github.update_issue_comment", + "github.add_issue_comment", + "github.assign_copilot_to_issue", + "github.list_commits", + "github.list_repository_collaborators", + "github.create_issue", + "github.issue_write", + "github.issue_read", + "github.create_pull_request", + "github.add_comment_to_pending_review" + ], + "hit1": "github.update_issue_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 338, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 519, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1190, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "kind": "ambiguous", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "ranking": [ + "github.merge_pull_request", + "github.update_pull_request_branch", + "github.create_pull_request", + "github.pull_request_read", + "github.update_pull_request", + "github.pull_request_review_write", + "github.add_comment_to_pending_review", + "github.request_copilot_review", + "gitlab.gitlab_create_merge_request", + "gitlab.gitlab_get_merge_request" + ], + "hit1": "github.merge_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 370, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 602, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1196, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "kind": "verb_product", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "ranking": [ + "github.actions_run_trigger", + "github.actions_get", + "github.actions_list", + "github.actions_get_job_logs", + "github.get_me", + "github.get_teams", + "github.get_team_members", + "github.search_repositories", + "github.get_file_contents", + "github.list_commits" + ], + "hit1": "github.actions_run_trigger", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 442, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 848, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "kind": "resource", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "ranking": [ + "github.get_dependabot_alert", + "github.list_dependabot_alerts", + "github.get_me", + "github.get_teams", + "github.get_team_members", + "github.get_file_contents", + "github.get_commit", + "github.get_file_blame", + "github.get_tag", + "github.get_latest_release" + ], + "hit1": "github.get_dependabot_alert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8597186998521972, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 243, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 792, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "kind": "verb_product", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_issues", + "github.list_issues", + "linear.linear_list_issues", + "sentry.sentry_list_events", + "sentry.sentry_list_projects", + "gitlab.gitlab_list_issues", + "github.get_team_members", + "github.list_commits", + "github.list_branches", + "github.list_tags" + ], + "hit1": "sentry.sentry_list_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 302, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 425, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 809, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_customer", + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_invoice", + "stripe.stripe_get_customer", + "stripe.stripe_list_invoices", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "stripe.stripe_list_subscriptions", + "stripe.stripe_create_refund", + "github.create_or_update_file" + ], + "hit1": "stripe.stripe_create_customer", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 749, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "kind": "paraphrase", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_refund", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_list_customers", + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_create_product", + "stripe.stripe_create_price" + ], + "hit1": "stripe.stripe_create_refund", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "kind": "verb_product", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "ranking": [ + "gmail.gmail_send_email", + "puppeteer.puppeteer_click", + "puppeteer.puppeteer_fill", + "gmail.gmail_get_email", + "gmail.gmail_modify_email", + "gmail.gmail_trash_email", + "twilio.twilio_send_sms", + "github.create_issue", + "github.update_pull_request_branch", + "github.assign_copilot_to_issue" + ], + "hit1": "gmail.gmail_send_email", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 859, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "kind": "verb_product", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "ranking": [ + "gmail.gmail_search_emails", + "github.search_repositories", + "github.search_code", + "github.search_commits", + "github.search_issues", + "github.search_pull_requests", + "github.search_users", + "github.search_orgs", + "filesystem.search_files", + "memory.search_nodes" + ], + "hit1": "gmail.gmail_search_emails", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "kind": "paraphrase", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "ranking": [ + "slack.slack_upload_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_update_file", + "google_drive.gdrive_share_file", + "filesystem.edit_file", + "git.git_add", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "aws_s3.s3_put_object" + ], + "hit1": "slack.slack_upload_file", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 446, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 801, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_create_event", + "github.pull_request_review_write", + "google_calendar.gcal_update_event", + "google_calendar.gcal_delete_event", + "github.create_or_update_file", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.issue_write", + "github.create_pull_request" + ], + "hit1": "google_calendar.gcal_create_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 371, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 588, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1186, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_list_pods", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "github.get_team_members", + "github.list_commits", + "github.list_branches", + "github.list_tags", + "github.list_releases", + "github.list_starred_repositories" + ], + "hit1": "kubernetes.k8s_list_pods", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_pod_logs", + "github.get_file_contents", + "github.get_tag", + "github.actions_get_job_logs", + "slack.slack_get_channel_history", + "docker.docker_logs", + "kubernetes.k8s_get_pod", + "datadog.datadog_list_logs", + "gitlab.gitlab_get_file", + "elasticsearch.es_get_doc" + ], + "hit1": "kubernetes.k8s_pod_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 314, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 481, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 881, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "ranking": [ + "docker.docker_run", + "docker.docker_exec", + "docker.docker_logs", + "docker.docker_stop", + "docker.docker_rm", + "github.actions_get", + "github.actions_run_trigger", + "postgres.query", + "docker.docker_ps", + "docker.docker_images" + ], + "hit1": "docker.docker_run", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "kind": "verb_product", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "ranking": [ + "docker.docker_logs", + "git.git_log", + "docker.docker_run", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_exec", + "kubernetes.k8s_pod_logs", + "github.actions_get_job_logs", + "git.git_status", + "git.git_diff_unstaged" + ], + "hit1": "docker.docker_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "kind": "ambiguous", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "ranking": [ + "playwright.browser_navigate", + "puppeteer.puppeteer_navigate", + "filesystem.list_allowed_directories", + "memory.add_observations", + "fetch.fetch", + "git.git_commit", + "git.git_add", + "playwright.browser_snapshot", + "playwright.browser_click", + "playwright.browser_take_screenshot" + ], + "hit1": "playwright.browser_navigate", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 170, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 304, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 609, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "kind": "ambiguous", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "ranking": [ + "playwright.browser_take_screenshot", + "playwright.browser_snapshot", + "github.get_me", + "github.get_repository_tree", + "filesystem.read_file", + "git.git_status", + "slack.slack_set_channel_topic", + "playwright.browser_click", + "playwright.browser_evaluate", + "playwright.browser_press_key" + ], + "hit1": "playwright.browser_take_screenshot", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 642, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_create_merge_request", + "gitlab.gitlab_get_merge_request", + "gitlab.gitlab_merge_merge_request", + "github.create_pull_request", + "github.merge_pull_request", + "github.pull_request_review_write", + "gitlab.gitlab_create_issue", + "gitlab.gitlab_list_merge_requests", + "gitlab.gitlab_create_pipeline", + "github.create_or_update_file" + ], + "hit1": "gitlab.gitlab_create_merge_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 552, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1075, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "kind": "verb_product", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "ranking": [ + "notion.notion_search", + "github.search_repositories", + "github.search_code", + "github.search_commits", + "github.search_issues", + "github.search_pull_requests", + "github.search_users", + "github.search_orgs", + "filesystem.search_files", + "memory.search_nodes" + ], + "hit1": "notion.notion_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 248, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 387, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "kind": "verb_product", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "ranking": [ + "notion.notion_create_page", + "notion.notion_get_page", + "notion.notion_update_page", + "notion.notion_create_database", + "notion.notion_get_comments", + "confluence.confluence_create_page", + "notion.create_comment", + "notion.create_users_token", + "notion.create_data_source", + "github.search_repositories" + ], + "hit1": "notion.notion_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 856, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "kind": "verb_product", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "ranking": [ + "notion.notion_query_database", + "notion.notion_search", + "notion.notion_create_page", + "notion.notion_create_database", + "github.search_repositories", + "github.search_code", + "github.search_commits", + "github.search_issues", + "github.search_pull_requests", + "github.search_users" + ], + "hit1": "notion.notion_query_database", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 439, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 749, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_create_ticket", + "jira.jira_create_issue", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "github.create_or_update_file", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.issue_write", + "github.create_pull_request" + ], + "hit1": "zendesk.zendesk_create_ticket", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 282, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 517, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1115, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_create_deal", + "github.create_or_update_file", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.issue_write", + "github.create_pull_request", + "github.pull_request_review_write" + ], + "hit1": "hubspot.hubspot_create_contact", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 480, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1141, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "kind": "verb_product", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_create_lead", + "salesforce.salesforce_create_record", + "salesforce.salesforce_convert_lead", + "github.create_or_update_file", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.issue_write", + "github.create_pull_request", + "github.pull_request_review_write" + ], + "hit1": "salesforce.salesforce_create_lead", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 477, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1138, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "kind": "identifier", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_query", + "postgres.query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "github.search_repositories", + "github.search_code", + "github.search_commits", + "github.search_issues", + "github.search_pull_requests", + "github.actions_get" + ], + "hit1": "salesforce.salesforce_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 676, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "kind": "verb_product", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "ranking": [ + "shopify.shopify_list_orders", + "shopify.shopify_list_products", + "shopify.shopify_list_customers", + "github.get_team_members", + "github.list_commits", + "github.list_branches", + "github.list_tags", + "github.list_releases", + "github.list_starred_repositories", + "github.list_repository_collaborators" + ], + "hit1": "shopify.shopify_list_orders", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "github.create_or_update_file", + "github.create_repository", + "github.create_branch", + "github.create_issue" + ], + "hit1": "pagerduty.pagerduty_create_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 922, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "kind": "paraphrase", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_list_oncalls", + "memory.search_nodes", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "github.list_commits", + "github.get_file_blame", + "github.list_branches" + ], + "hit1": "pagerduty.pagerduty_list_oncalls", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 220, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "kind": "verb_product", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "ranking": [ + "datadog.datadog_query_metrics", + "datadog.datadog_list_monitors", + "datadog.datadog_create_monitor", + "datadog.datadog_list_logs", + "github.search_repositories", + "github.search_code", + "github.search_commits", + "github.search_issues", + "github.search_pull_requests", + "github.search_users" + ], + "hit1": "datadog.datadog_query_metrics", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 490, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 800, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "kind": "verb_product", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "ranking": [ + "datadog.datadog_list_logs", + "github.search_repositories", + "github.search_code", + "github.search_commits", + "github.search_issues", + "github.search_pull_requests", + "github.actions_get_job_logs", + "github.search_users", + "github.search_orgs", + "filesystem.search_files" + ], + "hit1": "datadog.datadog_list_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 279, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 735, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_list_buckets", + "github.get_team_members", + "github.list_commits", + "github.list_branches", + "github.list_tags", + "github.list_releases", + "github.list_starred_repositories", + "github.list_repository_collaborators", + "github.list_issues", + "github.list_pull_requests" + ], + "hit1": "aws_s3.s3_list_buckets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 768, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "kind": "paraphrase", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_put_object", + "github.assign_copilot_to_issue", + "slack.slack_add_reaction", + "slack.slack_upload_file", + "gmail.gmail_send_email", + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "twilio.twilio_send_sms", + "github.push_files", + "github.create_issue" + ], + "hit1": "aws_s3.s3_put_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 303, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 479, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 954, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "cloudflare.cf_list_dns", + "cloudflare.cf_create_dns" + ], + "hit1": "cloudflare.cf_purge_cache", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_create_dns", + "salesforce.salesforce_create_record", + "cloudflare.cf_list_dns", + "github.create_or_update_file", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.issue_write", + "github.create_pull_request", + "github.pull_request_review_write" + ], + "hit1": "cloudflare.cf_create_dns", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 474, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1135, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_deployments", + "kubernetes.k8s_list_deployments", + "vercel.vercel_list_projects", + "github.get_team_members", + "github.list_commits", + "github.list_branches", + "github.list_tags", + "github.list_releases", + "github.list_starred_repositories", + "github.list_repository_collaborators" + ], + "hit1": "vercel.vercel_list_deployments", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 173, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 690, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "ranking": [ + "supabase.supabase_insert", + "mongodb.mongo_insert", + "playwright.browser_type", + "supabase.supabase_list_tables", + "supabase.supabase_select", + "supabase.supabase_update", + "supabase.supabase_invoke_fn" + ], + "hit1": "supabase.supabase_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 347, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "kind": "paraphrase", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "ranking": [ + "figma.figma_get_images", + "figma.figma_get_file_nodes", + "github.get_file_contents", + "github.get_tag", + "github.create_branch", + "github.delete_file", + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "memory.delete_entities" + ], + "hit1": "figma.figma_get_images", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 292, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 477, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 844, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "kind": "verb_product", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "ranking": [ + "twilio.twilio_send_sms", + "puppeteer.puppeteer_click", + "puppeteer.puppeteer_fill", + "gmail.gmail_send_email", + "twilio.twilio_list_messages", + "github.create_issue", + "github.update_pull_request_branch", + "github.assign_copilot_to_issue", + "filesystem.read_media_file", + "filesystem.write_file" + ], + "hit1": "twilio.twilio_send_sms", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 836, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "kind": "verb_product", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "ranking": [ + "okta.okta_create_user", + "okta.okta_get_user", + "okta.okta_deactivate_user", + "github.create_issue", + "filesystem.write_file", + "asana.asana_create_task", + "asana.asana_create_project", + "linear.create_user", + "github.get_me", + "github.get_teams" + ], + "hit1": "okta.okta_create_user", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 748, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query", + "prometheus.prom_query_range", + "github.search_repositories", + "github.search_code", + "github.search_commits", + "github.search_issues", + "github.search_pull_requests", + "github.search_users", + "github.search_orgs", + "memory.search_nodes" + ], + "hit1": "prometheus.prom_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 383, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 681, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "kind": "verb_product", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "ranking": [ + "grafana.grafana_search_dashboards", + "github.search_repositories", + "github.search_code", + "github.search_commits", + "github.search_issues", + "github.search_pull_requests", + "github.search_users", + "github.search_orgs", + "filesystem.search_files", + "memory.search_nodes" + ], + "hit1": "grafana.grafana_search_dashboards", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 657, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "kind": "verb_product", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "ranking": [ + "circleci.circleci_trigger_pipeline", + "circleci.circleci_get_pipeline", + "gitlab.gitlab_create_pipeline", + "circleci.circleci_list_pipelines", + "github.actions_run_trigger", + "hubspot.hubspot_create_deal", + "gitlab.create_pipeline", + "gitlab.get_pipeline", + "gitlab.update_pipeline", + "gitlab.delete_pipeline" + ], + "hit1": "circleci.circleci_trigger_pipeline", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "kind": "identifier", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "ranking": [ + "terraform.tf_plan", + "terraform.tf_apply", + "github.actions_get", + "github.actions_run_trigger", + "postgres.query", + "postgres.explain_query", + "docker.docker_run", + "docker.docker_exec", + "salesforce.salesforce_query", + "sqlite.sqlite_query" + ], + "hit1": "terraform.tf_plan", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 210, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "kind": "verb_product", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "ranking": [ + "asana.asana_create_task", + "asana.asana_get_task", + "asana.asana_update_task", + "asana.asana_create_project", + "github.create_issue", + "filesystem.write_file", + "okta.okta_create_user", + "clickup.clickup_create_task", + "github.create_or_update_file", + "github.create_repository" + ], + "hit1": "asana.asana_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 908, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "ranking": [ + "trello.trello_create_card", + "trello.trello_create_board", + "trello.trello_add_comment", + "github.create_or_update_file", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.issue_write", + "github.create_pull_request", + "github.pull_request_review_write" + ], + "hit1": "trello.trello_create_card", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 465, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1126, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_create_task", + "clickup.clickup_get_task", + "clickup.clickup_update_task", + "asana.asana_create_task", + "clickup.clickup_list_tasks", + "github.create_or_update_file", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.issue_write" + ], + "hit1": "clickup.clickup_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 415, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1008, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "kind": "verb_product", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "ranking": [ + "confluence.confluence_create_page", + "notion.notion_create_page", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "github.search_repositories", + "github.list_commits", + "github.search_code", + "github.create_or_update_file", + "github.create_repository", + "github.create_branch" + ], + "hit1": "confluence.confluence_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 438, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 934, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "kind": "verb_product", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "discord.discord_add_reaction", + "github.create_or_update_file", + "github.push_files", + "github.delete_file", + "git.git_commit", + "slack.slack_post_message", + "slack.slack_add_reaction", + "datadog.datadog_create_monitor", + "gmail.gmail_send_email" + ], + "hit1": "discord.discord_send_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 320, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 549, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 981, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_search", + "github.search_repositories", + "github.search_code", + "github.search_commits", + "github.search_issues", + "github.search_pull_requests", + "github.search_users", + "github.search_orgs", + "filesystem.search_files", + "memory.search_nodes" + ], + "hit1": "elasticsearch.es_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 243, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "jira.jira_get_issue", + "redis.redis_set", + "aws_s3.s3_get_object", + "figma.figma_get_file", + "figma.figma_get_file_nodes", + "figma.figma_get_comments", + "figma.figma_get_images", + "github.get_me", + "github.get_teams" + ], + "hit1": "redis.redis_get", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 627, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_find", + "mongodb.mongo_insert", + "mongodb.mongo_update", + "mongodb.mongo_delete", + "mongodb.mongo_list_collections" + ], + "hit1": "mongodb.mongo_find", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "kind": "verb_product", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "ranking": [ + "sqlite.sqlite_list_tables", + "postgres.list_tables", + "sqlite.sqlite_query", + "sqlite.sqlite_describe_table", + "mysql.mysql_list_tables", + "supabase.supabase_list_tables", + "github.get_team_members", + "github.list_commits", + "github.list_branches", + "github.list_tags" + ], + "hit1": "sqlite.sqlite_list_tables", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 156, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 252, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 604, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "kind": "verb_product", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "ranking": [ + "postgres.describe_table", + "sqlite.sqlite_describe_table", + "postgres.query", + "postgres.list_tables", + "postgres.list_schemas", + "postgres.explain_query", + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update" + ], + "hit1": "postgres.describe_table", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 272, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "github.issue_write", + "github.create_or_update_file", + "github.create_repository", + "github.create_branch", + "github.issue_read", + "github.add_issue_comment", + "github.update_issue_comment", + "github.create_pull_request", + "github.pull_request_review_write" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 432, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 612, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1214, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "ranking": [ + "github.create_issue", + "github.issue_write", + "linear.linear_create_issue", + "linear.linear_create_comment", + "jira.jira_create_issue", + "gitlab.gitlab_create_issue", + "bitbucket.bitbucket_create_issue", + "github.create_or_update_file", + "github.create_repository", + "github.create_branch" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9220433016555235, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 418, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 600, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1107, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "kind": "identifier", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "ranking": [ + "filesystem.list_directory", + "filesystem.list_directory_with_sizes", + "github.get_team_members", + "github.get_file_contents", + "github.list_commits", + "github.list_branches", + "github.list_tags", + "github.list_releases", + "github.list_starred_repositories", + "github.list_repository_collaborators" + ], + "hit1": "filesystem.list_directory", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 198, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 408, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "kind": "paraphrase", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "ranking": [ + "sequential_thinking.sequentialthinking" + ], + "hit1": "sequential_thinking.sequentialthinking", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "kind": "paraphrase", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "ranking": [ + "time.get_current_time", + "github.get_file_blame", + "github.list_branches", + "github.list_tags", + "github.create_or_update_file", + "github.create_branch", + "github.push_files", + "github.create_issue", + "github.list_issues", + "github.list_pull_requests" + ], + "hit1": "time.get_current_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 241, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 456, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 993, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "kind": "verb_product", + "gold": [ + "time.convert_time" + ], + "alts": [], + "ranking": [ + "time.convert_time", + "memory.create_relations", + "git.git_diff", + "time.get_current_time", + "jira.jira_add_worklog", + "playwright.browser_wait_for", + "salesforce.salesforce_convert_lead", + "google_calendar.gcal_list_events", + "grafana.grafana_create_annotation" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "kind": "verb_product", + "gold": [ + "github.star_repository" + ], + "alts": [], + "ranking": [ + "github.star_repository", + "github.get_file_contents", + "github.list_commits", + "github.get_commit", + "github.get_file_blame", + "github.list_branches", + "github.list_tags", + "github.get_tag", + "github.list_releases", + "github.get_latest_release" + ], + "hit1": "github.star_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 298, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 479, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 849, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "kind": "resource", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "ranking": [ + "github.fork_repository", + "github.get_file_contents", + "github.list_commits", + "github.get_file_blame", + "github.list_branches", + "github.list_tags", + "github.get_tag", + "github.list_releases", + "github.create_or_update_file", + "github.create_repository" + ], + "hit1": "github.fork_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 311, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 480, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 932, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "kind": "verb_product", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "ranking": [ + "github.get_latest_release", + "github.get_me", + "github.get_teams", + "github.get_release_by_tag", + "github.get_repository_tree", + "github.get_team_members", + "github.get_file_contents", + "github.get_commit", + "github.get_file_blame", + "github.get_tag" + ], + "hit1": "github.get_latest_release", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 383, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 823, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "kind": "resource", + "gold": [ + "github.create_gist" + ], + "alts": [], + "ranking": [ + "github.create_gist", + "github.create_or_update_file", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.issue_write", + "github.create_pull_request", + "github.pull_request_review_write", + "github.get_gist", + "github.update_gist" + ], + "hit1": "github.create_gist", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 322, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 554, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1115, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "kind": "verb_product", + "gold": [ + "github.search_code" + ], + "alts": [], + "ranking": [ + "github.search_code", + "github.search_repositories", + "github.search_commits", + "github.search_issues", + "github.search_pull_requests", + "github.get_code_scanning_alert", + "github.list_code_scanning_alerts", + "github.search_users", + "github.search_orgs", + "github.get_me" + ], + "hit1": "github.search_code", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 722, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "kind": "verb_product", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "ranking": [ + "github.list_notifications", + "github.get_team_members", + "github.list_commits", + "github.list_branches", + "github.list_tags", + "github.list_releases", + "github.list_starred_repositories", + "github.list_repository_collaborators", + "github.list_issues", + "github.list_pull_requests" + ], + "hit1": "github.list_notifications", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 792, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "kind": "verb_product", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "ranking": [ + "github.assign_copilot_to_issue", + "linear.linear_assign_issue", + "github.create_issue", + "github.add_issue_comment", + "slack.slack_add_reaction", + "linear.linear_create_comment", + "jira.jira_update_issue", + "jira.jira_add_comment", + "jira.jira_transition_issue", + "jira.jira_assign_issue" + ], + "hit1": "github.assign_copilot_to_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 332, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 527, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 864, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "kind": "resource", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "ranking": [ + "github.get_secret_scanning_alert", + "github.list_secret_scanning_alerts", + "github.get_code_scanning_alert", + "github.list_code_scanning_alerts", + "github.get_me", + "github.get_teams", + "github.get_team_members", + "github.get_file_contents", + "github.get_commit", + "github.get_file_blame" + ], + "hit1": "github.get_secret_scanning_alert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8597186998521972, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 446, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 839, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "ranking": [ + "github.create_or_update_file", + "github.issue_write", + "github.projects_write", + "github.label_write", + "github.get_file_contents", + "github.update_issue_comment", + "github.create_pull_request", + "github.update_pull_request", + "github.pull_request_review_write", + "github.search_repositories" + ], + "hit1": "github.create_or_update_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 389, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 612, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1243, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "kind": "resource", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "ranking": [ + "github.list_repository_collaborators", + "github.list_commits", + "github.list_branches", + "github.list_tags", + "github.list_releases", + "github.list_issues", + "github.update_issue_comment", + "github.create_pull_request", + "github.list_pull_requests", + "github.assign_copilot_to_issue" + ], + "hit1": "github.list_repository_collaborators", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 274, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 965, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "kind": "verb_product", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "ranking": [ + "slack.slack_reply_to_thread", + "slack.slack_get_thread_replies", + "slack.slack_list_channels", + "slack.create_reminder", + "slack.get_reminder", + "slack.list_reminders", + "slack.update_reminder", + "slack.delete_reminder", + "slack.create_canvase", + "slack.get_canvase" + ], + "hit1": "slack.slack_reply_to_thread", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 411, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "kind": "verb_product", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "ranking": [ + "slack.slack_add_reaction", + "discord.discord_add_reaction", + "github.assign_copilot_to_issue", + "playwright.browser_click", + "gmail.gmail_modify_email", + "trello.trello_add_comment", + "github.list_commits", + "github.list_repository_collaborators", + "github.create_issue", + "github.add_issue_comment" + ], + "hit1": "slack.slack_add_reaction", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 304, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 461, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 950, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "kind": "verb_product", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "ranking": [ + "slack.slack_search_messages", + "slack.slack_get_channel_history", + "gmail.gmail_search_emails", + "github.search_repositories", + "github.search_code", + "github.search_commits", + "github.search_issues", + "github.search_pull_requests", + "github.search_users", + "github.search_orgs" + ], + "hit1": "slack.slack_search_messages", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 685, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "kind": "verb_product", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "ranking": [ + "slack.slack_create_channel", + "slack.slack_post_message", + "slack.slack_reply_to_thread", + "slack.slack_add_reaction", + "slack.slack_get_channel_history", + "slack.slack_get_thread_replies", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_set_channel_topic", + "slack.slack_upload_file" + ], + "hit1": "slack.slack_create_channel", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 414, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "kind": "verb_product", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_transition_issue", + "jira.jira_create_issue", + "jira.jira_update_issue", + "jira.jira_get_issue", + "jira.jira_add_comment", + "jira.jira_assign_issue", + "jira.jira_add_worklog", + "github.create_issue", + "github.issue_write", + "github.issue_read" + ], + "hit1": "jira.jira_transition_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 270, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 914, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "kind": "identifier", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "ranking": [ + "jira.jira_search_issues", + "github.search_repositories", + "github.search_code", + "github.search_commits", + "github.search_issues", + "github.search_pull_requests", + "github.update_pull_request_branch", + "github.search_users", + "github.search_orgs", + "filesystem.list_directory_with_sizes" + ], + "hit1": "jira.jira_search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 700, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "kind": "verb_product", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "ranking": [ + "jira.jira_add_worklog", + "jira.jira_add_comment", + "github.add_issue_comment", + "github.add_comment_to_pending_review", + "memory.add_observations", + "git.git_add", + "slack.slack_add_reaction", + "linear.linear_create_comment", + "jira.jira_transition_issue", + "discord.discord_add_reaction" + ], + "hit1": "jira.jira_add_worklog", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 285, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 495, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 874, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "kind": "verb_product", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "ranking": [ + "linear.linear_list_teams", + "linear.linear_list_issues", + "linear.linear_list_projects", + "linear.list_cycles", + "linear.list_labels", + "linear.list_users", + "linear.list_comments", + "linear.list_attachments", + "github.get_teams", + "github.get_team_members" + ], + "hit1": "linear.linear_list_teams", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "ranking": [ + "github.update_issue_comment", + "linear.linear_create_comment", + "github.add_issue_comment", + "github.assign_copilot_to_issue", + "linear.linear_create_issue", + "linear.linear_update_issue", + "linear.linear_get_issue", + "linear.linear_assign_issue", + "jira.jira_add_comment", + "jira.jira_add_worklog" + ], + "hit1": "github.update_issue_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 298, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 534, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 936, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "kind": "paraphrase", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "ranking": [ + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "filesystem.move_file", + "github.get_file_contents", + "github.get_file_blame", + "github.create_or_update_file", + "github.delete_file", + "filesystem.write_file", + "filesystem.edit_file" + ], + "hit1": "filesystem.read_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 506, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 907, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "kind": "paraphrase", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "ranking": [ + "filesystem.search_files", + "github.search_repositories", + "linear.linear_search_issues", + "google_drive.gdrive_search", + "github.search_code", + "github.search_commits", + "github.get_commit", + "github.get_release_by_tag", + "github.push_files", + "github.list_starred_repositories" + ], + "hit1": "filesystem.search_files", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 236, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 359, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "kind": "paraphrase", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "ranking": [ + "filesystem.edit_file", + "github.get_file_blame", + "github.create_or_update_file", + "filesystem.write_file", + "filesystem.list_directory_with_sizes", + "filesystem.move_file", + "google_drive.gdrive_create_file", + "slack.create_file", + "slack.get_file", + "slack.update_file" + ], + "hit1": "filesystem.edit_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 360, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 489, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 875, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "ranking": [ + "memory.read_graph", + "memory.create_entities", + "memory.create_relations", + "memory.add_observations", + "memory.delete_entities", + "memory.delete_relations", + "memory.search_nodes", + "memory.open_nodes", + "filesystem.read_file", + "filesystem.read_text_file" + ], + "hit1": "memory.read_graph", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 154, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 561, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "kind": "verb_product", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "memory.open_nodes", + "github.search_repositories", + "github.search_code", + "github.search_commits", + "github.search_issues", + "github.search_pull_requests", + "github.search_users", + "github.search_orgs", + "filesystem.search_files" + ], + "hit1": "memory.search_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "kind": "verb_product", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "ranking": [ + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_diff", + "git.git_status", + "git.git_log", + "git.git_show", + "github.get_me", + "github.get_teams", + "github.get_team_members", + "github.search_repositories" + ], + "hit1": "git.git_diff_unstaged", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 304, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 615, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "kind": "identifier", + "gold": [ + "git.git_add" + ], + "alts": [], + "ranking": [ + "git.git_add", + "github.push_files", + "github.add_issue_comment", + "github.add_comment_to_pending_review", + "github.create_gist", + "github.update_gist", + "filesystem.list_directory_with_sizes", + "github.get_me", + "github.get_teams", + "github.get_team_members" + ], + "hit1": "git.git_add", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 306, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 538, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 836, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "kind": "verb_product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "ranking": [ + "shopify.shopify_create_product", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "shopify.shopify_get_product", + "shopify.shopify_update_product", + "shopify.shopify_create_order", + "github.create_or_update_file", + "github.create_repository", + "github.create_branch", + "github.create_issue" + ], + "hit1": "shopify.shopify_create_product", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 914, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "kind": "verb_product", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "ranking": [ + "stripe.stripe_cancel_subscription", + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_list_customers", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_create_product", + "stripe.stripe_create_price" + ], + "hit1": "stripe.stripe_cancel_subscription", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 643, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "kind": "resource", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_refund", + "stripe.stripe_get_payment_intent", + "github.create_or_update_file", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.issue_write", + "github.create_pull_request", + "github.pull_request_review_write" + ], + "hit1": "stripe.stripe_create_payment_intent", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1120, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "kind": "verb_product", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "ranking": [ + "datadog.datadog_create_monitor", + "datadog.datadog_get_monitor", + "github.create_or_update_file", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.issue_write", + "github.create_pull_request", + "github.pull_request_review_write", + "github.create_gist" + ], + "hit1": "datadog.datadog_create_monitor", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 341, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 521, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1160, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "kind": "paraphrase", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "ranking": [ + "sentry.sentry_get_issue", + "sentry.sentry_list_events", + "sentry.sentry_update_issue", + "github.create_issue", + "github.issue_write", + "github.issue_read", + "github.add_issue_comment", + "github.update_issue_comment", + "github.assign_copilot_to_issue", + "linear.linear_create_issue" + ], + "hit1": "sentry.sentry_get_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 519, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1064, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_apply", + "kubernetes.k8s_delete", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "vercel.vercel_get_deployment" + ], + "hit1": "kubernetes.k8s_scale_deployment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 253, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_apply", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_delete", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "terraform.tf_apply" + ], + "hit1": "kubernetes.k8s_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 222, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 408, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "ranking": [ + "docker.docker_stop", + "docker.docker_logs", + "docker.docker_run", + "docker.docker_rm", + "docker.docker_exec", + "docker.docker_ps", + "docker.docker_images", + "docker.docker_build", + "kubernetes.k8s_pod_logs" + ], + "hit1": "docker.docker_stop", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "kind": "verb_product", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "ranking": [ + "docker.docker_build", + "docker.docker_run", + "filesystem.read_media_file", + "docker.docker_ps", + "docker.docker_logs", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_images", + "docker.docker_exec" + ], + "hit1": "docker.docker_build", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 236, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "kind": "verb_product", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "ranking": [ + "playwright.browser_click", + "playwright.browser_type", + "playwright.browser_take_screenshot", + "puppeteer.puppeteer_click", + "github.create_issue", + "playwright.browser_snapshot", + "aws_s3.s3_list_objects", + "github.get_file_blame", + "github.list_branches", + "github.list_tags" + ], + "hit1": "playwright.browser_click", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 236, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 422, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 772, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "kind": "paraphrase", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "ranking": [ + "playwright.browser_type", + "playwright.browser_wait_for", + "github.projects_list", + "filesystem.read_text_file", + "filesystem.edit_file", + "slack.slack_post_message", + "slack.slack_reply_to_thread", + "linear.linear_search_issues", + "playwright.browser_navigate", + "playwright.browser_snapshot" + ], + "hit1": "playwright.browser_type", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 410, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_list_projects", + "github.projects_list", + "gitlab.gitlab_list_issues", + "gitlab.gitlab_list_merge_requests", + "github.get_team_members", + "github.list_commits", + "github.list_branches", + "github.list_tags", + "github.list_releases", + "github.list_starred_repositories" + ], + "hit1": "gitlab.gitlab_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 209, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "kind": "verb_product", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "ranking": [ + "bitbucket.bitbucket_create_pr", + "github.create_pull_request", + "github.pull_request_review_write", + "github.pull_request_read", + "github.update_pull_request", + "github.merge_pull_request", + "github.update_pull_request_branch", + "github.add_comment_to_pending_review", + "github.request_copilot_review", + "gitlab.gitlab_create_merge_request" + ], + "hit1": "bitbucket.bitbucket_create_pr", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 439, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 671, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1243, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "kind": "verb_product", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "ranking": [ + "gmail.gmail_create_draft", + "github.create_pull_request", + "github.create_or_update_file", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.issue_write", + "github.pull_request_review_write", + "github.create_gist", + "github.projects_write" + ], + "hit1": "gmail.gmail_create_draft", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 396, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 576, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1149, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "kind": "verb_product", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_share_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_update_file", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "github.get_file_contents", + "github.get_file_blame", + "github.create_or_update_file", + "github.delete_file" + ], + "hit1": "google_drive.gdrive_share_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 271, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 936, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "kind": "verb_product", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "kubernetes.k8s_get_events", + "sentry.sentry_list_events", + "google_drive.gdrive_list_folder", + "google_calendar.gcal_create_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_delete_event", + "github.get_team_members", + "github.list_commits" + ], + "hit1": "google_calendar.gcal_list_events", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 199, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_update_event", + "github.delete_repository", + "github.delete_file", + "github.pull_request_review_write", + "github.label_write", + "memory.delete_entities", + "memory.delete_observations", + "memory.delete_relations" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 481, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 895, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_projects", + "github.projects_list", + "linear.linear_list_projects", + "jira.jira_list_projects", + "sentry.sentry_list_projects", + "gitlab.gitlab_list_projects", + "vercel.vercel_list_deployments", + "github.get_team_members", + "github.list_commits", + "github.list_branches" + ], + "hit1": "vercel.vercel_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 256, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 607, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "ranking": [ + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update", + "github.get_file_contents", + "github.get_tag", + "github.create_branch", + "github.delete_file", + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file" + ], + "hit1": "supabase.supabase_select", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 844, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "kind": "verb_product", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "github.add_issue_comment", + "github.update_issue_comment", + "github.add_comment_to_pending_review", + "slack.slack_post_message", + "linear.linear_create_comment", + "jira.jira_add_comment", + "jira.jira_add_worklog", + "figma.figma_get_file", + "figma.figma_get_file_nodes" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 304, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 528, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 883, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "kind": "verb_product", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "ranking": [ + "okta.okta_list_users", + "slack.slack_get_users", + "slack.slack_list_users", + "notion.notion_list_users", + "pagerduty.pagerduty_list_oncalls", + "linear.list_users", + "notion.list_users_tokens", + "github.get_team_members", + "github.list_commits", + "github.list_branches" + ], + "hit1": "okta.okta_list_users", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 203, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query_range", + "prometheus.prom_query", + "github.search_repositories", + "github.search_code", + "github.search_commits", + "github.search_issues", + "github.search_pull_requests", + "github.search_users", + "github.search_orgs", + "memory.search_nodes" + ], + "hit1": "prometheus.prom_query_range", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 383, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 681, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "kind": "verb_product", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "ranking": [ + "grafana.grafana_create_annotation", + "github.create_or_update_file", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.issue_write", + "github.create_pull_request", + "github.pull_request_review_write", + "github.create_gist", + "github.projects_write" + ], + "hit1": "grafana.grafana_create_annotation", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 331, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 563, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1154, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "kind": "identifier", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "ranking": [ + "terraform.tf_apply", + "kubernetes.k8s_apply", + "terraform.tf_plan", + "terraform.tf_state_list" + ], + "hit1": "terraform.tf_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "kind": "verb_product", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "ranking": [ + "asana.asana_list_tasks", + "clickup.clickup_list_tasks", + "github.get_team_members", + "github.list_commits", + "github.list_branches", + "github.list_tags", + "github.list_releases", + "github.list_starred_repositories", + "github.list_repository_collaborators", + "github.list_issues" + ], + "hit1": "asana.asana_list_tasks", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 207, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "ranking": [ + "trello.trello_add_comment", + "github.update_issue_comment", + "figma.figma_post_comment", + "trello.trello_create_card", + "trello.trello_list_cards", + "github.list_commits", + "github.list_repository_collaborators", + "github.add_issue_comment", + "github.create_pull_request", + "github.add_comment_to_pending_review" + ], + "hit1": "trello.trello_add_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 271, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 438, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1028, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_update_task", + "clickup.clickup_create_task", + "clickup.clickup_get_task", + "asana.asana_update_task", + "clickup.clickup_list_tasks", + "github.create_or_update_file", + "github.issue_write", + "github.update_issue_comment", + "github.update_pull_request", + "github.update_pull_request_branch" + ], + "hit1": "clickup.clickup_update_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1022, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "kind": "verb_product", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "ranking": [ + "confluence.confluence_search", + "github.search_repositories", + "github.search_code", + "github.search_commits", + "github.search_issues", + "github.search_pull_requests", + "github.search_users", + "github.search_orgs", + "filesystem.search_files", + "memory.search_nodes" + ], + "hit1": "confluence.confluence_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 651, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_deal", + "hubspot.hubspot_create_contact", + "github.create_or_update_file", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.issue_write", + "github.create_pull_request", + "github.pull_request_review_write", + "github.create_gist" + ], + "hit1": "hubspot.hubspot_create_deal", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 342, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 522, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1161, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_search", + "github.get_team_members", + "github.list_commits", + "github.list_branches", + "github.list_tags", + "github.list_releases", + "github.list_starred_repositories", + "github.list_repository_collaborators", + "github.list_issues" + ], + "hit1": "zendesk.zendesk_list_tickets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 198, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 750, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "kind": "verb_product", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "ranking": [ + "discord.discord_list_channels", + "slack.slack_list_channels", + "discord.discord_list_members", + "github.get_team_members", + "github.list_commits", + "github.list_branches", + "github.list_tags", + "github.list_releases", + "github.list_starred_repositories", + "github.list_repository_collaborators" + ], + "hit1": "discord.discord_list_channels", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 701, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "ranking": [ + "redis.redis_set", + "redis.redis_get", + "slack.slack_set_channel_topic", + "jira.jira_create_issue", + "jira.jira_update_issue", + "jira.jira_get_issue", + "jira.jira_add_comment", + "jira.jira_transition_issue", + "jira.jira_assign_issue", + "jira.jira_add_worklog" + ], + "hit1": "redis.redis_set", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 392, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert", + "mongodb.mongo_find", + "mongodb.mongo_update", + "mongodb.mongo_delete", + "mongodb.mongo_list_collections", + "supabase.supabase_insert" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "kind": "verb_product", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "ranking": [ + "mysql.mysql_query", + "postgres.query", + "sqlite.sqlite_query", + "postgres.explain_query", + "salesforce.salesforce_query", + "mysql.mysql_list_tables", + "github.search_repositories", + "github.search_code", + "github.search_commits", + "github.search_issues" + ], + "hit1": "mysql.mysql_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 639, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "elasticsearch.es_search", + "github.get_file_blame", + "github.list_branches", + "github.list_tags", + "github.create_or_update_file", + "github.create_branch", + "github.push_files", + "github.create_issue" + ], + "hit1": "elasticsearch.es_index_doc", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 411, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 973, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_get_object", + "aws_s3.s3_put_object", + "aws_s3.s3_delete_object", + "okta.okta_get_user", + "asana.asana_get_task", + "github.get_me", + "github.get_teams", + "github.get_team_members", + "github.get_file_contents", + "github.get_commit" + ], + "hit1": "aws_s3.s3_get_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones", + "cloudflare.cf_list_dns", + "github.get_team_members", + "github.list_commits", + "github.list_branches", + "github.list_tags", + "github.list_releases", + "github.list_starred_repositories", + "github.list_repository_collaborators", + "github.list_issues" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 731, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_create_customer", + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "stripe.stripe_create_refund", + "github.create_or_update_file", + "github.create_repository", + "github.create_branch", + "github.create_issue" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 897, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "kind": "verb_product", + "gold": [ + "github.actions_list" + ], + "alts": [], + "ranking": [ + "github.actions_list", + "github.get_team_members", + "github.list_commits", + "github.list_branches", + "github.list_tags", + "github.list_releases", + "github.list_starred_repositories", + "github.list_repository_collaborators", + "github.list_issues", + "github.list_pull_requests" + ], + "hit1": "github.actions_list", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 400, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 795, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "kind": "verb_product", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "ranking": [ + "github.actions_get_job_logs", + "github.actions_get", + "github.get_me", + "github.get_teams", + "github.get_team_members", + "github.get_file_contents", + "github.get_commit", + "github.get_file_blame", + "github.get_tag", + "github.get_latest_release" + ], + "hit1": "github.actions_get_job_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 361, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 805, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "kind": "verb_product", + "gold": [ + "github.create_repository" + ], + "alts": [], + "ranking": [ + "github.create_or_update_file", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.get_file_contents", + "github.list_commits", + "github.get_file_blame", + "github.list_branches", + "github.list_tags", + "github.get_tag" + ], + "hit1": "github.create_or_update_file", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 346, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 585, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1012, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "kind": "verb_product", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "ranking": [ + "git.git_checkout", + "github.list_commits", + "github.create_or_update_file", + "github.create_branch", + "github.push_files", + "github.delete_file", + "git.git_create_branch", + "git.git_branch", + "gitlab.gitlab_create_merge_request", + "github.get_me" + ], + "hit1": "git.git_checkout", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 330, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 545, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 960, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "kind": "identifier", + "gold": [ + "git.git_log" + ], + "alts": [], + "ranking": [ + "git.git_log", + "git.git_status", + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_diff", + "git.git_show", + "github.get_me", + "github.get_teams", + "github.get_team_members", + "github.search_repositories" + ], + "hit1": "git.git_log", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 187, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 613, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "kind": "verb_product", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "ranking": [ + "postgres.list_schemas", + "postgres.list_tables", + "github.get_team_members", + "github.list_commits", + "github.list_branches", + "github.list_tags", + "github.list_releases", + "github.list_starred_repositories", + "github.list_repository_collaborators", + "github.list_issues" + ], + "hit1": "postgres.list_schemas", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 347, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "kind": "verb_product", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "ranking": [ + "memory.open_nodes", + "memory.search_nodes", + "memory.create_entities", + "memory.create_relations", + "memory.add_observations", + "memory.delete_entities", + "memory.delete_relations", + "memory.read_graph", + "postgres.list_tables", + "postgres.list_schemas" + ], + "hit1": "memory.open_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 284, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 505, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "kind": "paraphrase", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "ranking": [ + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "filesystem.get_file_info", + "github.get_file_contents", + "google_drive.gdrive_get_file", + "gitlab.gitlab_get_file", + "figma.figma_get_file_nodes", + "figma.figma_get_images", + "github.get_me" + ], + "hit1": "filesystem.read_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 463, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 750, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "kind": "paraphrase", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "ranking": [ + "filesystem.write_file", + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "filesystem.edit_file", + "github.get_file_contents", + "github.get_file_blame", + "github.create_or_update_file", + "github.create_repository", + "github.delete_file" + ], + "hit1": "filesystem.write_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 927, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "kind": "verb_product", + "gold": [ + "github.search_issues" + ], + "alts": [], + "ranking": [ + "github.search_issues", + "github.search_repositories", + "github.search_code", + "github.search_commits", + "github.list_issues", + "github.search_pull_requests", + "github.search_users", + "github.search_orgs", + "github.get_me", + "github.get_teams" + ], + "hit1": "github.search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 265, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 420, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "kind": "paraphrase", + "gold": [ + "github.get_me" + ], + "alts": [], + "ranking": [ + "github.get_me", + "github.get_teams", + "github.list_notifications", + "github.get_latest_release", + "github.get_repository_tree", + "github.get_team_members", + "github.get_file_contents", + "github.get_commit", + "github.get_file_blame", + "github.get_tag" + ], + "hit1": "github.get_me", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "cost": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.2, + "p95_calls": 1.0, + "success_within_1": 0.98, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 309.12666666666667, + "p95_tokens": 392.8499999999999, + "total_tokens": 46369, + "tokens_per_successful_query": 309.12666666666667, + "total_tokens_successful": 46369 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.1066666666666667, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 464.8933333333333, + "p95_tokens": 600.55, + "total_tokens": 69734, + "tokens_per_successful_query": 464.8933333333333, + "total_tokens_successful": 69734 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.0533333333333332, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 875.54, + "p95_tokens": 1188.2, + "total_tokens": 131331, + "tokens_per_successful_query": 875.54, + "total_tokens_successful": 131331 + } + } +} diff --git a/experiments/mcp-tool-search/results/char_trigram.csv b/experiments/mcp-tool-search/results/char_trigram.csv new file mode 100644 index 00000000..f04186be --- /dev/null +++ b/experiments/mcp-tool-search/results/char_trigram.csv @@ -0,0 +1,151 @@ +id,query,kind,hit1,recall@1,recall@3,recall@5,mrr@10,ndcg@10,hit_rank,k3_calls,k3_tokens,k3_success1,k5_calls,k5_tokens,k5_success1,k10_calls,k10_tokens,k10_success1,ranking +q001,Create a GitHub issue,verb_product,github.create_issue,1.0,1.0,1.0,1.0000,0.9502,1,1,380,1.0,1,557,1.0,1,1021,1.0,github.create_issue gitlab.gitlab_create_issue github.issue_write github.create_gist github.issue_read github.update_issue_comment github.create_branch github.projects_write github.create_hook github.create_repository +q002,Create a Jira ticket,verb_product,jira.jira_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,318,1.0,1,513,1.0,1,1004,1.0,jira.jira_create_issue zendesk.zendesk_create_ticket jira.create_board jira.create_version jira.create_sprint jira.create_filter jira.create_webhook jira.create_attachment jira.jira_create_sprint bitbucket.bitbucket_create_issue +q003,Send a Slack message,verb_product,slack.slack_search_messages,0.0,1.0,1.0,0.5000,0.6309,2,1,249,1.0,1,389,1.0,1,653,1.0,slack.slack_search_messages slack.slack_post_message slack.slack_add_reaction discord.discord_send_message slack.slack_get_channel_history slack.slack_join_channel slack.get_pin slack.slack_leave_channel gmail.gmail_get_email slack.get_file +q004,List open pull requests,resource,bitbucket.bitbucket_list_prs,0.0,1.0,1.0,0.5000,0.6697,2,1,237,1.0,1,477,1.0,1,1034,1.0,bitbucket.bitbucket_list_prs github.list_pull_requests github.search_pull_requests github.create_pull_request github.pull_request_read gitlab.gitlab_list_merge_requests github.merge_pull_request github.request_copilot_review bitbucket.bitbucket_create_pr github.add_comment_to_pending_review +q005,Open a pull request on GitHub,verb_product,github.create_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,327,1.0,1,505,1.0,1,1144,1.0,github.create_pull_request github.request_copilot_review github.search_pull_requests github.pull_request_read github.list_pull_requests github.merge_pull_request github.update_pull_request_branch github.add_comment_to_pending_review github.update_pull_request github.pull_request_review_write +q006,Run a SQL query against postgres,verb_product,postgres.query,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,295,1.0,1,592,1.0,postgres.query mysql.mysql_query sqlite.sqlite_query salesforce.salesforce_query postgres.explain_query postgres.list_schemas postgres.list_tables postgres.describe_table prometheus.prom_query sentry.sentry_list_issues +q007,Read a local file from disk,paraphrase,filesystem.read_text_file,1.0,1.0,1.0,1.0000,0.8597,1,1,223,1.0,1,428,1.0,1,849,1.0,filesystem.read_text_file filesystem.read_file filesystem.read_media_file gitlab.gitlab_get_file github.delete_file supabase.supabase_select github.get_file_contents figma.figma_get_file_nodes figma.figma_get_images slack.slack_upload_file +q008,What files are in this folder,paraphrase,google_drive.gdrive_list_folder,0.0,0.0,0.0,0.0000,0.0000,91,31,8311,0.0,19,8176,0.0,10,8240,0.0,google_drive.gdrive_list_folder slack.list_files slack.get_file filesystem.list_allowed_directories slack.delete_file cloudflare.cf_purge_cache git.git_add google_drive.gdrive_search github.get_file_blame slack.update_file +q009,Remember this fact in the knowledge graph,paraphrase,memory.read_graph,0.0,0.0,1.0,0.2500,0.4171,4,2,335,0.0,1,255,1.0,1,543,1.0,memory.read_graph memory.open_nodes memory.delete_relations memory.create_entities memory.create_relations memory.search_nodes memory.add_observations memory.delete_entities pagerduty.pagerduty_acknowledge_incident github.get_artifact +q010,Search the web with Brave,verb_product,brave_search.brave_web_search,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,403,1.0,1,687,1.0,brave_search.brave_web_search jira.jira_search_issues brave_search.brave_local_search github.update_pull_request_branch github.search_users memory.search_nodes github.search_commits github.search_orgs github.search_pull_requests slack.slack_search_messages +q011,Fetch the contents of a URL,verb_product,fetch.fetch,1.0,1.0,1.0,1.0000,1.0000,1,1,222,1.0,1,338,1.0,1,634,1.0,fetch.fetch filesystem.read_file git.git_add confluence.confluence_search docker.docker_logs github.get_me playwright.browser_navigate figma.figma_get_comments slack.slack_upload_file git.git_status +q012,Show git status,identifier,shopify.shopify_list_orders,0.0,1.0,1.0,0.5000,0.6309,2,1,217,1.0,1,344,1.0,1,724,1.0,shopify.shopify_list_orders git.git_status gitlab.list_snippets gitlab.list_jobs git.git_diff_staged github.list_hooks gitlab.list_wikis gitlab.list_labels github.list_runners github.list_packages +q013,Commit staged changes,paraphrase,git.git_commit,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,330,1.0,1,664,1.0,git.git_commit git.git_reset git.git_diff_staged linear.list_comments notion.list_comments github.list_packages slack.slack_get_channel_history git.git_show git.git_add git.git_log +q014,Create a Linear issue,verb_product,linear.linear_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,293,1.0,1,463,1.0,1,925,1.0,linear.linear_create_issue linear.linear_create_comment linear.linear_assign_issue linear.linear_update_issue linear.linear_get_issue linear.linear_create_project linear.create_user linear.create_label linear.create_cycle linear.create_comment +q015,Make a task in my project tracker Linear,paraphrase,linear.linear_list_projects,0.0,0.0,0.0,0.0000,0.0000,146,49,12254,0.0,30,11994,0.0,15,11634,0.0,linear.linear_list_projects linear.linear_create_project linear.delete_user asana.asana_list_tasks linear.get_user supabase.supabase_list_tables linear.update_user linear.delete_cycle linear.delete_label linear.delete_comment +q016,Post a comment on a GitHub issue,verb_product,github.update_issue_comment,0.0,1.0,1.0,0.5000,0.6309,2,1,307,1.0,1,468,1.0,1,892,1.0,github.update_issue_comment github.add_issue_comment figma.figma_post_comment github.get_discussion_comments jira.jira_add_comment github.list_commits linear.linear_create_comment github.list_issues github.issue_read notion.get_comment +q017,Merge the pull request,ambiguous,github.merge_pull_request,1.0,1.0,1.0,1.0000,0.9502,1,1,299,1.0,1,428,1.0,1,859,1.0,github.merge_pull_request github.update_pull_request_branch gitlab.gitlab_merge_merge_request github.search_pull_requests bitbucket.bitbucket_list_prs gitlab.gitlab_get_merge_request github.pull_request_read gitlab.gitlab_list_merge_requests github.list_pull_requests github.request_copilot_review +q018,Trigger a GitHub Actions workflow,verb_product,github.actions_run_trigger,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,470,1.0,1,847,1.0,github.actions_run_trigger github.actions_get github.actions_list github.actions_get_job_logs github.fork_repository github.star_repository github.unstar_repository github.list_discussions github.get_discussion github.update_gist +q019,Get Dependabot alerts,resource,github.get_dependabot_alert,1.0,1.0,1.0,1.0000,0.8597,1,1,286,1.0,1,463,1.0,1,738,1.0,github.get_dependabot_alert github.list_dependabot_alerts github.get_secret_scanning_alert github.get_code_scanning_alert github.list_secret_scanning_alerts vercel.vercel_get_deployment gitlab.get_job github.get_hook gitlab.get_wiki gitlab.get_label +q020,List Sentry issues,verb_product,sentry.sentry_list_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,363,1.0,1,796,1.0,sentry.sentry_list_issues sentry.sentry_list_events sentry.sentry_list_projects sentry.sentry_get_issue linear.linear_search_issues gitlab.gitlab_list_issues sentry.sentry_update_issue linear.linear_list_issues github.list_issues github.search_issues +q021,Create a Stripe customer,verb_product,stripe.stripe_create_invoice,0.0,1.0,1.0,0.5000,0.6309,2,1,223,1.0,1,322,1.0,1,669,1.0,stripe.stripe_create_invoice stripe.stripe_create_customer stripe.stripe_create_payment_intent stripe.stripe_get_customer stripe.stripe_list_invoices stripe.stripe_create_price stripe.stripe_list_customers stripe.stripe_create_product stripe.stripe_create_refund stripe.stripe_list_subscriptions +q022,Refund a Stripe payment,paraphrase,stripe.stripe_create_refund,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,364,1.0,1,647,1.0,stripe.stripe_create_refund stripe.stripe_get_payment_intent stripe.stripe_create_payment_intent stripe.stripe_get_customer stripe.stripe_create_price stripe.stripe_create_product stripe.stripe_cancel_subscription stripe.stripe_create_invoice stripe.stripe_create_customer stripe.stripe_list_invoices +q023,Send an email with Gmail,verb_product,gmail.gmail_send_email,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,380,1.0,1,671,1.0,gmail.gmail_send_email puppeteer.puppeteer_click gmail.gmail_search_emails twilio.twilio_send_sms gmail.gmail_get_email puppeteer.puppeteer_fill gmail.gmail_list_labels gmail.gmail_trash_email stripe.stripe_list_customers stripe.stripe_create_customer +q024,Search my Gmail inbox,verb_product,gmail.gmail_search_emails,1.0,1.0,1.0,1.0000,1.0000,1,1,226,1.0,1,331,1.0,1,579,1.0,gmail.gmail_search_emails elasticsearch.es_search slack.slack_search_messages github.search_users google_drive.gdrive_search confluence.confluence_search gitlab.gitlab_list_projects hubspot.hubspot_search_contacts gmail.gmail_list_labels github.search_commits +q025,Upload a file to Google Drive,paraphrase,google_drive.gdrive_update_file,0.0,0.0,0.0,0.1667,0.3562,6,2,523,0.0,2,786,0.0,1,764,1.0,google_drive.gdrive_update_file google_drive.gdrive_search google_drive.gdrive_share_file google_drive.gdrive_list_folder slack.slack_upload_file google_drive.gdrive_create_file google_drive.gdrive_get_file slack.delete_file aws_s3.s3_put_object slack.update_file +q026,Create a calendar event,resource,google_calendar.gcal_create_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,484,1.0,1,939,1.0,google_calendar.gcal_create_event google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_list_events linear.create_comment cloudflare.cf_create_dns notion.create_comment jira.create_attachment stripe.stripe_create_refund confluence.confluence_create_page +q027,List Kubernetes pods,verb_product,kubernetes.k8s_list_pods,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,314,1.0,1,698,1.0,kubernetes.k8s_list_pods kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_list_deployments kubernetes.k8s_get_pod kubernetes.k8s_pod_logs kubernetes.k8s_apply kubernetes.k8s_delete kubernetes.k8s_scale_deployment docker.docker_images +q028,Get logs from a k8s pod,verb_product,kubernetes.k8s_pod_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,262,1.0,1,441,1.0,1,858,1.0,kubernetes.k8s_pod_logs docker.docker_logs kubernetes.k8s_get_pod github.actions_get_job_logs github.get_tag figma.figma_get_file_nodes gitlab.gitlab_get_file github.get_file_contents slack.slack_get_channel_history elasticsearch.es_get_doc +q029,Run a docker container,verb_product,docker.docker_stop,0.0,1.0,1.0,0.5000,0.6309,2,1,223,1.0,1,329,1.0,1,640,1.0,docker.docker_stop docker.docker_run docker.docker_exec docker.docker_rm docker.docker_ps docker.docker_logs docker.docker_build kubernetes.k8s_pod_logs docker.docker_images mysql.mysql_query +q030,Show docker container logs,verb_product,docker.docker_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,298,1.0,1,649,1.0,docker.docker_logs docker.docker_stop docker.docker_rm docker.docker_ps docker.docker_exec docker.docker_run kubernetes.k8s_pod_logs git.git_log docker.docker_images docker.docker_build +q031,Navigate the browser to a URL,ambiguous,playwright.browser_navigate,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,282,1.0,1,574,1.0,playwright.browser_navigate puppeteer.puppeteer_navigate playwright.browser_tab_new playwright.browser_press_key playwright.browser_evaluate playwright.browser_wait_for playwright.browser_tab_close playwright.browser_tab_list playwright.browser_click playwright.browser_type +q032,Take a screenshot of the page,ambiguous,playwright.browser_take_screenshot,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,314,1.0,1,646,1.0,playwright.browser_take_screenshot puppeteer.puppeteer_screenshot playwright.browser_snapshot playwright.browser_evaluate playwright.browser_press_key confluence.confluence_create_page playwright.browser_click git.git_status confluence.confluence_get_page notion.notion_update_page +q033,Create a GitLab merge request,verb_product,gitlab.gitlab_create_merge_request,1.0,1.0,1.0,1.0000,1.0000,1,1,297,1.0,1,475,1.0,1,967,1.0,gitlab.gitlab_create_merge_request gitlab.gitlab_merge_merge_request gitlab.gitlab_get_merge_request gitlab.gitlab_list_merge_requests github.merge_pull_request gitlab.gitlab_create_pipeline bitbucket.bitbucket_create_pr gitlab.gitlab_create_issue gitlab.create_release gitlab.create_milestone +q034,Search Notion pages,verb_product,notion.notion_search,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,335,1.0,1,693,1.0,notion.notion_search notion.notion_get_page notion.notion_update_page slack.slack_search_messages notion.notion_get_comments notion.notion_create_page github.search_orgs github.search_repositories gmail.gmail_search_emails memory.search_nodes +q035,Create a Notion page,verb_product,notion.notion_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,230,1.0,1,413,1.0,1,860,1.0,notion.notion_create_page notion.notion_update_page notion.notion_get_page notion.notion_create_database notion.create_comment confluence.confluence_create_page github.create_package notion.notion_get_comments notion.create_users_token notion.create_data_source +q036,Query a Notion database,verb_product,notion.notion_query_database,1.0,1.0,1.0,1.0000,1.0000,1,1,261,1.0,1,404,1.0,1,730,1.0,notion.notion_query_database notion.notion_search notion.notion_create_database notion.notion_create_page notion.delete_data_source mongodb.mongo_insert notion.get_data_source mongodb.mongo_find mongodb.mongo_list_collections notion.update_data_source +q037,Create a Zendesk ticket,verb_product,zendesk.zendesk_get_ticket,0.0,1.0,1.0,0.5000,0.6309,2,1,257,1.0,1,368,1.0,1,892,1.0,zendesk.zendesk_get_ticket zendesk.zendesk_create_ticket zendesk.zendesk_update_ticket zendesk.zendesk_list_tickets zendesk.zendesk_search jira.jira_create_issue bitbucket.bitbucket_create_issue clickup.clickup_create_task bitbucket.bitbucket_create_pr gitlab.create_snippet +q038,Create a HubSpot contact,verb_product,hubspot.hubspot_create_contact,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,364,1.0,1,797,1.0,hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_list_contacts hubspot.hubspot_search_contacts hubspot.hubspot_create_deal vercel.vercel_create_project github.create_artifact confluence.confluence_create_page cloudflare.cf_create_dns gmail.gmail_create_draft +q039,Create a Salesforce lead,verb_product,salesforce.salesforce_create_lead,1.0,1.0,1.0,1.0000,1.0000,1,1,252,1.0,1,416,1.0,1,710,1.0,salesforce.salesforce_create_lead salesforce.salesforce_convert_lead salesforce.salesforce_create_record salesforce.salesforce_update_record salesforce.salesforce_delete_record salesforce.salesforce_query salesforce.salesforce_get_record stripe.stripe_create_invoice salesforce.salesforce_list_opportunities vercel.vercel_create_project +q040,Run a SOQL query,identifier,salesforce.salesforce_query,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,294,1.0,1,592,1.0,salesforce.salesforce_query mysql.mysql_query sqlite.sqlite_query postgres.query postgres.explain_query notion.notion_query_database github.search_users prometheus.prom_query confluence.confluence_search memory.search_nodes +q041,List Shopify orders,verb_product,shopify.shopify_list_orders,1.0,1.0,1.0,1.0000,1.0000,1,1,193,1.0,1,333,1.0,1,628,1.0,shopify.shopify_list_orders shopify.shopify_list_customers shopify.shopify_get_order shopify.shopify_create_order shopify.shopify_list_products shopify.shopify_get_product okta.okta_list_users discord.discord_list_members stripe.stripe_list_customers shopify.shopify_update_product +q042,Create a PagerDuty incident,verb_product,pagerduty.pagerduty_create_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,403,1.0,1,853,1.0,pagerduty.pagerduty_create_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_list_incidents notion.notion_create_page confluence.confluence_create_page github.create_package stripe.stripe_create_refund google_drive.gdrive_create_file +q043,Who is on call in PagerDuty,paraphrase,pagerduty.pagerduty_list_oncalls,1.0,1.0,1.0,1.0000,1.0000,1,1,215,1.0,1,363,1.0,1,765,1.0,pagerduty.pagerduty_list_oncalls pagerduty.pagerduty_get_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_create_incident notion.notion_get_comments notion.notion_create_page notion.notion_get_page github.list_commits +q044,Query Datadog metrics,verb_product,datadog.datadog_query_metrics,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,458,1.0,1,780,1.0,datadog.datadog_query_metrics datadog.datadog_list_monitors datadog.datadog_create_monitor datadog.datadog_list_logs datadog.datadog_get_monitor notion.notion_query_database notion.notion_search prometheus.prom_query prometheus.prom_list_metrics elasticsearch.es_search +q045,Search Datadog logs,verb_product,datadog.datadog_list_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,248,1.0,1,363,1.0,1,646,1.0,datadog.datadog_list_logs notion.notion_search datadog.datadog_get_monitor datadog.datadog_list_monitors github.search_orgs grafana.grafana_search_dashboards slack.slack_search_messages github.search_users confluence.confluence_search gitlab.gitlab_list_projects +q046,List S3 buckets,verb_product,aws_s3.s3_list_buckets,1.0,1.0,1.0,1.0000,1.0000,1,1,187,1.0,1,316,1.0,1,616,1.0,aws_s3.s3_list_buckets aws_s3.s3_list_objects zendesk.zendesk_list_tickets bitbucket.bitbucket_list_repos bitbucket.bitbucket_list_prs aws_s3.s3_delete_object aws_s3.s3_get_object linear.linear_list_projects gitlab.list_snippets docker.docker_images +q047,Upload an object to S3,paraphrase,aws_s3.s3_put_object,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,426,1.0,1,830,1.0,aws_s3.s3_put_object aws_s3.s3_get_object aws_s3.s3_delete_object gmail.gmail_send_email slack.slack_upload_file aws_s3.s3_list_objects gmail.gmail_create_draft salesforce.salesforce_update_record twilio.twilio_send_sms salesforce.salesforce_delete_record +q048,Purge Cloudflare cache,verb_product,cloudflare.cf_purge_cache,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,335,1.0,1,548,1.0,cloudflare.cf_purge_cache cloudflare.cf_list_zones cloudflare.cf_list_dns cloudflare.cf_create_dns playwright.browser_tab_close playwright.browser_snapshot postgres.list_tables memory.read_graph postgres.list_schemas puppeteer.puppeteer_click +q049,Create a Cloudflare DNS record,verb_product,cloudflare.cf_create_dns,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,356,1.0,1,781,1.0,cloudflare.cf_create_dns cloudflare.cf_list_dns salesforce.salesforce_create_record cloudflare.cf_list_zones cloudflare.cf_purge_cache clickup.clickup_create_task trello.trello_create_board jira.create_board trello.trello_create_card confluence.confluence_create_page +q050,List Vercel deployments,verb_product,vercel.vercel_list_deployments,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,306,1.0,1,710,1.0,vercel.vercel_list_deployments vercel.vercel_get_deployment vercel.vercel_list_projects kubernetes.k8s_list_deployments linear.list_comments linear.list_attachments kubernetes.k8s_scale_deployment notion.list_comments jira.list_attachments github.list_environments +q051,Insert a row into Supabase,verb_product,supabase.supabase_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,245,1.0,1,352,1.0,1,681,1.0,supabase.supabase_insert mongodb.mongo_insert supabase.supabase_update supabase.supabase_select supabase.supabase_list_tables supabase.supabase_invoke_fn gitlab.get_release slack.get_canvase gitlab.delete_release notion.notion_create_database +q052,Export images from Figma,paraphrase,figma.figma_get_images,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,349,1.0,1,692,1.0,figma.figma_get_images figma.figma_get_file_nodes docker.docker_images filesystem.read_media_file slack.slack_get_channel_history figma.figma_get_file supabase.supabase_select docker.docker_run figma.figma_post_comment figma.figma_get_comments +q053,Send an SMS with Twilio,verb_product,twilio.twilio_send_sms,1.0,1.0,1.0,1.0000,1.0000,1,1,215,1.0,1,377,1.0,1,740,1.0,twilio.twilio_send_sms twilio.twilio_list_messages puppeteer.puppeteer_click gmail.gmail_send_email puppeteer.puppeteer_fill jira.jira_search_issues filesystem.list_directory_with_sizes twilio.twilio_make_call puppeteer.puppeteer_screenshot discord.discord_send_message +q054,Create an Okta user,verb_product,okta.okta_create_user,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,329,1.0,1,719,1.0,okta.okta_create_user okta.okta_deactivate_user okta.okta_get_user linear.create_user okta.okta_list_users slack.create_usergroup notion.create_users_token asana.asana_create_project linear.update_user stripe.stripe_create_invoice +q055,Instant query Prometheus,verb_product,prometheus.prom_query,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,345,1.0,1,660,1.0,prometheus.prom_query prometheus.prom_query_range prometheus.prom_list_metrics shopify.shopify_list_products mysql.mysql_query sqlite.sqlite_query salesforce.salesforce_query sentry.sentry_list_issues postgres.query postgres.explain_query +q056,Search Grafana dashboards,verb_product,grafana.grafana_search_dashboards,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,287,1.0,1,666,1.0,grafana.grafana_search_dashboards grafana.grafana_get_dashboard github.search_users github.search_orgs github.search_pull_requests slack.slack_search_messages github.search_issues confluence.confluence_search datadog.datadog_list_logs gmail.gmail_search_emails +q057,Trigger a CircleCI pipeline,verb_product,circleci.circleci_trigger_pipeline,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,325,1.0,1,755,1.0,circleci.circleci_trigger_pipeline circleci.circleci_get_pipeline circleci.circleci_list_pipelines gitlab.gitlab_create_pipeline gitlab.get_pipeline gitlab.delete_pipeline gitlab.update_pipeline hubspot.hubspot_create_deal gitlab.create_pipeline github.actions_run_trigger +q058,Run terraform plan,identifier,terraform.tf_plan,1.0,1.0,1.0,1.0000,1.0000,1,1,172,1.0,1,281,1.0,1,593,1.0,terraform.tf_plan terraform.tf_apply terraform.tf_state_list playwright.browser_fill_form salesforce.salesforce_query playwright.browser_wait_for playwright.browser_click playwright.browser_type mysql.mysql_query sqlite.sqlite_query +q059,Create an Asana task,verb_product,asana.asana_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,429,1.0,1,822,1.0,asana.asana_create_task asana.asana_get_task asana.asana_update_task asana.asana_create_project clickup.clickup_create_task grafana.grafana_create_annotation okta.okta_create_user asana.asana_list_tasks trello.trello_create_board github.create_issue +q060,Create a Trello card,verb_product,trello.trello_create_card,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,411,1.0,1,745,1.0,trello.trello_create_card trello.trello_create_board trello.trello_add_comment trello.trello_list_cards jira.create_board vercel.vercel_create_project salesforce.salesforce_create_record confluence.confluence_create_page stripe.stripe_create_invoice stripe.stripe_create_customer +q061,Create a ClickUp task,verb_product,clickup.clickup_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,415,1.0,1,802,1.0,clickup.clickup_create_task clickup.clickup_get_task clickup.clickup_update_task clickup.clickup_list_tasks asana.asana_create_task confluence.confluence_create_page cloudflare.cf_create_dns trello.trello_create_board vercel.vercel_create_project github.create_gist +q062,Create a Confluence page,verb_product,confluence.confluence_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,418,1.0,1,870,1.0,confluence.confluence_create_page confluence.confluence_get_page confluence.confluence_update_page confluence.confluence_search pagerduty.pagerduty_create_incident cloudflare.cf_create_dns stripe.stripe_create_price notion.notion_create_page github.create_package stripe.stripe_create_payment_intent +q063,Send a Discord message,verb_product,discord.discord_send_message,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,327,1.0,1,701,1.0,discord.discord_send_message discord.discord_add_reaction discord.discord_get_channel discord.discord_list_members gmail.gmail_get_email slack.slack_post_message gmail.gmail_trash_email git.git_commit slack.slack_search_messages datadog.datadog_create_monitor +q064,Search Elasticsearch,verb_product,elasticsearch.es_search,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,358,1.0,1,630,1.0,elasticsearch.es_search elasticsearch.es_index_doc elasticsearch.es_get_doc slack.slack_search_messages github.search_users zendesk.zendesk_search confluence.confluence_search gitlab.gitlab_list_projects hubspot.hubspot_search_contacts github.search_commits +q065,Get a Redis key,verb_product,redis.redis_get,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,283,1.0,1,582,1.0,redis.redis_get redis.redis_set redis.redis_delete redis.redis_keys gitlab.get_release jira.jira_get_issue figma.figma_get_file github.get_runner slack.get_reminder github.get_discussion +q066,Find MongoDB documents,verb_product,mongodb.mongo_find,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,417,1.0,1,742,1.0,mongodb.mongo_find mongodb.mongo_update mongodb.mongo_delete mongodb.mongo_insert elasticsearch.es_index_doc elasticsearch.es_get_doc mongodb.mongo_list_collections figma.figma_get_comments github.get_discussion_comments filesystem.read_file +q067,List SQLite tables,verb_product,sqlite.sqlite_list_tables,1.0,1.0,1.0,1.0000,1.0000,1,1,151,1.0,1,240,1.0,1,487,1.0,sqlite.sqlite_list_tables sqlite.sqlite_describe_table mysql.mysql_list_tables supabase.supabase_list_tables postgres.list_tables sqlite.sqlite_query postgres.list_schemas salesforce.salesforce_list_opportunities linear.list_cycles docker.docker_images +q068,Describe a postgres table,verb_product,postgres.describe_table,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,274,1.0,1,697,1.0,postgres.describe_table sqlite.sqlite_describe_table postgres.explain_query postgres.list_tables postgres.list_schemas postgres.query slack.create_pin github.create_package gitlab.create_pipeline github.update_gist +q069,github_create_issue,identifier,gitlab.gitlab_create_issue,0.0,1.0,1.0,0.5000,0.5995,2,1,347,1.0,1,516,1.0,1,1032,1.0,gitlab.gitlab_create_issue github.create_issue github.update_issue_comment bitbucket.bitbucket_create_issue github.create_gist github.search_issues github.issue_read github.issue_write github.create_branch github.create_hook +q070,createIssue,identifier,bitbucket.bitbucket_create_issue,0.0,1.0,1.0,0.3333,0.5584,3,1,332,1.0,1,579,1.0,1,982,1.0,bitbucket.bitbucket_create_issue gitlab.gitlab_create_issue linear.linear_create_issue github.create_issue jira.jira_create_issue linear.linear_create_comment github.issue_write confluence.confluence_create_page trello.trello_create_board okta.okta_create_user +q071,list-directory,identifier,filesystem.list_directory,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,333,1.0,1,586,1.0,filesystem.list_directory filesystem.list_directory_with_sizes filesystem.create_directory filesystem.list_allowed_directories github.get_file_contents filesystem.directory_tree linear.linear_list_projects filesystem.get_file_info datadog.datadog_list_monitors linear.linear_list_teams +q072,sequential thinking next thought,paraphrase,sequential_thinking.sequentialthinking,1.0,1.0,1.0,1.0000,1.0000,1,1,278,1.0,1,434,1.0,1,862,1.0,sequential_thinking.sequentialthinking filesystem.edit_file confluence.confluence_search playwright.browser_wait_for brave_search.brave_web_search playwright.browser_type brave_search.brave_local_search github.add_comment_to_pending_review github.list_notifications memory.add_observations +q073,What time is it in Tokyo,paraphrase,time.get_current_time,1.0,1.0,1.0,1.0000,0.9502,1,1,217,1.0,1,333,1.0,1,804,1.0,time.get_current_time playwright.browser_wait_for time.convert_time notion.list_users_tokens sqlite.sqlite_list_tables notion.create_users_token gitlab.list_wikis github.list_tags grafana.grafana_create_annotation linear.linear_create_issue +q074,Convert time between timezones,verb_product,time.convert_time,1.0,1.0,1.0,1.0000,1.0000,1,1,213,1.0,1,319,1.0,1,726,1.0,time.convert_time time.get_current_time git.git_diff salesforce.salesforce_convert_lead memory.create_relations cloudflare.cf_list_zones cloudflare.cf_create_dns confluence.confluence_update_page grafana.grafana_create_annotation confluence.confluence_create_page +q075,Star a GitHub repo,verb_product,github.star_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,226,1.0,1,416,1.0,1,778,1.0,github.star_repository github.unstar_repository github.delete_repository github.list_issues github.list_pull_requests github.list_tags github.list_labels github.get_label github.get_tag github.list_starred_repositories +q076,Fork a repository,resource,github.fork_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,378,1.0,1,764,1.0,github.fork_repository github.list_releases github.star_repository github.delete_repository github.unstar_repository github.list_dependabot_alerts github.list_discussions github.get_file_blame github.list_tags github.list_labels +q077,Get the latest GitHub release,verb_product,github.get_latest_release,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,349,1.0,1,728,1.0,github.get_latest_release github.get_release_by_tag github.list_releases gitlab.get_release github.get_teams github.get_gist github.get_repository_tree github.get_artifact github.get_label github.get_tag +q078,Create a gist,resource,github.create_gist,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,407,1.0,1,817,1.0,github.create_gist trello.trello_create_card gitlab.gitlab_create_pipeline clickup.clickup_create_task trello.trello_create_board gmail.gmail_create_draft github.create_branch gitlab.gitlab_create_issue vercel.vercel_create_project github.update_gist +q079,Search GitHub code,verb_product,github.search_users,0.0,1.0,1.0,0.5000,0.6309,2,1,191,1.0,1,296,1.0,1,719,1.0,github.search_users github.search_code github.search_commits github.search_orgs github.search_pull_requests github.search_issues github.search_repositories github.list_code_scanning_alerts github.get_code_scanning_alert gitlab.gitlab_list_projects +q080,List GitHub notifications,verb_product,github.list_notifications,1.0,1.0,1.0,1.0000,1.0000,1,1,176,1.0,1,295,1.0,1,606,1.0,github.list_notifications github.mark_all_notifications_read github.get_notification_details github.dismiss_notification github.actions_list github.list_discussions github.list_gists github.projects_list github.list_tags github.get_gist +q081,Assign Copilot to an issue,verb_product,github.assign_copilot_to_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,467,1.0,1,883,1.0,github.assign_copilot_to_issue linear.linear_assign_issue jira.jira_assign_issue github.create_issue jira.jira_add_comment linear.linear_create_comment github.add_issue_comment jira.jira_transition_issue linear.linear_list_issues jira.jira_add_worklog +q082,Get secret scanning alerts,resource,github.list_secret_scanning_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,483,1.0,1,777,1.0,github.list_secret_scanning_alerts github.get_secret_scanning_alert github.get_code_scanning_alert github.list_code_scanning_alerts github.get_dependabot_alert github.list_dependabot_alerts docker.docker_ps git.git_add gitlab.get_job github.get_hook +q083,Create or update a file on GitHub,verb_product,github.projects_write,0.0,1.0,1.0,0.5000,0.4796,2,1,318,1.0,1,469,1.0,1,881,1.0,github.projects_write github.create_or_update_file github.update_runner github.update_gist github.update_milestone github.update_hook github.create_gist github.update_package github.update_artifact github.issue_write +q084,List collaborators on a repo,resource,github.list_repository_collaborators,1.0,1.0,1.0,1.0000,1.0000,1,1,271,1.0,1,413,1.0,1,813,1.0,github.list_repository_collaborators github.list_commits github.list_labels github.list_discussions github.list_tags github.list_branches github.list_releases github.list_issues github.list_dependabot_alerts trello.trello_list_cards +q085,Reply in a Slack thread,verb_product,slack.slack_get_thread_replies,0.0,1.0,1.0,0.5000,0.6309,2,1,242,1.0,1,347,1.0,1,621,1.0,slack.slack_get_thread_replies slack.slack_reply_to_thread slack.get_reminder slack.get_pin slack.slack_join_channel slack.delete_reminder slack.get_file slack.delete_pin slack.get_canvase slack.get_bookmark +q086,Add an emoji reaction on Slack,verb_product,slack.slack_add_reaction,1.0,1.0,1.0,1.0000,1.0000,1,1,302,1.0,1,496,1.0,1,919,1.0,slack.slack_add_reaction discord.discord_add_reaction slack.create_reminder slack.create_pin slack.create_file slack.create_bookmark slack.create_canvase slack.create_usergroup slack.get_reminder slack.delete_reminder +q087,Search Slack messages,verb_product,slack.slack_search_messages,1.0,1.0,1.0,1.0000,1.0000,1,1,232,1.0,1,392,1.0,1,698,1.0,slack.slack_search_messages gmail.gmail_search_emails slack.slack_get_channel_history slack.slack_post_message slack.slack_add_reaction notion.notion_search twilio.twilio_list_messages brave_search.brave_local_search confluence.confluence_search google_drive.gdrive_search +q088,Create a Slack channel,verb_product,slack.slack_create_channel,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,339,1.0,1,801,1.0,slack.slack_create_channel slack.slack_leave_channel slack.slack_join_channel slack.slack_set_channel_topic slack.slack_post_message slack.create_pin slack.slack_get_thread_replies slack.create_file slack.slack_reply_to_thread slack.create_bookmark +q089,Transition a Jira issue,verb_product,jira.jira_transition_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,391,1.0,1,740,1.0,jira.jira_transition_issue jira.jira_get_issue jira.jira_assign_issue jira.jira_create_issue jira.jira_add_comment jira.jira_update_issue jira.jira_add_worklog jira.jira_search_issues sentry.sentry_get_issue jira.get_version +q090,Search Jira with JQL,identifier,jira.jira_search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,301,1.0,1,627,1.0,jira.jira_search_issues github.search_users github.search_commits github.search_orgs github.search_pull_requests slack.slack_search_messages github.search_issues confluence.confluence_search gitlab.gitlab_list_projects elasticsearch.es_search +q091,Add a worklog to Jira,verb_product,jira.jira_add_worklog,1.0,1.0,1.0,1.0000,1.0000,1,1,233,1.0,1,353,1.0,1,645,1.0,jira.jira_add_worklog jira.jira_add_comment jira.get_webhook jira.delete_webhook linear.linear_create_comment jira.jira_transition_issue jira.update_webhook jira.get_board jira.get_sprint jira.get_filter +q092,List Linear teams,verb_product,linear.linear_list_teams,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,330,1.0,1,686,1.0,linear.linear_list_teams linear.linear_list_projects linear.linear_list_issues linear.list_labels linear.list_cycles linear.list_users linear.list_comments linear.list_attachments linear.linear_create_project linear.get_user +q093,Comment on a Linear issue,verb_product,linear.linear_create_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,320,1.0,1,831,1.0,linear.linear_create_comment linear.get_comment linear.delete_comment linear.linear_get_issue linear.linear_assign_issue linear.update_comment github.update_issue_comment linear.create_comment linear.linear_update_issue linear.linear_create_issue +q094,Move a local file,paraphrase,filesystem.read_text_file,0.0,0.0,1.0,0.2000,0.3869,5,2,422,0.0,1,335,1.0,1,689,1.0,filesystem.read_text_file filesystem.read_file filesystem.read_media_file slack.delete_file filesystem.move_file slack.update_file google_drive.gdrive_share_file gmail.gmail_trash_email google_drive.gdrive_update_file figma.figma_get_file +q095,Recursively search files by pattern,paraphrase,filesystem.search_files,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,324,1.0,1,658,1.0,filesystem.search_files google_drive.gdrive_search filesystem.directory_tree linear.linear_search_issues notion.notion_search redis.redis_keys github.search_repositories github.search_issues memory.search_nodes github.search_pull_requests +q096,Edit a file in place with old and new string,paraphrase,filesystem.edit_file,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,350,1.0,1,733,1.0,filesystem.edit_file slack.get_file slack.delete_file filesystem.list_directory_with_sizes slack.update_file github.get_file_blame filesystem.list_directory filesystem.write_file slack.create_file git.git_add +q097,Read the whole knowledge graph,paraphrase,memory.read_graph,1.0,1.0,1.0,1.0000,1.0000,1,1,152,1.0,1,253,1.0,1,574,1.0,memory.read_graph memory.delete_relations memory.create_entities memory.create_relations memory.open_nodes memory.search_nodes memory.delete_entities memory.add_observations pagerduty.pagerduty_acknowledge_incident filesystem.read_text_file +q098,Search memory nodes,verb_product,memory.search_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,195,1.0,1,319,1.0,1,621,1.0,memory.search_nodes slack.slack_search_messages memory.open_nodes notion.notion_search linear.linear_search_issues github.search_users github.search_issues zendesk.zendesk_search confluence.confluence_search google_drive.gdrive_search +q099,Show unstaged git diff,verb_product,git.git_diff_unstaged,1.0,1.0,1.0,1.0000,1.0000,1,1,174,1.0,1,299,1.0,1,614,1.0,git.git_diff_unstaged git.git_diff_staged git.git_reset git.git_diff git.git_show git.git_log git.git_status git.git_commit github.unstar_repository shopify.shopify_list_orders +q100,Stage files with git add,identifier,git.git_add,1.0,1.0,1.0,1.0000,1.0000,1,1,209,1.0,1,349,1.0,1,745,1.0,git.git_add slack.list_files git.git_reset filesystem.list_directory_with_sizes github.list_milestones git.git_commit salesforce.salesforce_list_opportunities github.push_files github.delete_file git.git_diff_staged +q101,Create a Shopify product,verb_product,shopify.shopify_create_product,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,377,1.0,1,724,1.0,shopify.shopify_create_product shopify.shopify_update_product shopify.shopify_get_product stripe.stripe_create_product shopify.shopify_create_order shopify.shopify_list_products stripe.stripe_create_price vercel.vercel_create_project shopify.shopify_get_order linear.linear_create_project +q102,Cancel a Stripe subscription,verb_product,stripe.stripe_cancel_subscription,1.0,1.0,1.0,1.0000,1.0000,1,1,203,1.0,1,361,1.0,1,724,1.0,stripe.stripe_cancel_subscription stripe.stripe_list_subscriptions stripe.stripe_create_product slack.create_canvase stripe.stripe_get_customer stripe.stripe_create_invoice stripe.stripe_create_customer clickup.clickup_create_task gitlab.create_label stripe.stripe_get_payment_intent +q103,Create a payment intent,resource,stripe.stripe_create_refund,0.0,1.0,1.0,0.5000,0.6309,2,1,234,1.0,1,439,1.0,1,931,1.0,stripe.stripe_create_refund stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent notion.create_comment google_drive.gdrive_create_file pagerduty.pagerduty_create_incident linear.create_comment jira.create_attachment linear.create_attachment cloudflare.cf_create_dns +q104,Acknowledge a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,405,1.0,1,665,1.0,pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_list_incidents memory.read_graph github.delete_package memory.create_entities pagerduty.pagerduty_list_oncalls github.update_package +q105,Create a Datadog monitor,verb_product,datadog.datadog_create_monitor,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,448,1.0,1,805,1.0,datadog.datadog_create_monitor datadog.datadog_get_monitor datadog.datadog_list_monitors notion.create_data_source notion.notion_create_database notion.notion_create_page filesystem.create_directory trello.trello_create_board github.create_branch vercel.vercel_create_project +q106,Resolve a Sentry issue,paraphrase,sentry.sentry_get_issue,0.0,1.0,1.0,0.5000,0.6309,2,1,251,1.0,1,412,1.0,1,848,1.0,sentry.sentry_get_issue sentry.sentry_update_issue sentry.sentry_list_events pagerduty.pagerduty_resolve_incident sentry.sentry_list_issues linear.linear_get_issue bitbucket.bitbucket_create_issue github.search_issues jira.jira_assign_issue github.issue_write +q107,Scale a Kubernetes deployment,verb_product,kubernetes.k8s_scale_deployment,1.0,1.0,1.0,1.0000,1.0000,1,1,262,1.0,1,400,1.0,1,726,1.0,kubernetes.k8s_scale_deployment kubernetes.k8s_list_deployments kubernetes.k8s_delete vercel.vercel_get_deployment kubernetes.k8s_get_pod kubernetes.k8s_apply kubernetes.k8s_get_events kubernetes.k8s_list_pods kubernetes.k8s_pod_logs kubernetes.k8s_list_services +q108,Apply a Kubernetes manifest,verb_product,kubernetes.k8s_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,222,1.0,1,362,1.0,1,713,1.0,kubernetes.k8s_apply kubernetes.k8s_get_pod kubernetes.k8s_list_pods kubernetes.k8s_delete kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_list_deployments kubernetes.k8s_scale_deployment kubernetes.k8s_pod_logs redis.redis_keys +q109,Stop a docker container,verb_product,docker.docker_stop,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,327,1.0,1,655,1.0,docker.docker_stop docker.docker_rm docker.docker_logs docker.docker_run docker.docker_ps docker.docker_exec docker.docker_build kubernetes.k8s_pod_logs docker.docker_images stripe.stripe_create_customer +q110,Build a docker image,verb_product,docker.docker_build,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,317,1.0,1,598,1.0,docker.docker_build docker.docker_images docker.docker_run docker.docker_stop docker.docker_rm docker.docker_logs docker.docker_exec docker.docker_ps filesystem.read_media_file jira.delete_filter +q111,Click an element in Playwright,verb_product,playwright.browser_click,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,339,1.0,1,566,1.0,playwright.browser_click playwright.browser_type puppeteer.puppeteer_click playwright.browser_take_screenshot playwright.browser_tab_close playwright.browser_snapshot playwright.browser_tab_list playwright.browser_tab_new playwright.browser_navigate playwright.browser_press_key +q112,Type text into a browser input,paraphrase,playwright.browser_type,1.0,1.0,1.0,1.0000,0.7602,1,1,233,1.0,1,333,1.0,1,629,1.0,playwright.browser_type supabase.supabase_insert playwright.browser_wait_for playwright.browser_navigate playwright.browser_tab_close playwright.browser_tab_new slack.slack_post_message playwright.browser_tab_list playwright.browser_press_key slack.slack_reply_to_thread +q113,List GitLab projects,verb_product,gitlab.gitlab_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,209,1.0,1,328,1.0,1,609,1.0,gitlab.gitlab_list_projects gitlab.gitlab_list_issues github.projects_list gitlab.gitlab_list_merge_requests linear.linear_list_projects jira.jira_list_projects vercel.vercel_list_projects gitlab.gitlab_get_merge_request gitlab.gitlab_merge_merge_request sentry.sentry_list_projects +q114,Create a Bitbucket pull request,verb_product,bitbucket.bitbucket_create_pr,1.0,1.0,1.0,1.0000,1.0000,1,1,313,1.0,1,489,1.0,1,1072,1.0,bitbucket.bitbucket_create_pr bitbucket.bitbucket_list_prs bitbucket.bitbucket_create_issue zendesk.zendesk_create_ticket github.list_pull_requests github.pull_request_review_write github.create_pull_request github.pull_request_read github.update_pull_request_branch github.merge_pull_request +q115,Create a Gmail draft,verb_product,gmail.gmail_create_draft,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,431,1.0,1,780,1.0,gmail.gmail_create_draft stripe.stripe_create_customer shopify.shopify_create_order hubspot.hubspot_create_contact salesforce.salesforce_create_lead grafana.grafana_create_annotation trello.trello_create_board vercel.vercel_create_project github.create_gist gitlab.gitlab_create_pipeline +q116,Share a Google Drive file,verb_product,google_drive.gdrive_share_file,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,417,1.0,1,753,1.0,google_drive.gdrive_share_file google_drive.gdrive_update_file google_drive.gdrive_list_folder google_drive.gdrive_search google_drive.gdrive_create_file google_drive.gdrive_get_file slack.delete_file google_calendar.gcal_delete_event google_calendar.gcal_update_event slack.update_file +q117,List Google Calendar events,verb_product,google_calendar.gcal_list_events,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,427,1.0,1,757,1.0,google_calendar.gcal_list_events google_calendar.gcal_list_calendars google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event kubernetes.k8s_get_events google_drive.gdrive_list_folder sentry.sentry_list_events linear.list_comments linear.list_attachments +q118,Delete a calendar event,resource,google_calendar.gcal_delete_event,1.0,1.0,1.0,1.0000,1.0000,1,1,231,1.0,1,404,1.0,1,704,1.0,google_calendar.gcal_delete_event google_calendar.gcal_update_event linear.delete_comment google_calendar.gcal_create_event notion.delete_comment linear.delete_attachment linear.delete_cycle google_calendar.gcal_list_events slack.delete_canvase jira.delete_attachment +q119,List Vercel projects,verb_product,vercel.vercel_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,163,1.0,1,255,1.0,1,530,1.0,vercel.vercel_list_projects vercel.vercel_list_deployments linear.linear_list_projects jira.jira_list_projects vercel.vercel_create_project gitlab.gitlab_list_projects sentry.sentry_list_projects github.projects_list asana.asana_list_tasks supabase.supabase_list_tables +q120,Select rows from Supabase,verb_product,supabase.supabase_select,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,351,1.0,1,670,1.0,supabase.supabase_select supabase.supabase_insert supabase.supabase_update mongodb.mongo_delete supabase.supabase_list_tables supabase.supabase_invoke_fn aws_s3.s3_get_object mongodb.mongo_list_collections slack.slack_get_channel_history memory.delete_relations +q121,Post a Figma comment,verb_product,figma.figma_post_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,201,1.0,1,319,1.0,1,612,1.0,figma.figma_post_comment figma.figma_get_comments linear.get_comment notion.get_comment linear.update_comment notion.update_comment linear.delete_comment notion.delete_comment figma.figma_get_file jira.jira_add_comment +q122,List Okta users,verb_product,okta.okta_list_users,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,303,1.0,1,607,1.0,okta.okta_list_users linear.list_users slack.slack_get_users notion.notion_list_users okta.okta_get_user okta.okta_create_user slack.slack_list_users github.list_gists okta.okta_deactivate_user notion.list_users_tokens +q123,Range query Prometheus,verb_product,prometheus.prom_query_range,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,347,1.0,1,776,1.0,prometheus.prom_query_range prometheus.prom_query prometheus.prom_list_metrics shopify.shopify_list_products postgres.explain_query datadog.datadog_query_metrics github.search_code datadog.datadog_create_monitor sentry.sentry_list_issues github.search_users +q124,Create a Grafana annotation,verb_product,grafana.grafana_create_annotation,1.0,1.0,1.0,1.0000,1.0000,1,1,275,1.0,1,432,1.0,1,893,1.0,grafana.grafana_create_annotation github.create_gist clickup.clickup_create_task asana.asana_create_project notion.notion_create_database asana.asana_create_task notion.create_comment stripe.stripe_create_product gitlab.gitlab_create_issue github.create_artifact +q125,Apply terraform,identifier,terraform.tf_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,171,1.0,1,327,1.0,1,644,1.0,terraform.tf_apply terraform.tf_plan terraform.tf_state_list kubernetes.k8s_apply slack.slack_reply_to_thread playwright.browser_fill_form filesystem.search_files linear.linear_list_issues docker.docker_rm linear.linear_search_issues +q126,List Asana tasks,verb_product,asana.asana_list_tasks,1.0,1.0,1.0,1.0000,1.0000,1,1,192,1.0,1,386,1.0,1,631,1.0,asana.asana_list_tasks asana.asana_get_task clickup.clickup_list_tasks asana.asana_update_task asana.asana_create_task linear.linear_list_teams aws_s3.s3_list_buckets linear.linear_list_projects mysql.mysql_list_tables linear.list_attachments +q127,Comment on a Trello card,verb_product,trello.trello_add_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,390,1.0,1,712,1.0,trello.trello_add_comment trello.trello_list_cards trello.trello_create_card figma.figma_post_comment trello.trello_create_board github.update_issue_comment figma.figma_get_comments notion.get_comment notion.delete_comment notion.notion_get_comments +q128,Update a ClickUp task,verb_product,clickup.clickup_update_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,399,1.0,1,760,1.0,clickup.clickup_update_task clickup.clickup_get_task clickup.clickup_create_task clickup.clickup_list_tasks asana.asana_update_task linear.update_cycle zendesk.zendesk_update_ticket slack.update_canvase linear.update_comment notion.update_comment +q129,Search Confluence,verb_product,confluence.confluence_search,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,369,1.0,1,660,1.0,confluence.confluence_search confluence.confluence_get_page confluence.confluence_update_page confluence.confluence_create_page hubspot.hubspot_search_contacts github.search_commits slack.slack_search_messages github.search_users github.search_code gitlab.gitlab_list_projects +q130,Create a HubSpot deal,verb_product,hubspot.hubspot_create_deal,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,428,1.0,1,805,1.0,hubspot.hubspot_create_deal hubspot.hubspot_create_contact hubspot.hubspot_get_contact github.create_hook github.create_gist clickup.clickup_create_task trello.trello_create_board trello.trello_create_card vercel.vercel_create_project github.create_repository +q131,List Zendesk tickets,verb_product,zendesk.zendesk_list_tickets,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,367,1.0,1,593,1.0,zendesk.zendesk_list_tickets zendesk.zendesk_get_ticket zendesk.zendesk_search zendesk.zendesk_update_ticket zendesk.zendesk_create_ticket aws_s3.s3_list_buckets linear.linear_list_projects aws_s3.s3_list_objects docker.docker_images linear.linear_list_teams +q132,List Discord channels,verb_product,discord.discord_list_channels,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,321,1.0,1,598,1.0,discord.discord_list_channels discord.discord_get_channel discord.discord_send_message discord.discord_list_members discord.discord_add_reaction slack.slack_list_channels gmail.gmail_list_labels linear.list_labels slack.slack_join_channel slack.slack_leave_channel +q133,Set a Redis key,verb_product,redis.redis_get,0.0,1.0,1.0,0.5000,0.6309,2,1,182,1.0,1,284,1.0,1,550,1.0,redis.redis_get redis.redis_set redis.redis_delete redis.redis_keys slack.get_reminder jira.jira_get_issue figma.figma_get_file gitlab.get_release slack.get_pin slack.get_file +q134,Insert MongoDB documents,verb_product,mongodb.mongo_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,381,1.0,1,751,1.0,mongodb.mongo_insert mongodb.mongo_find mongodb.mongo_update mongodb.mongo_delete mongodb.mongo_list_collections elasticsearch.es_index_doc elasticsearch.es_get_doc supabase.supabase_insert linear.list_comments notion.list_comments +q135,Run a MySQL query,verb_product,mysql.mysql_query,1.0,1.0,1.0,1.0000,1.0000,1,1,179,1.0,1,294,1.0,1,554,1.0,mysql.mysql_query sqlite.sqlite_query postgres.query salesforce.salesforce_query postgres.explain_query github.search_users prometheus.prom_query mysql.mysql_list_tables confluence.confluence_search memory.search_nodes +q136,Index a document in Elasticsearch,verb_product,elasticsearch.es_index_doc,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,369,1.0,1,638,1.0,elasticsearch.es_index_doc elasticsearch.es_get_doc elasticsearch.es_search mongodb.mongo_find linear.get_comment linear.delete_comment linear.get_attachment jira.delete_attachment linear.delete_attachment linear.update_comment +q137,Get an S3 object,verb_product,aws_s3.s3_get_object,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,397,1.0,1,696,1.0,aws_s3.s3_get_object aws_s3.s3_delete_object aws_s3.s3_list_objects aws_s3.s3_put_object salesforce.salesforce_get_record github.projects_get okta.okta_get_user gitlab.gitlab_get_issue asana.asana_get_task github.get_artifact +q138,List Cloudflare zones,verb_product,cloudflare.cf_list_zones,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,366,1.0,1,711,1.0,cloudflare.cf_list_zones cloudflare.cf_list_dns cloudflare.cf_purge_cache cloudflare.cf_create_dns github.list_milestones gitlab.list_milestones clickup.clickup_list_tasks linear.list_cycles circleci.circleci_list_pipelines gitlab.list_pipelines +q139,Create a Stripe invoice,verb_product,stripe.stripe_create_invoice,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,353,1.0,1,762,1.0,stripe.stripe_create_invoice stripe.stripe_create_customer stripe.stripe_create_price stripe.stripe_create_product stripe.stripe_create_refund stripe.stripe_list_invoices stripe.stripe_create_payment_intent notion.create_data_source gitlab.create_pipeline slack.slack_create_channel +q140,List GitHub Actions workflows,verb_product,github.actions_list,1.0,1.0,1.0,1.0000,1.0000,1,1,293,1.0,1,459,1.0,1,763,1.0,github.actions_list github.actions_get github.actions_run_trigger github.list_discussions github.actions_get_job_logs github.list_notifications github.list_gists github.projects_list github.list_tags github.get_gist +q141,Get GitHub Actions job logs,verb_product,github.actions_get_job_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,374,1.0,1,645,1.0,github.actions_get_job_logs github.actions_get github.actions_list github.get_artifact github.get_gist github.get_hook github.get_environment github.get_runner github.projects_get github.get_package +q142,Create a GitHub repository,verb_product,github.create_branch,0.0,1.0,1.0,0.3333,0.5000,3,1,268,1.0,1,406,1.0,1,844,1.0,github.create_branch github.delete_repository github.create_repository github.star_repository github.unstar_repository github.create_issue github.fork_repository github.list_tags github.list_labels github.get_tag +q143,Checkout a git branch,verb_product,git.git_checkout,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,384,1.0,1,814,1.0,git.git_checkout git.git_branch github.list_branches git.git_create_branch github.create_branch github.get_check_run github.delete_file github.update_pull_request_branch git.git_diff github.list_commits +q144,Show git log,identifier,git.git_log,1.0,1.0,1.0,1.0000,1.0000,1,1,193,1.0,1,311,1.0,1,646,1.0,git.git_log git.git_show git.git_diff_staged git.git_diff_unstaged git.git_diff git.git_status shopify.shopify_list_customers github.actions_get_job_logs shopify.shopify_list_orders shopify.shopify_list_products +q145,List postgres schemas,verb_product,postgres.list_schemas,1.0,1.0,1.0,1.0000,1.0000,1,1,152,1.0,1,269,1.0,1,525,1.0,postgres.list_schemas postgres.list_tables mysql.mysql_list_tables postgres.describe_table postgres.query postgres.explain_query pagerduty.pagerduty_list_oncalls github.list_branches docker.docker_images linear.linear_list_teams +q146,Open nodes in the knowledge graph,verb_product,memory.open_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,262,1.0,1,525,1.0,memory.open_nodes memory.search_nodes memory.create_entities memory.create_relations memory.read_graph memory.add_observations memory.delete_relations memory.delete_entities pagerduty.pagerduty_acknowledge_incident sqlite.sqlite_list_tables +q147,Get file metadata from the filesystem,paraphrase,filesystem.get_file_info,1.0,1.0,1.0,1.0000,1.0000,1,1,220,1.0,1,334,1.0,1,635,1.0,filesystem.get_file_info filesystem.read_text_file filesystem.read_file filesystem.read_media_file google_drive.gdrive_get_file gitlab.gitlab_get_file filesystem.list_directory filesystem.list_allowed_directories filesystem.directory_tree filesystem.move_file +q148,Write a new local file,paraphrase,filesystem.write_file,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,342,1.0,1,740,1.0,filesystem.write_file filesystem.read_file filesystem.read_text_file filesystem.create_directory slack.delete_file filesystem.read_media_file slack.update_file filesystem.edit_file git.git_create_branch slack.create_file +q149,Search GitHub issues,verb_product,github.search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,209,1.0,1,333,1.0,1,702,1.0,github.search_issues github.search_users github.search_pull_requests github.search_orgs jira.jira_search_issues github.search_commits linear.linear_search_issues github.search_code github.search_repositories github.list_issues +q150,Get the authenticated GitHub user,paraphrase,github.get_teams,0.0,1.0,1.0,0.5000,0.6309,2,1,179,1.0,1,300,1.0,1,611,1.0,github.get_teams github.get_me github.list_notifications github.get_latest_release github.search_users github.get_runner github.get_repository_tree github.get_hook github.get_environment github.get_gist diff --git a/experiments/mcp-tool-search/results/char_trigram.json b/experiments/mcp-tool-search/results/char_trigram.json new file mode 100644 index 00000000..1afaf463 --- /dev/null +++ b/experiments/mcp-tool-search/results/char_trigram.json @@ -0,0 +1,7013 @@ +{ + "summary": { + "n": 150, + "index_s": 0.005107007999981761, + "rss_mb": 0.1484375, + "p50_ms": 0.34626800004389224, + "p95_ms": 0.4699199499782477, + "recall@1": 0.8533333333333334, + "recall@1_lo": 0.7933333333333333, + "recall@1_hi": 0.9066666666666666, + "recall@3": 0.9666666666666667, + "recall@3_lo": 0.9333333333333333, + "recall@3_hi": 0.9933333333333333, + "recall@5": 0.98, + "recall@5_lo": 0.9533333333333334, + "recall@5_hi": 1.0, + "mrr@10": 0.9118888888888889, + "mrr@10_lo": 0.8766666666666667, + "mrr@10_hi": 0.9456666666666667, + "ndcg@10": 0.9257922189326149, + "ndcg@10_lo": 0.8958366369759104, + "ndcg@10_hi": 0.9537385986569289 + }, + "meta": { + "name": "char_trigram", + "needs_model": false, + "needs_api": false, + "notes": "character trigram cosine \u2014 local analog of codedb trigrams" + }, + "rows": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "gitlab.gitlab_create_issue", + "github.issue_write", + "github.create_gist", + "github.issue_read", + "github.update_issue_comment", + "github.create_branch", + "github.projects_write", + "github.create_hook", + "github.create_repository" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 380, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 557, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1021, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "kind": "verb_product", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_create_issue", + "zendesk.zendesk_create_ticket", + "jira.create_board", + "jira.create_version", + "jira.create_sprint", + "jira.create_filter", + "jira.create_webhook", + "jira.create_attachment", + "jira.jira_create_sprint", + "bitbucket.bitbucket_create_issue" + ], + "hit1": "jira.jira_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 318, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 513, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1004, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "kind": "verb_product", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "ranking": [ + "slack.slack_search_messages", + "slack.slack_post_message", + "slack.slack_add_reaction", + "discord.discord_send_message", + "slack.slack_get_channel_history", + "slack.slack_join_channel", + "slack.get_pin", + "slack.slack_leave_channel", + "gmail.gmail_get_email", + "slack.get_file" + ], + "hit1": "slack.slack_search_messages", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 653, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "kind": "resource", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "ranking": [ + "bitbucket.bitbucket_list_prs", + "github.list_pull_requests", + "github.search_pull_requests", + "github.create_pull_request", + "github.pull_request_read", + "gitlab.gitlab_list_merge_requests", + "github.merge_pull_request", + "github.request_copilot_review", + "bitbucket.bitbucket_create_pr", + "github.add_comment_to_pending_review" + ], + "hit1": "bitbucket.bitbucket_list_prs", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.66967181649423, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 477, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1034, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "ranking": [ + "github.create_pull_request", + "github.request_copilot_review", + "github.search_pull_requests", + "github.pull_request_read", + "github.list_pull_requests", + "github.merge_pull_request", + "github.update_pull_request_branch", + "github.add_comment_to_pending_review", + "github.update_pull_request", + "github.pull_request_review_write" + ], + "hit1": "github.create_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 327, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 505, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1144, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "kind": "verb_product", + "gold": [ + "postgres.query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "mysql.mysql_query", + "sqlite.sqlite_query", + "salesforce.salesforce_query", + "postgres.explain_query", + "postgres.list_schemas", + "postgres.list_tables", + "postgres.describe_table", + "prometheus.prom_query", + "sentry.sentry_list_issues" + ], + "hit1": "postgres.query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 592, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "kind": "paraphrase", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "ranking": [ + "filesystem.read_text_file", + "filesystem.read_file", + "filesystem.read_media_file", + "gitlab.gitlab_get_file", + "github.delete_file", + "supabase.supabase_select", + "github.get_file_contents", + "figma.figma_get_file_nodes", + "figma.figma_get_images", + "slack.slack_upload_file" + ], + "hit1": "filesystem.read_text_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8597186998521972, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 849, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "kind": "paraphrase", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "ranking": [ + "google_drive.gdrive_list_folder", + "slack.list_files", + "slack.get_file", + "filesystem.list_allowed_directories", + "slack.delete_file", + "cloudflare.cf_purge_cache", + "git.git_add", + "google_drive.gdrive_search", + "github.get_file_blame", + "slack.update_file" + ], + "hit1": "google_drive.gdrive_list_folder", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 91, + "k3_calls": 31, + "k3_tokens": 8311, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 19, + "k5_tokens": 8176, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 10, + "k10_tokens": 8240, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "ranking": [ + "memory.read_graph", + "memory.open_nodes", + "memory.delete_relations", + "memory.create_entities", + "memory.create_relations", + "memory.search_nodes", + "memory.add_observations", + "memory.delete_entities", + "pagerduty.pagerduty_acknowledge_incident", + "github.get_artifact" + ], + "hit1": "memory.read_graph", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.4170933196716592, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 335, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 255, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 543, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "kind": "verb_product", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "ranking": [ + "brave_search.brave_web_search", + "jira.jira_search_issues", + "brave_search.brave_local_search", + "github.update_pull_request_branch", + "github.search_users", + "memory.search_nodes", + "github.search_commits", + "github.search_orgs", + "github.search_pull_requests", + "slack.slack_search_messages" + ], + "hit1": "brave_search.brave_web_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "kind": "verb_product", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "ranking": [ + "fetch.fetch", + "filesystem.read_file", + "git.git_add", + "confluence.confluence_search", + "docker.docker_logs", + "github.get_me", + "playwright.browser_navigate", + "figma.figma_get_comments", + "slack.slack_upload_file", + "git.git_status" + ], + "hit1": "fetch.fetch", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 222, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 634, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "kind": "identifier", + "gold": [ + "git.git_status" + ], + "alts": [], + "ranking": [ + "shopify.shopify_list_orders", + "git.git_status", + "gitlab.list_snippets", + "gitlab.list_jobs", + "git.git_diff_staged", + "github.list_hooks", + "gitlab.list_wikis", + "gitlab.list_labels", + "github.list_runners", + "github.list_packages" + ], + "hit1": "shopify.shopify_list_orders", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "kind": "paraphrase", + "gold": [ + "git.git_commit" + ], + "alts": [], + "ranking": [ + "git.git_commit", + "git.git_reset", + "git.git_diff_staged", + "linear.list_comments", + "notion.list_comments", + "github.list_packages", + "slack.slack_get_channel_history", + "git.git_show", + "git.git_add", + "git.git_log" + ], + "hit1": "git.git_commit", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 664, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_issue", + "linear.linear_create_comment", + "linear.linear_assign_issue", + "linear.linear_update_issue", + "linear.linear_get_issue", + "linear.linear_create_project", + "linear.create_user", + "linear.create_label", + "linear.create_cycle", + "linear.create_comment" + ], + "hit1": "linear.linear_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 925, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "kind": "paraphrase", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_list_projects", + "linear.linear_create_project", + "linear.delete_user", + "asana.asana_list_tasks", + "linear.get_user", + "supabase.supabase_list_tables", + "linear.update_user", + "linear.delete_cycle", + "linear.delete_label", + "linear.delete_comment" + ], + "hit1": "linear.linear_list_projects", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 146, + "k3_calls": 49, + "k3_tokens": 12254, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 30, + "k5_tokens": 11994, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11634, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "ranking": [ + "github.update_issue_comment", + "github.add_issue_comment", + "figma.figma_post_comment", + "github.get_discussion_comments", + "jira.jira_add_comment", + "github.list_commits", + "linear.linear_create_comment", + "github.list_issues", + "github.issue_read", + "notion.get_comment" + ], + "hit1": "github.update_issue_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 307, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 468, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 892, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "kind": "ambiguous", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "ranking": [ + "github.merge_pull_request", + "github.update_pull_request_branch", + "gitlab.gitlab_merge_merge_request", + "github.search_pull_requests", + "bitbucket.bitbucket_list_prs", + "gitlab.gitlab_get_merge_request", + "github.pull_request_read", + "gitlab.gitlab_list_merge_requests", + "github.list_pull_requests", + "github.request_copilot_review" + ], + "hit1": "github.merge_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 299, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 859, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "kind": "verb_product", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "ranking": [ + "github.actions_run_trigger", + "github.actions_get", + "github.actions_list", + "github.actions_get_job_logs", + "github.fork_repository", + "github.star_repository", + "github.unstar_repository", + "github.list_discussions", + "github.get_discussion", + "github.update_gist" + ], + "hit1": "github.actions_run_trigger", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 847, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "kind": "resource", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "ranking": [ + "github.get_dependabot_alert", + "github.list_dependabot_alerts", + "github.get_secret_scanning_alert", + "github.get_code_scanning_alert", + "github.list_secret_scanning_alerts", + "vercel.vercel_get_deployment", + "gitlab.get_job", + "github.get_hook", + "gitlab.get_wiki", + "gitlab.get_label" + ], + "hit1": "github.get_dependabot_alert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8597186998521972, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 286, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "kind": "verb_product", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_issues", + "sentry.sentry_list_events", + "sentry.sentry_list_projects", + "sentry.sentry_get_issue", + "linear.linear_search_issues", + "gitlab.gitlab_list_issues", + "sentry.sentry_update_issue", + "linear.linear_list_issues", + "github.list_issues", + "github.search_issues" + ], + "hit1": "sentry.sentry_list_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_create_customer", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_customer", + "stripe.stripe_list_invoices", + "stripe.stripe_create_price", + "stripe.stripe_list_customers", + "stripe.stripe_create_product", + "stripe.stripe_create_refund", + "stripe.stripe_list_subscriptions" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "kind": "paraphrase", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_refund", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_customer", + "stripe.stripe_create_price", + "stripe.stripe_create_product", + "stripe.stripe_cancel_subscription", + "stripe.stripe_create_invoice", + "stripe.stripe_create_customer", + "stripe.stripe_list_invoices" + ], + "hit1": "stripe.stripe_create_refund", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "kind": "verb_product", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "ranking": [ + "gmail.gmail_send_email", + "puppeteer.puppeteer_click", + "gmail.gmail_search_emails", + "twilio.twilio_send_sms", + "gmail.gmail_get_email", + "puppeteer.puppeteer_fill", + "gmail.gmail_list_labels", + "gmail.gmail_trash_email", + "stripe.stripe_list_customers", + "stripe.stripe_create_customer" + ], + "hit1": "gmail.gmail_send_email", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 671, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "kind": "verb_product", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "ranking": [ + "gmail.gmail_search_emails", + "elasticsearch.es_search", + "slack.slack_search_messages", + "github.search_users", + "google_drive.gdrive_search", + "confluence.confluence_search", + "gitlab.gitlab_list_projects", + "hubspot.hubspot_search_contacts", + "gmail.gmail_list_labels", + "github.search_commits" + ], + "hit1": "gmail.gmail_search_emails", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 579, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "kind": "paraphrase", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_update_file", + "google_drive.gdrive_search", + "google_drive.gdrive_share_file", + "google_drive.gdrive_list_folder", + "slack.slack_upload_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_get_file", + "slack.delete_file", + "aws_s3.s3_put_object", + "slack.update_file" + ], + "hit1": "google_drive.gdrive_update_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.16666666666666666, + "ndcg@10": 0.3562071871080222, + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 523, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 786, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_create_event", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_list_events", + "linear.create_comment", + "cloudflare.cf_create_dns", + "notion.create_comment", + "jira.create_attachment", + "stripe.stripe_create_refund", + "confluence.confluence_create_page" + ], + "hit1": "google_calendar.gcal_create_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 484, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 939, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_list_pods", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_apply", + "kubernetes.k8s_delete", + "kubernetes.k8s_scale_deployment", + "docker.docker_images" + ], + "hit1": "kubernetes.k8s_list_pods", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_pod_logs", + "docker.docker_logs", + "kubernetes.k8s_get_pod", + "github.actions_get_job_logs", + "github.get_tag", + "figma.figma_get_file_nodes", + "gitlab.gitlab_get_file", + "github.get_file_contents", + "slack.slack_get_channel_history", + "elasticsearch.es_get_doc" + ], + "hit1": "kubernetes.k8s_pod_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 441, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 858, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "ranking": [ + "docker.docker_stop", + "docker.docker_run", + "docker.docker_exec", + "docker.docker_rm", + "docker.docker_ps", + "docker.docker_logs", + "docker.docker_build", + "kubernetes.k8s_pod_logs", + "docker.docker_images", + "mysql.mysql_query" + ], + "hit1": "docker.docker_stop", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 640, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "kind": "verb_product", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "ranking": [ + "docker.docker_logs", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_ps", + "docker.docker_exec", + "docker.docker_run", + "kubernetes.k8s_pod_logs", + "git.git_log", + "docker.docker_images", + "docker.docker_build" + ], + "hit1": "docker.docker_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 298, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 649, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "kind": "ambiguous", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "ranking": [ + "playwright.browser_navigate", + "puppeteer.puppeteer_navigate", + "playwright.browser_tab_new", + "playwright.browser_press_key", + "playwright.browser_evaluate", + "playwright.browser_wait_for", + "playwright.browser_tab_close", + "playwright.browser_tab_list", + "playwright.browser_click", + "playwright.browser_type" + ], + "hit1": "playwright.browser_navigate", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 282, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 574, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "kind": "ambiguous", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "ranking": [ + "playwright.browser_take_screenshot", + "puppeteer.puppeteer_screenshot", + "playwright.browser_snapshot", + "playwright.browser_evaluate", + "playwright.browser_press_key", + "confluence.confluence_create_page", + "playwright.browser_click", + "git.git_status", + "confluence.confluence_get_page", + "notion.notion_update_page" + ], + "hit1": "playwright.browser_take_screenshot", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 646, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_create_merge_request", + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_get_merge_request", + "gitlab.gitlab_list_merge_requests", + "github.merge_pull_request", + "gitlab.gitlab_create_pipeline", + "bitbucket.bitbucket_create_pr", + "gitlab.gitlab_create_issue", + "gitlab.create_release", + "gitlab.create_milestone" + ], + "hit1": "gitlab.gitlab_create_merge_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 967, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "kind": "verb_product", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "ranking": [ + "notion.notion_search", + "notion.notion_get_page", + "notion.notion_update_page", + "slack.slack_search_messages", + "notion.notion_get_comments", + "notion.notion_create_page", + "github.search_orgs", + "github.search_repositories", + "gmail.gmail_search_emails", + "memory.search_nodes" + ], + "hit1": "notion.notion_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 693, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "kind": "verb_product", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "ranking": [ + "notion.notion_create_page", + "notion.notion_update_page", + "notion.notion_get_page", + "notion.notion_create_database", + "notion.create_comment", + "confluence.confluence_create_page", + "github.create_package", + "notion.notion_get_comments", + "notion.create_users_token", + "notion.create_data_source" + ], + "hit1": "notion.notion_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 413, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 860, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "kind": "verb_product", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "ranking": [ + "notion.notion_query_database", + "notion.notion_search", + "notion.notion_create_database", + "notion.notion_create_page", + "notion.delete_data_source", + "mongodb.mongo_insert", + "notion.get_data_source", + "mongodb.mongo_find", + "mongodb.mongo_list_collections", + "notion.update_data_source" + ], + "hit1": "notion.notion_query_database", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 730, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_search", + "jira.jira_create_issue", + "bitbucket.bitbucket_create_issue", + "clickup.clickup_create_task", + "bitbucket.bitbucket_create_pr", + "gitlab.create_snippet" + ], + "hit1": "zendesk.zendesk_get_ticket", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 892, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_search_contacts", + "hubspot.hubspot_create_deal", + "vercel.vercel_create_project", + "github.create_artifact", + "confluence.confluence_create_page", + "cloudflare.cf_create_dns", + "gmail.gmail_create_draft" + ], + "hit1": "hubspot.hubspot_create_contact", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "kind": "verb_product", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_create_lead", + "salesforce.salesforce_convert_lead", + "salesforce.salesforce_create_record", + "salesforce.salesforce_update_record", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_query", + "salesforce.salesforce_get_record", + "stripe.stripe_create_invoice", + "salesforce.salesforce_list_opportunities", + "vercel.vercel_create_project" + ], + "hit1": "salesforce.salesforce_create_lead", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 710, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "kind": "identifier", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_query", + "mysql.mysql_query", + "sqlite.sqlite_query", + "postgres.query", + "postgres.explain_query", + "notion.notion_query_database", + "github.search_users", + "prometheus.prom_query", + "confluence.confluence_search", + "memory.search_nodes" + ], + "hit1": "salesforce.salesforce_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 592, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "kind": "verb_product", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "ranking": [ + "shopify.shopify_list_orders", + "shopify.shopify_list_customers", + "shopify.shopify_get_order", + "shopify.shopify_create_order", + "shopify.shopify_list_products", + "shopify.shopify_get_product", + "okta.okta_list_users", + "discord.discord_list_members", + "stripe.stripe_list_customers", + "shopify.shopify_update_product" + ], + "hit1": "shopify.shopify_list_orders", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 628, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_list_incidents", + "notion.notion_create_page", + "confluence.confluence_create_page", + "github.create_package", + "stripe.stripe_create_refund", + "google_drive.gdrive_create_file" + ], + "hit1": "pagerduty.pagerduty_create_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 853, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "kind": "paraphrase", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_list_oncalls", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_create_incident", + "notion.notion_get_comments", + "notion.notion_create_page", + "notion.notion_get_page", + "github.list_commits" + ], + "hit1": "pagerduty.pagerduty_list_oncalls", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "kind": "verb_product", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "ranking": [ + "datadog.datadog_query_metrics", + "datadog.datadog_list_monitors", + "datadog.datadog_create_monitor", + "datadog.datadog_list_logs", + "datadog.datadog_get_monitor", + "notion.notion_query_database", + "notion.notion_search", + "prometheus.prom_query", + "prometheus.prom_list_metrics", + "elasticsearch.es_search" + ], + "hit1": "datadog.datadog_query_metrics", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 780, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "kind": "verb_product", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "ranking": [ + "datadog.datadog_list_logs", + "notion.notion_search", + "datadog.datadog_get_monitor", + "datadog.datadog_list_monitors", + "github.search_orgs", + "grafana.grafana_search_dashboards", + "slack.slack_search_messages", + "github.search_users", + "confluence.confluence_search", + "gitlab.gitlab_list_projects" + ], + "hit1": "datadog.datadog_list_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 248, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 646, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_list_buckets", + "aws_s3.s3_list_objects", + "zendesk.zendesk_list_tickets", + "bitbucket.bitbucket_list_repos", + "bitbucket.bitbucket_list_prs", + "aws_s3.s3_delete_object", + "aws_s3.s3_get_object", + "linear.linear_list_projects", + "gitlab.list_snippets", + "docker.docker_images" + ], + "hit1": "aws_s3.s3_list_buckets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 187, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 616, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "kind": "paraphrase", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_put_object", + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "gmail.gmail_send_email", + "slack.slack_upload_file", + "aws_s3.s3_list_objects", + "gmail.gmail_create_draft", + "salesforce.salesforce_update_record", + "twilio.twilio_send_sms", + "salesforce.salesforce_delete_record" + ], + "hit1": "aws_s3.s3_put_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 426, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 830, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "cloudflare.cf_list_dns", + "cloudflare.cf_create_dns", + "playwright.browser_tab_close", + "playwright.browser_snapshot", + "postgres.list_tables", + "memory.read_graph", + "postgres.list_schemas", + "puppeteer.puppeteer_click" + ], + "hit1": "cloudflare.cf_purge_cache", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 548, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "salesforce.salesforce_create_record", + "cloudflare.cf_list_zones", + "cloudflare.cf_purge_cache", + "clickup.clickup_create_task", + "trello.trello_create_board", + "jira.create_board", + "trello.trello_create_card", + "confluence.confluence_create_page" + ], + "hit1": "cloudflare.cf_create_dns", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_deployments", + "vercel.vercel_get_deployment", + "vercel.vercel_list_projects", + "kubernetes.k8s_list_deployments", + "linear.list_comments", + "linear.list_attachments", + "kubernetes.k8s_scale_deployment", + "notion.list_comments", + "jira.list_attachments", + "github.list_environments" + ], + "hit1": "vercel.vercel_list_deployments", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 306, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 710, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "ranking": [ + "supabase.supabase_insert", + "mongodb.mongo_insert", + "supabase.supabase_update", + "supabase.supabase_select", + "supabase.supabase_list_tables", + "supabase.supabase_invoke_fn", + "gitlab.get_release", + "slack.get_canvase", + "gitlab.delete_release", + "notion.notion_create_database" + ], + "hit1": "supabase.supabase_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 681, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "kind": "paraphrase", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "ranking": [ + "figma.figma_get_images", + "figma.figma_get_file_nodes", + "docker.docker_images", + "filesystem.read_media_file", + "slack.slack_get_channel_history", + "figma.figma_get_file", + "supabase.supabase_select", + "docker.docker_run", + "figma.figma_post_comment", + "figma.figma_get_comments" + ], + "hit1": "figma.figma_get_images", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 692, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "kind": "verb_product", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "ranking": [ + "twilio.twilio_send_sms", + "twilio.twilio_list_messages", + "puppeteer.puppeteer_click", + "gmail.gmail_send_email", + "puppeteer.puppeteer_fill", + "jira.jira_search_issues", + "filesystem.list_directory_with_sizes", + "twilio.twilio_make_call", + "puppeteer.puppeteer_screenshot", + "discord.discord_send_message" + ], + "hit1": "twilio.twilio_send_sms", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "kind": "verb_product", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "ranking": [ + "okta.okta_create_user", + "okta.okta_deactivate_user", + "okta.okta_get_user", + "linear.create_user", + "okta.okta_list_users", + "slack.create_usergroup", + "notion.create_users_token", + "asana.asana_create_project", + "linear.update_user", + "stripe.stripe_create_invoice" + ], + "hit1": "okta.okta_create_user", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 719, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query", + "prometheus.prom_query_range", + "prometheus.prom_list_metrics", + "shopify.shopify_list_products", + "mysql.mysql_query", + "sqlite.sqlite_query", + "salesforce.salesforce_query", + "sentry.sentry_list_issues", + "postgres.query", + "postgres.explain_query" + ], + "hit1": "prometheus.prom_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "kind": "verb_product", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "ranking": [ + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "github.search_users", + "github.search_orgs", + "github.search_pull_requests", + "slack.slack_search_messages", + "github.search_issues", + "confluence.confluence_search", + "datadog.datadog_list_logs", + "gmail.gmail_search_emails" + ], + "hit1": "grafana.grafana_search_dashboards", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 287, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 666, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "kind": "verb_product", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "ranking": [ + "circleci.circleci_trigger_pipeline", + "circleci.circleci_get_pipeline", + "circleci.circleci_list_pipelines", + "gitlab.gitlab_create_pipeline", + "gitlab.get_pipeline", + "gitlab.delete_pipeline", + "gitlab.update_pipeline", + "hubspot.hubspot_create_deal", + "gitlab.create_pipeline", + "github.actions_run_trigger" + ], + "hit1": "circleci.circleci_trigger_pipeline", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 325, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "kind": "identifier", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "ranking": [ + "terraform.tf_plan", + "terraform.tf_apply", + "terraform.tf_state_list", + "playwright.browser_fill_form", + "salesforce.salesforce_query", + "playwright.browser_wait_for", + "playwright.browser_click", + "playwright.browser_type", + "mysql.mysql_query", + "sqlite.sqlite_query" + ], + "hit1": "terraform.tf_plan", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 593, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "kind": "verb_product", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "ranking": [ + "asana.asana_create_task", + "asana.asana_get_task", + "asana.asana_update_task", + "asana.asana_create_project", + "clickup.clickup_create_task", + "grafana.grafana_create_annotation", + "okta.okta_create_user", + "asana.asana_list_tasks", + "trello.trello_create_board", + "github.create_issue" + ], + "hit1": "asana.asana_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 822, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "ranking": [ + "trello.trello_create_card", + "trello.trello_create_board", + "trello.trello_add_comment", + "trello.trello_list_cards", + "jira.create_board", + "vercel.vercel_create_project", + "salesforce.salesforce_create_record", + "confluence.confluence_create_page", + "stripe.stripe_create_invoice", + "stripe.stripe_create_customer" + ], + "hit1": "trello.trello_create_card", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 411, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 745, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_create_task", + "clickup.clickup_get_task", + "clickup.clickup_update_task", + "clickup.clickup_list_tasks", + "asana.asana_create_task", + "confluence.confluence_create_page", + "cloudflare.cf_create_dns", + "trello.trello_create_board", + "vercel.vercel_create_project", + "github.create_gist" + ], + "hit1": "clickup.clickup_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 415, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 802, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "kind": "verb_product", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "ranking": [ + "confluence.confluence_create_page", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "confluence.confluence_search", + "pagerduty.pagerduty_create_incident", + "cloudflare.cf_create_dns", + "stripe.stripe_create_price", + "notion.notion_create_page", + "github.create_package", + "stripe.stripe_create_payment_intent" + ], + "hit1": "confluence.confluence_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 870, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "kind": "verb_product", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "discord.discord_add_reaction", + "discord.discord_get_channel", + "discord.discord_list_members", + "gmail.gmail_get_email", + "slack.slack_post_message", + "gmail.gmail_trash_email", + "git.git_commit", + "slack.slack_search_messages", + "datadog.datadog_create_monitor" + ], + "hit1": "discord.discord_send_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 701, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_search", + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "slack.slack_search_messages", + "github.search_users", + "zendesk.zendesk_search", + "confluence.confluence_search", + "gitlab.gitlab_list_projects", + "hubspot.hubspot_search_contacts", + "github.search_commits" + ], + "hit1": "elasticsearch.es_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 630, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "redis.redis_set", + "redis.redis_delete", + "redis.redis_keys", + "gitlab.get_release", + "jira.jira_get_issue", + "figma.figma_get_file", + "github.get_runner", + "slack.get_reminder", + "github.get_discussion" + ], + "hit1": "redis.redis_get", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 283, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 582, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_find", + "mongodb.mongo_update", + "mongodb.mongo_delete", + "mongodb.mongo_insert", + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "mongodb.mongo_list_collections", + "figma.figma_get_comments", + "github.get_discussion_comments", + "filesystem.read_file" + ], + "hit1": "mongodb.mongo_find", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "kind": "verb_product", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "ranking": [ + "sqlite.sqlite_list_tables", + "sqlite.sqlite_describe_table", + "mysql.mysql_list_tables", + "supabase.supabase_list_tables", + "postgres.list_tables", + "sqlite.sqlite_query", + "postgres.list_schemas", + "salesforce.salesforce_list_opportunities", + "linear.list_cycles", + "docker.docker_images" + ], + "hit1": "sqlite.sqlite_list_tables", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 240, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 487, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "kind": "verb_product", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "ranking": [ + "postgres.describe_table", + "sqlite.sqlite_describe_table", + "postgres.explain_query", + "postgres.list_tables", + "postgres.list_schemas", + "postgres.query", + "slack.create_pin", + "github.create_package", + "gitlab.create_pipeline", + "github.update_gist" + ], + "hit1": "postgres.describe_table", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 274, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 697, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "gitlab.gitlab_create_issue", + "github.create_issue", + "github.update_issue_comment", + "bitbucket.bitbucket_create_issue", + "github.create_gist", + "github.search_issues", + "github.issue_read", + "github.issue_write", + "github.create_branch", + "github.create_hook" + ], + "hit1": "gitlab.gitlab_create_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.5995311664203287, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 347, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 516, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1032, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "ranking": [ + "bitbucket.bitbucket_create_issue", + "gitlab.gitlab_create_issue", + "linear.linear_create_issue", + "github.create_issue", + "jira.jira_create_issue", + "linear.linear_create_comment", + "github.issue_write", + "confluence.confluence_create_page", + "trello.trello_create_board", + "okta.okta_create_user" + ], + "hit1": "bitbucket.bitbucket_create_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5583663834639362, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 332, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 579, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 982, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "kind": "identifier", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "ranking": [ + "filesystem.list_directory", + "filesystem.list_directory_with_sizes", + "filesystem.create_directory", + "filesystem.list_allowed_directories", + "github.get_file_contents", + "filesystem.directory_tree", + "linear.linear_list_projects", + "filesystem.get_file_info", + "datadog.datadog_list_monitors", + "linear.linear_list_teams" + ], + "hit1": "filesystem.list_directory", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 586, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "kind": "paraphrase", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "ranking": [ + "sequential_thinking.sequentialthinking", + "filesystem.edit_file", + "confluence.confluence_search", + "playwright.browser_wait_for", + "brave_search.brave_web_search", + "playwright.browser_type", + "brave_search.brave_local_search", + "github.add_comment_to_pending_review", + "github.list_notifications", + "memory.add_observations" + ], + "hit1": "sequential_thinking.sequentialthinking", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 278, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 434, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 862, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "kind": "paraphrase", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "ranking": [ + "time.get_current_time", + "playwright.browser_wait_for", + "time.convert_time", + "notion.list_users_tokens", + "sqlite.sqlite_list_tables", + "notion.create_users_token", + "gitlab.list_wikis", + "github.list_tags", + "grafana.grafana_create_annotation", + "linear.linear_create_issue" + ], + "hit1": "time.get_current_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 804, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "kind": "verb_product", + "gold": [ + "time.convert_time" + ], + "alts": [], + "ranking": [ + "time.convert_time", + "time.get_current_time", + "git.git_diff", + "salesforce.salesforce_convert_lead", + "memory.create_relations", + "cloudflare.cf_list_zones", + "cloudflare.cf_create_dns", + "confluence.confluence_update_page", + "grafana.grafana_create_annotation", + "confluence.confluence_create_page" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "kind": "verb_product", + "gold": [ + "github.star_repository" + ], + "alts": [], + "ranking": [ + "github.star_repository", + "github.unstar_repository", + "github.delete_repository", + "github.list_issues", + "github.list_pull_requests", + "github.list_tags", + "github.list_labels", + "github.get_label", + "github.get_tag", + "github.list_starred_repositories" + ], + "hit1": "github.star_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 778, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "kind": "resource", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "ranking": [ + "github.fork_repository", + "github.list_releases", + "github.star_repository", + "github.delete_repository", + "github.unstar_repository", + "github.list_dependabot_alerts", + "github.list_discussions", + "github.get_file_blame", + "github.list_tags", + "github.list_labels" + ], + "hit1": "github.fork_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "kind": "verb_product", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "ranking": [ + "github.get_latest_release", + "github.get_release_by_tag", + "github.list_releases", + "gitlab.get_release", + "github.get_teams", + "github.get_gist", + "github.get_repository_tree", + "github.get_artifact", + "github.get_label", + "github.get_tag" + ], + "hit1": "github.get_latest_release", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "kind": "resource", + "gold": [ + "github.create_gist" + ], + "alts": [], + "ranking": [ + "github.create_gist", + "trello.trello_create_card", + "gitlab.gitlab_create_pipeline", + "clickup.clickup_create_task", + "trello.trello_create_board", + "gmail.gmail_create_draft", + "github.create_branch", + "gitlab.gitlab_create_issue", + "vercel.vercel_create_project", + "github.update_gist" + ], + "hit1": "github.create_gist", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 817, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "kind": "verb_product", + "gold": [ + "github.search_code" + ], + "alts": [], + "ranking": [ + "github.search_users", + "github.search_code", + "github.search_commits", + "github.search_orgs", + "github.search_pull_requests", + "github.search_issues", + "github.search_repositories", + "github.list_code_scanning_alerts", + "github.get_code_scanning_alert", + "gitlab.gitlab_list_projects" + ], + "hit1": "github.search_users", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 191, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 719, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "kind": "verb_product", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "ranking": [ + "github.list_notifications", + "github.mark_all_notifications_read", + "github.get_notification_details", + "github.dismiss_notification", + "github.actions_list", + "github.list_discussions", + "github.list_gists", + "github.projects_list", + "github.list_tags", + "github.get_gist" + ], + "hit1": "github.list_notifications", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 176, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "kind": "verb_product", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "ranking": [ + "github.assign_copilot_to_issue", + "linear.linear_assign_issue", + "jira.jira_assign_issue", + "github.create_issue", + "jira.jira_add_comment", + "linear.linear_create_comment", + "github.add_issue_comment", + "jira.jira_transition_issue", + "linear.linear_list_issues", + "jira.jira_add_worklog" + ], + "hit1": "github.assign_copilot_to_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 467, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 883, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "kind": "resource", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "ranking": [ + "github.list_secret_scanning_alerts", + "github.get_secret_scanning_alert", + "github.get_code_scanning_alert", + "github.list_code_scanning_alerts", + "github.get_dependabot_alert", + "github.list_dependabot_alerts", + "docker.docker_ps", + "git.git_add", + "gitlab.get_job", + "github.get_hook" + ], + "hit1": "github.list_secret_scanning_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 483, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "ranking": [ + "github.projects_write", + "github.create_or_update_file", + "github.update_runner", + "github.update_gist", + "github.update_milestone", + "github.update_hook", + "github.create_gist", + "github.update_package", + "github.update_artifact", + "github.issue_write" + ], + "hit1": "github.projects_write", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.4796249331362629, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 318, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 469, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 881, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "kind": "resource", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "ranking": [ + "github.list_repository_collaborators", + "github.list_commits", + "github.list_labels", + "github.list_discussions", + "github.list_tags", + "github.list_branches", + "github.list_releases", + "github.list_issues", + "github.list_dependabot_alerts", + "trello.trello_list_cards" + ], + "hit1": "github.list_repository_collaborators", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 271, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 413, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 813, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "kind": "verb_product", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "ranking": [ + "slack.slack_get_thread_replies", + "slack.slack_reply_to_thread", + "slack.get_reminder", + "slack.get_pin", + "slack.slack_join_channel", + "slack.delete_reminder", + "slack.get_file", + "slack.delete_pin", + "slack.get_canvase", + "slack.get_bookmark" + ], + "hit1": "slack.slack_get_thread_replies", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 347, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 621, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "kind": "verb_product", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "ranking": [ + "slack.slack_add_reaction", + "discord.discord_add_reaction", + "slack.create_reminder", + "slack.create_pin", + "slack.create_file", + "slack.create_bookmark", + "slack.create_canvase", + "slack.create_usergroup", + "slack.get_reminder", + "slack.delete_reminder" + ], + "hit1": "slack.slack_add_reaction", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 302, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 496, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 919, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "kind": "verb_product", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "ranking": [ + "slack.slack_search_messages", + "gmail.gmail_search_emails", + "slack.slack_get_channel_history", + "slack.slack_post_message", + "slack.slack_add_reaction", + "notion.notion_search", + "twilio.twilio_list_messages", + "brave_search.brave_local_search", + "confluence.confluence_search", + "google_drive.gdrive_search" + ], + "hit1": "slack.slack_search_messages", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 392, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "kind": "verb_product", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "ranking": [ + "slack.slack_create_channel", + "slack.slack_leave_channel", + "slack.slack_join_channel", + "slack.slack_set_channel_topic", + "slack.slack_post_message", + "slack.create_pin", + "slack.slack_get_thread_replies", + "slack.create_file", + "slack.slack_reply_to_thread", + "slack.create_bookmark" + ], + "hit1": "slack.slack_create_channel", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 801, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "kind": "verb_product", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_transition_issue", + "jira.jira_get_issue", + "jira.jira_assign_issue", + "jira.jira_create_issue", + "jira.jira_add_comment", + "jira.jira_update_issue", + "jira.jira_add_worklog", + "jira.jira_search_issues", + "sentry.sentry_get_issue", + "jira.get_version" + ], + "hit1": "jira.jira_transition_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "kind": "identifier", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "ranking": [ + "jira.jira_search_issues", + "github.search_users", + "github.search_commits", + "github.search_orgs", + "github.search_pull_requests", + "slack.slack_search_messages", + "github.search_issues", + "confluence.confluence_search", + "gitlab.gitlab_list_projects", + "elasticsearch.es_search" + ], + "hit1": "jira.jira_search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 627, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "kind": "verb_product", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "ranking": [ + "jira.jira_add_worklog", + "jira.jira_add_comment", + "jira.get_webhook", + "jira.delete_webhook", + "linear.linear_create_comment", + "jira.jira_transition_issue", + "jira.update_webhook", + "jira.get_board", + "jira.get_sprint", + "jira.get_filter" + ], + "hit1": "jira.jira_add_worklog", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 233, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "kind": "verb_product", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "ranking": [ + "linear.linear_list_teams", + "linear.linear_list_projects", + "linear.linear_list_issues", + "linear.list_labels", + "linear.list_cycles", + "linear.list_users", + "linear.list_comments", + "linear.list_attachments", + "linear.linear_create_project", + "linear.get_user" + ], + "hit1": "linear.linear_list_teams", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "ranking": [ + "linear.linear_create_comment", + "linear.get_comment", + "linear.delete_comment", + "linear.linear_get_issue", + "linear.linear_assign_issue", + "linear.update_comment", + "github.update_issue_comment", + "linear.create_comment", + "linear.linear_update_issue", + "linear.linear_create_issue" + ], + "hit1": "linear.linear_create_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 831, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "kind": "paraphrase", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "ranking": [ + "filesystem.read_text_file", + "filesystem.read_file", + "filesystem.read_media_file", + "slack.delete_file", + "filesystem.move_file", + "slack.update_file", + "google_drive.gdrive_share_file", + "gmail.gmail_trash_email", + "google_drive.gdrive_update_file", + "figma.figma_get_file" + ], + "hit1": "filesystem.read_text_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.2, + "ndcg@10": 0.38685280723454163, + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 422, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 689, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "kind": "paraphrase", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "ranking": [ + "filesystem.search_files", + "google_drive.gdrive_search", + "filesystem.directory_tree", + "linear.linear_search_issues", + "notion.notion_search", + "redis.redis_keys", + "github.search_repositories", + "github.search_issues", + "memory.search_nodes", + "github.search_pull_requests" + ], + "hit1": "filesystem.search_files", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "kind": "paraphrase", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "ranking": [ + "filesystem.edit_file", + "slack.get_file", + "slack.delete_file", + "filesystem.list_directory_with_sizes", + "slack.update_file", + "github.get_file_blame", + "filesystem.list_directory", + "filesystem.write_file", + "slack.create_file", + "git.git_add" + ], + "hit1": "filesystem.edit_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "ranking": [ + "memory.read_graph", + "memory.delete_relations", + "memory.create_entities", + "memory.create_relations", + "memory.open_nodes", + "memory.search_nodes", + "memory.delete_entities", + "memory.add_observations", + "pagerduty.pagerduty_acknowledge_incident", + "filesystem.read_text_file" + ], + "hit1": "memory.read_graph", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 253, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 574, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "kind": "verb_product", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "slack.slack_search_messages", + "memory.open_nodes", + "notion.notion_search", + "linear.linear_search_issues", + "github.search_users", + "github.search_issues", + "zendesk.zendesk_search", + "confluence.confluence_search", + "google_drive.gdrive_search" + ], + "hit1": "memory.search_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 621, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "kind": "verb_product", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "ranking": [ + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_reset", + "git.git_diff", + "git.git_show", + "git.git_log", + "git.git_status", + "git.git_commit", + "github.unstar_repository", + "shopify.shopify_list_orders" + ], + "hit1": "git.git_diff_unstaged", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 174, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 614, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "kind": "identifier", + "gold": [ + "git.git_add" + ], + "alts": [], + "ranking": [ + "git.git_add", + "slack.list_files", + "git.git_reset", + "filesystem.list_directory_with_sizes", + "github.list_milestones", + "git.git_commit", + "salesforce.salesforce_list_opportunities", + "github.push_files", + "github.delete_file", + "git.git_diff_staged" + ], + "hit1": "git.git_add", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 209, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 745, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "kind": "verb_product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "ranking": [ + "shopify.shopify_create_product", + "shopify.shopify_update_product", + "shopify.shopify_get_product", + "stripe.stripe_create_product", + "shopify.shopify_create_order", + "shopify.shopify_list_products", + "stripe.stripe_create_price", + "vercel.vercel_create_project", + "shopify.shopify_get_order", + "linear.linear_create_project" + ], + "hit1": "shopify.shopify_create_product", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "kind": "verb_product", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "ranking": [ + "stripe.stripe_cancel_subscription", + "stripe.stripe_list_subscriptions", + "stripe.stripe_create_product", + "slack.create_canvase", + "stripe.stripe_get_customer", + "stripe.stripe_create_invoice", + "stripe.stripe_create_customer", + "clickup.clickup_create_task", + "gitlab.create_label", + "stripe.stripe_get_payment_intent" + ], + "hit1": "stripe.stripe_cancel_subscription", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 203, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 361, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "kind": "resource", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_refund", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent", + "notion.create_comment", + "google_drive.gdrive_create_file", + "pagerduty.pagerduty_create_incident", + "linear.create_comment", + "jira.create_attachment", + "linear.create_attachment", + "cloudflare.cf_create_dns" + ], + "hit1": "stripe.stripe_create_refund", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 439, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 931, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_list_incidents", + "memory.read_graph", + "github.delete_package", + "memory.create_entities", + "pagerduty.pagerduty_list_oncalls", + "github.update_package" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "kind": "verb_product", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "ranking": [ + "datadog.datadog_create_monitor", + "datadog.datadog_get_monitor", + "datadog.datadog_list_monitors", + "notion.create_data_source", + "notion.notion_create_database", + "notion.notion_create_page", + "filesystem.create_directory", + "trello.trello_create_board", + "github.create_branch", + "vercel.vercel_create_project" + ], + "hit1": "datadog.datadog_create_monitor", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 448, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 805, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "kind": "paraphrase", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "ranking": [ + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "sentry.sentry_list_events", + "pagerduty.pagerduty_resolve_incident", + "sentry.sentry_list_issues", + "linear.linear_get_issue", + "bitbucket.bitbucket_create_issue", + "github.search_issues", + "jira.jira_assign_issue", + "github.issue_write" + ], + "hit1": "sentry.sentry_get_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 848, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_delete", + "vercel.vercel_get_deployment", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_apply", + "kubernetes.k8s_get_events", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_list_services" + ], + "hit1": "kubernetes.k8s_scale_deployment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 400, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_apply", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_delete", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_pod_logs", + "redis.redis_keys" + ], + "hit1": "kubernetes.k8s_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 222, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "ranking": [ + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_run", + "docker.docker_ps", + "docker.docker_exec", + "docker.docker_build", + "kubernetes.k8s_pod_logs", + "docker.docker_images", + "stripe.stripe_create_customer" + ], + "hit1": "docker.docker_stop", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 655, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "kind": "verb_product", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "ranking": [ + "docker.docker_build", + "docker.docker_images", + "docker.docker_run", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_exec", + "docker.docker_ps", + "filesystem.read_media_file", + "jira.delete_filter" + ], + "hit1": "docker.docker_build", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 598, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "kind": "verb_product", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "ranking": [ + "playwright.browser_click", + "playwright.browser_type", + "puppeteer.puppeteer_click", + "playwright.browser_take_screenshot", + "playwright.browser_tab_close", + "playwright.browser_snapshot", + "playwright.browser_tab_list", + "playwright.browser_tab_new", + "playwright.browser_navigate", + "playwright.browser_press_key" + ], + "hit1": "playwright.browser_click", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 566, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "kind": "paraphrase", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "ranking": [ + "playwright.browser_type", + "supabase.supabase_insert", + "playwright.browser_wait_for", + "playwright.browser_navigate", + "playwright.browser_tab_close", + "playwright.browser_tab_new", + "slack.slack_post_message", + "playwright.browser_tab_list", + "playwright.browser_press_key", + "slack.slack_reply_to_thread" + ], + "hit1": "playwright.browser_type", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 233, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 629, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_list_projects", + "gitlab.gitlab_list_issues", + "github.projects_list", + "gitlab.gitlab_list_merge_requests", + "linear.linear_list_projects", + "jira.jira_list_projects", + "vercel.vercel_list_projects", + "gitlab.gitlab_get_merge_request", + "gitlab.gitlab_merge_merge_request", + "sentry.sentry_list_projects" + ], + "hit1": "gitlab.gitlab_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 209, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 609, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "kind": "verb_product", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "ranking": [ + "bitbucket.bitbucket_create_pr", + "bitbucket.bitbucket_list_prs", + "bitbucket.bitbucket_create_issue", + "zendesk.zendesk_create_ticket", + "github.list_pull_requests", + "github.pull_request_review_write", + "github.create_pull_request", + "github.pull_request_read", + "github.update_pull_request_branch", + "github.merge_pull_request" + ], + "hit1": "bitbucket.bitbucket_create_pr", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 489, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1072, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "kind": "verb_product", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "ranking": [ + "gmail.gmail_create_draft", + "stripe.stripe_create_customer", + "shopify.shopify_create_order", + "hubspot.hubspot_create_contact", + "salesforce.salesforce_create_lead", + "grafana.grafana_create_annotation", + "trello.trello_create_board", + "vercel.vercel_create_project", + "github.create_gist", + "gitlab.gitlab_create_pipeline" + ], + "hit1": "gmail.gmail_create_draft", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 431, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 780, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "kind": "verb_product", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_share_file", + "google_drive.gdrive_update_file", + "google_drive.gdrive_list_folder", + "google_drive.gdrive_search", + "google_drive.gdrive_create_file", + "google_drive.gdrive_get_file", + "slack.delete_file", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "slack.update_file" + ], + "hit1": "google_drive.gdrive_share_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "kind": "verb_product", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "kubernetes.k8s_get_events", + "google_drive.gdrive_list_folder", + "sentry.sentry_list_events", + "linear.list_comments", + "linear.list_attachments" + ], + "hit1": "google_calendar.gcal_list_events", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "linear.delete_comment", + "google_calendar.gcal_create_event", + "notion.delete_comment", + "linear.delete_attachment", + "linear.delete_cycle", + "google_calendar.gcal_list_events", + "slack.delete_canvase", + "jira.delete_attachment" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_projects", + "vercel.vercel_list_deployments", + "linear.linear_list_projects", + "jira.jira_list_projects", + "vercel.vercel_create_project", + "gitlab.gitlab_list_projects", + "sentry.sentry_list_projects", + "github.projects_list", + "asana.asana_list_tasks", + "supabase.supabase_list_tables" + ], + "hit1": "vercel.vercel_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 255, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 530, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "ranking": [ + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update", + "mongodb.mongo_delete", + "supabase.supabase_list_tables", + "supabase.supabase_invoke_fn", + "aws_s3.s3_get_object", + "mongodb.mongo_list_collections", + "slack.slack_get_channel_history", + "memory.delete_relations" + ], + "hit1": "supabase.supabase_select", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 351, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 670, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "kind": "verb_product", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "figma.figma_get_comments", + "linear.get_comment", + "notion.get_comment", + "linear.update_comment", + "notion.update_comment", + "linear.delete_comment", + "notion.delete_comment", + "figma.figma_get_file", + "jira.jira_add_comment" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 201, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 612, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "kind": "verb_product", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "ranking": [ + "okta.okta_list_users", + "linear.list_users", + "slack.slack_get_users", + "notion.notion_list_users", + "okta.okta_get_user", + "okta.okta_create_user", + "slack.slack_list_users", + "github.list_gists", + "okta.okta_deactivate_user", + "notion.list_users_tokens" + ], + "hit1": "okta.okta_list_users", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 607, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query_range", + "prometheus.prom_query", + "prometheus.prom_list_metrics", + "shopify.shopify_list_products", + "postgres.explain_query", + "datadog.datadog_query_metrics", + "github.search_code", + "datadog.datadog_create_monitor", + "sentry.sentry_list_issues", + "github.search_users" + ], + "hit1": "prometheus.prom_query_range", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 347, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 776, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "kind": "verb_product", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "ranking": [ + "grafana.grafana_create_annotation", + "github.create_gist", + "clickup.clickup_create_task", + "asana.asana_create_project", + "notion.notion_create_database", + "asana.asana_create_task", + "notion.create_comment", + "stripe.stripe_create_product", + "gitlab.gitlab_create_issue", + "github.create_artifact" + ], + "hit1": "grafana.grafana_create_annotation", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 275, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 432, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 893, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "kind": "identifier", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "ranking": [ + "terraform.tf_apply", + "terraform.tf_plan", + "terraform.tf_state_list", + "kubernetes.k8s_apply", + "slack.slack_reply_to_thread", + "playwright.browser_fill_form", + "filesystem.search_files", + "linear.linear_list_issues", + "docker.docker_rm", + "linear.linear_search_issues" + ], + "hit1": "terraform.tf_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 644, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "kind": "verb_product", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "ranking": [ + "asana.asana_list_tasks", + "asana.asana_get_task", + "clickup.clickup_list_tasks", + "asana.asana_update_task", + "asana.asana_create_task", + "linear.linear_list_teams", + "aws_s3.s3_list_buckets", + "linear.linear_list_projects", + "mysql.mysql_list_tables", + "linear.list_attachments" + ], + "hit1": "asana.asana_list_tasks", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 631, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "ranking": [ + "trello.trello_add_comment", + "trello.trello_list_cards", + "trello.trello_create_card", + "figma.figma_post_comment", + "trello.trello_create_board", + "github.update_issue_comment", + "figma.figma_get_comments", + "notion.get_comment", + "notion.delete_comment", + "notion.notion_get_comments" + ], + "hit1": "trello.trello_add_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 712, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_update_task", + "clickup.clickup_get_task", + "clickup.clickup_create_task", + "clickup.clickup_list_tasks", + "asana.asana_update_task", + "linear.update_cycle", + "zendesk.zendesk_update_ticket", + "slack.update_canvase", + "linear.update_comment", + "notion.update_comment" + ], + "hit1": "clickup.clickup_update_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 760, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "kind": "verb_product", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "ranking": [ + "confluence.confluence_search", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "confluence.confluence_create_page", + "hubspot.hubspot_search_contacts", + "github.search_commits", + "slack.slack_search_messages", + "github.search_users", + "github.search_code", + "gitlab.gitlab_list_projects" + ], + "hit1": "confluence.confluence_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 369, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_deal", + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "github.create_hook", + "github.create_gist", + "clickup.clickup_create_task", + "trello.trello_create_board", + "trello.trello_create_card", + "vercel.vercel_create_project", + "github.create_repository" + ], + "hit1": "hubspot.hubspot_create_deal", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 805, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_search", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_create_ticket", + "aws_s3.s3_list_buckets", + "linear.linear_list_projects", + "aws_s3.s3_list_objects", + "docker.docker_images", + "linear.linear_list_teams" + ], + "hit1": "zendesk.zendesk_list_tickets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 593, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "kind": "verb_product", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "ranking": [ + "discord.discord_list_channels", + "discord.discord_get_channel", + "discord.discord_send_message", + "discord.discord_list_members", + "discord.discord_add_reaction", + "slack.slack_list_channels", + "gmail.gmail_list_labels", + "linear.list_labels", + "slack.slack_join_channel", + "slack.slack_leave_channel" + ], + "hit1": "discord.discord_list_channels", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 598, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "redis.redis_set", + "redis.redis_delete", + "redis.redis_keys", + "slack.get_reminder", + "jira.jira_get_issue", + "figma.figma_get_file", + "gitlab.get_release", + "slack.get_pin", + "slack.get_file" + ], + "hit1": "redis.redis_get", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 284, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert", + "mongodb.mongo_find", + "mongodb.mongo_update", + "mongodb.mongo_delete", + "mongodb.mongo_list_collections", + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "supabase.supabase_insert", + "linear.list_comments", + "notion.list_comments" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "kind": "verb_product", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "ranking": [ + "mysql.mysql_query", + "sqlite.sqlite_query", + "postgres.query", + "salesforce.salesforce_query", + "postgres.explain_query", + "github.search_users", + "prometheus.prom_query", + "mysql.mysql_list_tables", + "confluence.confluence_search", + "memory.search_nodes" + ], + "hit1": "mysql.mysql_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 554, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "elasticsearch.es_search", + "mongodb.mongo_find", + "linear.get_comment", + "linear.delete_comment", + "linear.get_attachment", + "jira.delete_attachment", + "linear.delete_attachment", + "linear.update_comment" + ], + "hit1": "elasticsearch.es_index_doc", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 369, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 638, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_list_objects", + "aws_s3.s3_put_object", + "salesforce.salesforce_get_record", + "github.projects_get", + "okta.okta_get_user", + "gitlab.gitlab_get_issue", + "asana.asana_get_task", + "github.get_artifact" + ], + "hit1": "aws_s3.s3_get_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 696, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones", + "cloudflare.cf_list_dns", + "cloudflare.cf_purge_cache", + "cloudflare.cf_create_dns", + "github.list_milestones", + "gitlab.list_milestones", + "clickup.clickup_list_tasks", + "linear.list_cycles", + "circleci.circleci_list_pipelines", + "gitlab.list_pipelines" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 711, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_create_customer", + "stripe.stripe_create_price", + "stripe.stripe_create_product", + "stripe.stripe_create_refund", + "stripe.stripe_list_invoices", + "stripe.stripe_create_payment_intent", + "notion.create_data_source", + "gitlab.create_pipeline", + "slack.slack_create_channel" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 762, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "kind": "verb_product", + "gold": [ + "github.actions_list" + ], + "alts": [], + "ranking": [ + "github.actions_list", + "github.actions_get", + "github.actions_run_trigger", + "github.list_discussions", + "github.actions_get_job_logs", + "github.list_notifications", + "github.list_gists", + "github.projects_list", + "github.list_tags", + "github.get_gist" + ], + "hit1": "github.actions_list", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "kind": "verb_product", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "ranking": [ + "github.actions_get_job_logs", + "github.actions_get", + "github.actions_list", + "github.get_artifact", + "github.get_gist", + "github.get_hook", + "github.get_environment", + "github.get_runner", + "github.projects_get", + "github.get_package" + ], + "hit1": "github.actions_get_job_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 374, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "kind": "verb_product", + "gold": [ + "github.create_repository" + ], + "alts": [], + "ranking": [ + "github.create_branch", + "github.delete_repository", + "github.create_repository", + "github.star_repository", + "github.unstar_repository", + "github.create_issue", + "github.fork_repository", + "github.list_tags", + "github.list_labels", + "github.get_tag" + ], + "hit1": "github.create_branch", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 406, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 844, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "kind": "verb_product", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "ranking": [ + "git.git_checkout", + "git.git_branch", + "github.list_branches", + "git.git_create_branch", + "github.create_branch", + "github.get_check_run", + "github.delete_file", + "github.update_pull_request_branch", + "git.git_diff", + "github.list_commits" + ], + "hit1": "git.git_checkout", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 814, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "kind": "identifier", + "gold": [ + "git.git_log" + ], + "alts": [], + "ranking": [ + "git.git_log", + "git.git_show", + "git.git_diff_staged", + "git.git_diff_unstaged", + "git.git_diff", + "git.git_status", + "shopify.shopify_list_customers", + "github.actions_get_job_logs", + "shopify.shopify_list_orders", + "shopify.shopify_list_products" + ], + "hit1": "git.git_log", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 311, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 646, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "kind": "verb_product", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "ranking": [ + "postgres.list_schemas", + "postgres.list_tables", + "mysql.mysql_list_tables", + "postgres.describe_table", + "postgres.query", + "postgres.explain_query", + "pagerduty.pagerduty_list_oncalls", + "github.list_branches", + "docker.docker_images", + "linear.linear_list_teams" + ], + "hit1": "postgres.list_schemas", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 269, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 525, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "kind": "verb_product", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "ranking": [ + "memory.open_nodes", + "memory.search_nodes", + "memory.create_entities", + "memory.create_relations", + "memory.read_graph", + "memory.add_observations", + "memory.delete_relations", + "memory.delete_entities", + "pagerduty.pagerduty_acknowledge_incident", + "sqlite.sqlite_list_tables" + ], + "hit1": "memory.open_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 525, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "kind": "paraphrase", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "ranking": [ + "filesystem.get_file_info", + "filesystem.read_text_file", + "filesystem.read_file", + "filesystem.read_media_file", + "google_drive.gdrive_get_file", + "gitlab.gitlab_get_file", + "filesystem.list_directory", + "filesystem.list_allowed_directories", + "filesystem.directory_tree", + "filesystem.move_file" + ], + "hit1": "filesystem.get_file_info", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 220, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 635, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "kind": "paraphrase", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "ranking": [ + "filesystem.write_file", + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.create_directory", + "slack.delete_file", + "filesystem.read_media_file", + "slack.update_file", + "filesystem.edit_file", + "git.git_create_branch", + "slack.create_file" + ], + "hit1": "filesystem.write_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "kind": "verb_product", + "gold": [ + "github.search_issues" + ], + "alts": [], + "ranking": [ + "github.search_issues", + "github.search_users", + "github.search_pull_requests", + "github.search_orgs", + "jira.jira_search_issues", + "github.search_commits", + "linear.linear_search_issues", + "github.search_code", + "github.search_repositories", + "github.list_issues" + ], + "hit1": "github.search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 209, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "kind": "paraphrase", + "gold": [ + "github.get_me" + ], + "alts": [], + "ranking": [ + "github.get_teams", + "github.get_me", + "github.list_notifications", + "github.get_latest_release", + "github.search_users", + "github.get_runner", + "github.get_repository_tree", + "github.get_hook", + "github.get_environment", + "github.get_gist" + ], + "hit1": "github.get_teams", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 611, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "cost": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.54, + "p95_calls": 1.0, + "success_within_1": 0.9666666666666667, + "success_within_2": 0.9866666666666667, + "success_within_3": 0.9866666666666667, + "mean_tokens": 368.2, + "p95_tokens": 329.7499999999999, + "total_tokens": 55230, + "tokens_per_successful_query": 368.2, + "total_tokens_successful": 55230 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.32, + "p95_calls": 1.0, + "success_within_1": 0.98, + "success_within_2": 0.9866666666666667, + "success_within_3": 0.9866666666666667, + "mean_tokens": 503.08, + "p95_tokens": 500.9499999999998, + "total_tokens": 75462, + "tokens_per_successful_query": 503.08, + "total_tokens_successful": 75462 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.1533333333333333, + "p95_calls": 1.0, + "success_within_1": 0.9866666666666667, + "success_within_2": 0.9866666666666667, + "success_within_3": 0.9866666666666667, + "mean_tokens": 850.7866666666666, + "p95_tokens": 994.0999999999997, + "total_tokens": 127618, + "tokens_per_successful_query": 850.7866666666666, + "total_tokens_successful": 127618 + } + } +} diff --git a/experiments/mcp-tool-search/results/codedb_find.csv b/experiments/mcp-tool-search/results/codedb_find.csv new file mode 100644 index 00000000..ba986e28 --- /dev/null +++ b/experiments/mcp-tool-search/results/codedb_find.csv @@ -0,0 +1,151 @@ +id,query,kind,hit1,recall@1,recall@3,recall@5,mrr@10,ndcg@10,hit_rank,k3_calls,k3_tokens,k3_success1,k5_calls,k5_tokens,k5_success1,k10_calls,k10_tokens,k10_success1,ranking +q001,Create a GitHub issue,verb_product,github.create_issue,1.0,1.0,1.0,1.0000,0.7602,1,1,326,1.0,1,521,1.0,1,1013,1.0,github.create_issue github.create_hook github.create_gist github.create_runner github.create_branch github.create_package github.create_artifact github.create_milestone github.create_check_run gitlab.gitlab_create_issue +q002,Create a Jira ticket,verb_product,jira.create_board,0.0,0.0,0.0,0.1429,0.3333,7,3,963,0.0,2,1040,0.0,1,1020,1.0,jira.create_board jira.create_sprint jira.create_filter jira.create_version jira.create_webhook jira.create_attachment jira.jira_create_issue jira.jira_create_sprint gitlab.create_job notion.create_comment +q003,Send a Slack message,verb_product,memory.search_nodes,0.0,0.0,0.0,0.0000,0.0000,130,44,11040,0.0,26,10501,0.0,13,10241,0.0,memory.search_nodes slack.slack_get_thread_replies slack.slack_get_channel_history elasticsearch.es_get_doc elasticsearch.es_index_doc zendesk.zendesk_create_ticket zendesk.zendesk_update_ticket salesforce.salesforce_get_record salesforce.salesforce_convert_lead salesforce.salesforce_delete_record +q004,List open pull requests,resource,github.update_pull_request_branch,0.0,0.0,0.0,0.0000,0.0000,47,16,4347,0.0,10,4459,0.0,5,4359,0.0,github.update_pull_request_branch cloudflare.cf_list_dns pagerduty.pagerduty_list_oncalls prometheus.prom_list_metrics cloudflare.cf_list_zones kubernetes.k8s_list_deployments pagerduty.pagerduty_list_incidents github.list_code_scanning_alerts pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_acknowledge_incident +q005,Open a pull request on GitHub,verb_product,,0.0,0.0,0.0,0.0000,0.0000,36,12,3495,0.0,8,3776,0.0,4,3684,0.0, +q006,Run a SQL query against postgres,verb_product,gitlab.gitlab_list_merge_requests,0.0,0.0,0.0,0.0000,0.0000,103,35,8997,0.0,21,8689,0.0,11,8745,0.0,gitlab.gitlab_list_merge_requests github.actions_get_job_logs pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_oncalls github.get_secret_scanning_alert github.list_secret_scanning_alerts salesforce.salesforce_list_opportunities github.list_code_scanning_alerts google_calendar.gcal_list_calendars +q007,Read a local file from disk,paraphrase,filesystem.list_allowed_directories,0.0,0.0,0.0,0.0000,0.0000,77,26,7264,0.0,16,7172,0.0,8,6996,0.0,filesystem.list_allowed_directories +q008,What files are in this folder,paraphrase,github.list_dependabot_alerts,0.0,0.0,0.0,0.0000,0.0000,82,28,7711,0.0,17,7530,0.0,9,7616,0.0,github.list_dependabot_alerts github.list_code_scanning_alerts github.get_notification_details github.get_code_scanning_alert +q009,Remember this fact in the knowledge graph,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,89,30,8108,0.0,18,7832,0.0,9,7625,0.0, +q010,Search the web with Brave,verb_product,github.create_branch,0.0,0.0,0.0,0.0000,0.0000,124,42,10467,0.0,25,10040,0.0,13,10187,0.0,github.create_branch git.git_create_branch playwright.browser_tab_close playwright.browser_take_screenshot playwright.browser_click github.update_pull_request_branch github.list_dependabot_alerts playwright.browser_snapshot playwright.browser_wait_for github.list_repository_collaborators +q011,Fetch the contents of a URL,verb_product,cloudflare.cf_list_zones,0.0,0.0,0.0,0.0000,0.0000,110,37,9458,0.0,22,9076,0.0,11,8834,0.0,cloudflare.cf_list_zones cloudflare.cf_create_dns github.get_notification_details confluence.confluence_get_page confluence.confluence_create_page confluence.confluence_update_page salesforce.salesforce_convert_lead salesforce.salesforce_list_opportunities +q012,Show git status,identifier,github.search_commits,0.0,0.0,0.0,0.0000,0.0000,112,38,9466,0.0,23,9245,0.0,12,9359,0.0,github.search_commits github.search_repositories hubspot.hubspot_list_contacts playwright.browser_tab_list shopify.shopify_list_customers shopify.shopify_list_products playwright.browser_tab_close hubspot.hubspot_get_contact slack.slack_get_channel_history shopify.shopify_create_product +q013,Commit staged changes,paraphrase,discord.discord_list_channels,0.0,0.0,0.0,0.0000,0.0000,110,37,9257,0.0,22,8907,0.0,11,8698,0.0,discord.discord_list_channels github.list_code_scanning_alerts discord.discord_get_channel github.get_code_scanning_alert github.add_comment_to_pending_review +q014,Create a Linear issue,verb_product,linear.linear_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,303,1.0,1,461,1.0,1,820,1.0,linear.linear_create_issue linear.create_user linear.linear_get_issue linear.list_users linear.linear_list_issues linear.linear_update_issue linear.get_user linear.linear_assign_issue linear.update_user linear.linear_search_issues +q015,Make a task in my project tracker Linear,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,136,46,11712,0.0,28,11391,0.0,14,11055,0.0, +q016,Post a comment on a GitHub issue,verb_product,github.update_issue_comment,0.0,0.0,0.0,0.0000,0.0000,38,13,3791,0.0,8,3759,0.0,4,3663,0.0,github.update_issue_comment github.update_environment github.list_code_scanning_alerts github.get_code_scanning_alert +q017,Merge the pull request,ambiguous,github.merge_pull_request,1.0,1.0,1.0,1.0000,0.7602,1,1,365,1.0,1,560,1.0,1,988,1.0,github.merge_pull_request github.pull_request_read github.create_pull_request github.update_pull_request github.search_pull_requests github.pull_request_review_write github.update_pull_request_branch prometheus.prom_list_metrics pagerduty.pagerduty_list_oncalls pagerduty.pagerduty_list_incidents +q018,Trigger a GitHub Actions workflow,verb_product,,0.0,0.0,0.0,0.0000,0.0000,62,21,6123,0.0,13,6050,0.0,7,6293,0.0, +q019,Get Dependabot alerts,resource,github.get_dependabot_alert,1.0,1.0,1.0,1.0000,0.8597,1,1,240,1.0,1,389,1.0,1,671,1.0,github.get_dependabot_alert github.list_dependabot_alerts github.delete_environment google_calendar.gcal_list_events vercel.vercel_get_deployment notion.delete_data_source linear.update_comment github.update_environment vercel.vercel_list_deployments google_calendar.gcal_list_calendars +q020,List Sentry issues,verb_product,sentry.sentry_list_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,363,1.0,1,716,1.0,sentry.sentry_list_issues linear.list_users sentry.sentry_list_events sentry.sentry_list_projects sqlite.sqlite_query okta.okta_list_users github.list_runners gitlab.list_snippets slack.slack_list_users notion.list_data_sources +q021,Create a Stripe customer,verb_product,stripe.stripe_create_customer,1.0,1.0,1.0,1.0000,1.0000,1,1,293,1.0,1,450,1.0,1,892,1.0,stripe.stripe_create_customer slack.create_reminder slack.create_usergroup slack.create_bookmark slack.get_reminder slack.list_reminders slack.update_reminder github.create_repository gitlab.create_milestone github.create_milestone +q022,Refund a Stripe payment,paraphrase,pagerduty.pagerduty_get_incident,0.0,0.0,0.0,0.0000,0.0000,217,73,17304,0.0,44,16814,0.0,22,16374,0.0,pagerduty.pagerduty_get_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_list_oncalls +q023,Send an email with Gmail,verb_product,github.mark_all_notifications_read,0.0,0.0,0.0,0.0000,0.0000,245,82,19583,0.0,49,18819,0.0,25,18648,0.0,github.mark_all_notifications_read github.get_notification_details github.get_secret_scanning_alert github.list_code_scanning_alerts github.list_secret_scanning_alerts playwright.browser_navigate discord.discord_get_channel sequential_thinking.sequentialthinking pagerduty.pagerduty_acknowledge_incident +q024,Search my Gmail inbox,verb_product,github.search_commits,0.0,0.0,0.0,0.0000,0.0000,245,82,19506,0.0,49,18792,0.0,25,18630,0.0,github.search_commits +q025,Upload a file to Google Drive,paraphrase,github.list_code_scanning_alerts,0.0,0.0,0.0,0.0000,0.0000,253,85,20366,0.0,51,19618,0.0,26,19523,0.0,github.list_code_scanning_alerts playwright.browser_evaluate github.list_starred_repositories playwright.browser_tab_close salesforce.salesforce_delete_record github.mark_all_notifications_read +q026,Create a calendar event,resource,google_calendar.gcal_create_event,1.0,1.0,1.0,1.0000,1.0000,1,1,324,1.0,1,484,1.0,1,984,1.0,google_calendar.gcal_create_event google_calendar.gcal_list_events github.create_environment google_calendar.gcal_update_event google_calendar.gcal_delete_event slack.create_reminder jira.create_filter github.create_artifact github.create_runner slack.create_canvase +q027,List Kubernetes pods,verb_product,kubernetes.k8s_list_pods,1.0,1.0,1.0,1.0000,1.0000,1,1,198,1.0,1,299,1.0,1,630,1.0,kubernetes.k8s_list_pods kubernetes.k8s_list_deployments github.projects_list github.list_starred_repositories linear.linear_list_projects github.search_repositories github.delete_repository vercel.vercel_list_projects github.list_code_scanning_alerts cloudflare.cf_list_zones +q028,Get logs from a k8s pod,verb_product,docker.docker_images,0.0,0.0,0.0,0.0000,0.0000,193,65,15818,0.0,39,15194,0.0,20,15064,0.0,docker.docker_images +q029,Run a docker container,verb_product,slack.get_reminder,0.0,0.0,0.0,0.0000,0.0000,194,65,15640,0.0,39,15120,0.0,20,15019,0.0,slack.get_reminder slack.create_reminder notion.create_users_token slack.update_reminder slack.delete_reminder slack.list_reminders notion.update_users_token github.get_code_scanning_alert notion.create_data_source confluence.confluence_search +q030,Show docker container logs,verb_product,slack.slack_get_channel_history,0.0,0.0,0.0,0.0000,0.0000,184,62,14888,0.0,37,14342,0.0,19,14250,0.0,slack.slack_get_channel_history +q031,Navigate the browser to a URL,ambiguous,playwright.browser_evaluate,0.0,0.0,0.0,0.0000,0.0000,171,57,13984,0.0,35,13740,0.0,18,13672,0.0,playwright.browser_evaluate playwright.browser_click playwright.browser_tab_list playwright.browser_tab_close playwright.browser_fill_form github.list_code_scanning_alerts github.get_code_scanning_alert github.mark_all_notifications_read +q032,Take a screenshot of the page,ambiguous,github.create_or_update_file,0.0,0.0,0.0,0.0000,0.0000,171,57,14038,0.0,35,13740,0.0,18,13672,0.0,github.create_or_update_file +q033,Create a GitLab merge request,verb_product,gitlab.gitlab_create_merge_request,1.0,1.0,1.0,1.0000,1.0000,1,1,397,1.0,1,592,1.0,1,930,1.0,gitlab.gitlab_create_merge_request gitlab.create_milestone github.create_pull_request github.pull_request_read github.create_milestone github.create_repository github.get_release_by_tag github.search_pull_requests postgres.describe_table sqlite.sqlite_describe_table +q034,Search Notion pages,verb_product,notion.get_data_source,0.0,0.0,0.0,0.0000,0.0000,165,55,13537,0.0,33,13097,0.0,17,13110,0.0,notion.get_data_source notion.update_data_source notion.create_data_source notion.list_data_sources postgres.explain_query linear.get_comment hubspot.hubspot_search_contacts pagerduty.pagerduty_list_oncalls stripe.stripe_cancel_subscription linear.list_comments +q035,Create a Notion page,verb_product,notion.notion_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,430,1.0,1,773,1.0,notion.notion_create_page grafana.grafana_create_annotation jira.get_version notion.update_data_source datadog.datadog_create_monitor jira.update_version stripe.stripe_create_payment_intent hubspot.hubspot_create_contact linear.get_comment postgres.explain_query +q036,Query a Notion database,verb_product,notion.notion_query_database,1.0,1.0,1.0,1.0000,1.0000,1,1,233,1.0,1,416,1.0,1,752,1.0,notion.notion_query_database notion.update_data_source notion.get_data_source notion.list_data_sources notion.create_data_source notion.delete_data_source notion.update_users_token notion.notion_update_page github.actions_get memory.delete_observations +q037,Create a Zendesk ticket,verb_product,zendesk.zendesk_create_ticket,1.0,1.0,1.0,1.0000,1.0000,1,1,252,1.0,1,428,1.0,1,756,1.0,zendesk.zendesk_create_ticket zendesk.zendesk_update_ticket github.get_notification_details kubernetes.k8s_scale_deployment github.list_dependabot_alerts pagerduty.pagerduty_list_incidents +q038,Create a HubSpot contact,verb_product,hubspot.hubspot_create_contact,1.0,1.0,1.0,1.0000,1.0000,1,1,301,1.0,1,508,1.0,1,832,1.0,hubspot.hubspot_create_contact github.list_code_scanning_alerts github.get_code_scanning_alert github.add_comment_to_pending_review github.get_notification_details playwright.browser_take_screenshot +q039,Create a Salesforce lead,verb_product,salesforce.salesforce_create_lead,1.0,1.0,1.0,1.0000,1.0000,1,1,252,1.0,1,445,1.0,1,895,1.0,salesforce.salesforce_create_lead salesforce.salesforce_create_record salesforce.salesforce_convert_lead salesforce.salesforce_update_record notion.create_data_source salesforce.salesforce_query salesforce.salesforce_get_record salesforce.salesforce_delete_record linear.linear_create_issue gitlab.create_milestone +q040,Run a SOQL query,identifier,prometheus.prom_query_range,0.0,1.0,1.0,0.5000,0.6309,2,1,242,1.0,1,398,1.0,1,837,1.0,prometheus.prom_query_range salesforce.salesforce_query pagerduty.pagerduty_list_oncalls salesforce.salesforce_update_record pagerduty.pagerduty_resolve_incident sequential_thinking.sequentialthinking salesforce.salesforce_list_opportunities github.assign_copilot_to_issue github.list_dependabot_alerts github.actions_get_job_logs +q041,List Shopify orders,verb_product,shopify.shopify_list_orders,1.0,1.0,1.0,1.0000,1.0000,1,1,210,1.0,1,333,1.0,1,692,1.0,shopify.shopify_list_orders shopify.shopify_get_order shopify.shopify_create_order shopify.shopify_list_customers shopify.shopify_list_products shopify.shopify_create_product shopify.shopify_update_product playwright.browser_snapshot playwright.browser_take_screenshot github.fork_repository +q042,Create a PagerDuty incident,verb_product,pagerduty.pagerduty_create_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,403,1.0,1,823,1.0,pagerduty.pagerduty_create_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_acknowledge_incident jira.update_sprint github.list_dependabot_alerts google_calendar.gcal_delete_event linear.linear_update_issue google_calendar.gcal_update_event +q043,Who is on call in PagerDuty,paraphrase,github.get_code_scanning_alert,0.0,0.0,0.0,0.0000,0.0000,243,81,19511,0.0,49,18939,0.0,25,18705,0.0,github.get_code_scanning_alert github.list_code_scanning_alerts +q044,Query Datadog metrics,verb_product,datadog.datadog_query_metrics,1.0,1.0,1.0,1.0000,1.0000,1,1,277,1.0,1,403,1.0,1,835,1.0,datadog.datadog_query_metrics elasticsearch.es_get_doc elasticsearch.es_index_doc pagerduty.pagerduty_list_oncalls pagerduty.pagerduty_get_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_acknowledge_incident github.add_comment_to_pending_review pagerduty.pagerduty_create_incident google_calendar.gcal_list_calendars +q045,Search Datadog logs,verb_product,notion.get_data_source,0.0,0.0,0.0,0.0000,0.0000,241,81,19283,0.0,49,18785,0.0,25,18646,0.0,notion.get_data_source notion.list_data_sources pagerduty.pagerduty_list_oncalls google_calendar.gcal_list_calendars +q046,List S3 buckets,verb_product,aws_s3.s3_list_buckets,1.0,1.0,1.0,1.0000,1.0000,1,1,153,1.0,1,311,1.0,1,726,1.0,aws_s3.s3_list_buckets +q047,Upload an object to S3,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,296,99,23677,0.0,60,23014,0.0,30,22354,0.0, +q048,Purge Cloudflare cache,verb_product,cloudflare.cf_purge_cache,1.0,1.0,1.0,1.0000,1.0000,1,1,231,1.0,1,356,1.0,1,844,1.0,cloudflare.cf_purge_cache cloudflare.cf_create_dns cloudflare.cf_list_zones cloudflare.cf_list_dns slack.update_reminder salesforce.salesforce_update_record github.create_or_update_file notion.update_data_source clickup.clickup_create_task confluence.confluence_create_page +q049,Create a Cloudflare DNS record,verb_product,cloudflare.cf_create_dns,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,354,1.0,1,642,1.0,cloudflare.cf_create_dns cloudflare.cf_list_dns cloudflare.cf_list_zones discord.discord_add_reaction discord.discord_list_channels google_calendar.gcal_delete_event github.mark_all_notifications_read google_calendar.gcal_update_event google_calendar.gcal_list_calendars salesforce.salesforce_list_opportunities +q050,List Vercel deployments,verb_product,vercel.vercel_list_deployments,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,336,1.0,1,647,1.0,vercel.vercel_list_deployments kubernetes.k8s_list_deployments linear.list_comments linear.delete_comment linear.list_attachments memory.delete_entities linear.update_comment stripe.stripe_list_customers linear.delete_attachment pagerduty.pagerduty_list_incidents +q051,Insert a row into Supabase,verb_product,playwright.browser_type,0.0,0.0,0.0,0.0000,0.0000,309,103,24489,0.0,62,23669,0.0,31,22987,0.0,playwright.browser_type notion.update_users_token playwright.browser_tab_close playwright.browser_tab_new playwright.browser_tab_list playwright.browser_wait_for playwright.browser_evaluate playwright.browser_navigate playwright.browser_snapshot playwright.browser_click +q052,Export images from Figma,paraphrase,terraform.tf_state_list,0.0,0.0,0.0,0.0000,0.0000,316,106,25071,0.0,64,24328,0.0,32,23656,0.0,terraform.tf_state_list stripe.stripe_get_customer filesystem.list_allowed_directories filesystem.list_directory_with_sizes +q053,Send an SMS with Twilio,verb_product,github.list_code_scanning_alerts,0.0,0.0,0.0,0.0000,0.0000,316,106,25217,0.0,64,24398,0.0,32,23694,0.0,github.list_code_scanning_alerts github.mark_all_notifications_read sequential_thinking.sequentialthinking +q054,Create an Okta user,verb_product,notion.create_users_token,0.0,0.0,0.0,0.0000,0.0000,325,109,25800,0.0,65,24768,0.0,33,24366,0.0,notion.create_users_token hubspot.hubspot_create_contact slack.update_canvase playwright.browser_tab_new postgres.explain_query prometheus.prom_query_range memory.delete_observations kubernetes.k8s_scale_deployment playwright.browser_take_screenshot salesforce.salesforce_convert_lead +q055,Instant query Prometheus,verb_product,linear.list_comments,0.0,0.0,0.0,0.0000,0.0000,329,110,25880,0.0,66,25000,0.0,33,24340,0.0,linear.list_comments notion.create_users_token notion.delete_users_token github.list_environments stripe.stripe_create_customer github.update_environment playwright.browser_take_screenshot playwright.browser_tab_new hubspot.hubspot_search_contacts stripe.stripe_create_refund +q056,Search Grafana dashboards,verb_product,grafana.grafana_search_dashboards,1.0,1.0,1.0,1.0000,1.0000,1,1,225,1.0,1,342,1.0,1,699,1.0,grafana.grafana_search_dashboards google_calendar.gcal_list_calendars salesforce.salesforce_create_lead salesforce.salesforce_convert_lead pagerduty.pagerduty_list_oncalls +q057,Trigger a CircleCI pipeline,verb_product,slack.list_reminders,0.0,0.0,0.0,0.0000,0.0000,332,111,26129,0.0,67,25314,0.0,34,25005,0.0,slack.list_reminders stripe.stripe_create_invoice playwright.browser_click stripe.stripe_cancel_subscription stripe.stripe_create_price stripe.stripe_create_payment_intent stripe.stripe_create_customer stripe.stripe_create_product playwright.browser_tab_close github.get_secret_scanning_alert +q058,Run terraform plan,identifier,terraform.tf_plan,1.0,1.0,1.0,1.0000,1.0000,1,1,227,1.0,1,365,1.0,1,645,1.0,terraform.tf_plan stripe.stripe_create_refund linear.create_comment trello.trello_add_comment linear.linear_create_comment stripe.stripe_create_invoice github.mark_all_notifications_read salesforce.salesforce_list_opportunities stripe.stripe_create_payment_intent salesforce.salesforce_convert_lead +q059,Create an Asana task,verb_product,asana.asana_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,337,1.0,1,467,1.0,1,843,1.0,asana.asana_create_task slack.create_canvase gitlab.create_snippet slack.slack_create_channel slack.get_canvase slack.slack_add_reaction memory.create_relations slack.update_canvase notion.create_users_token slack.delete_canvase +q060,Create a Trello card,verb_product,trello.trello_create_card,1.0,1.0,1.0,1.0000,1.0000,1,1,245,1.0,1,394,1.0,1,895,1.0,trello.trello_create_card trello.trello_create_board trello.trello_list_cards memory.create_relations gitlab.create_release github.create_repository github.create_pull_request github.create_or_update_file stripe.stripe_create_product vercel.vercel_create_project +q061,Create a ClickUp task,verb_product,clickup.clickup_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,304,1.0,1,467,1.0,1,854,1.0,clickup.clickup_create_task clickup.clickup_update_task github.create_check_run slack.slack_create_channel gitlab.gitlab_create_issue slack.slack_get_users slack.slack_add_reaction bitbucket.bitbucket_create_issue slack.update_canvase slack.slack_list_users +q062,Create a Confluence page,verb_product,confluence.confluence_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,450,1.0,1,889,1.0,confluence.confluence_create_page confluence.confluence_update_page confluence.confluence_get_page cloudflare.cf_purge_cache google_calendar.gcal_create_event google_calendar.gcal_list_events notion.update_data_source github.actions_run_trigger google_calendar.gcal_update_event google_calendar.gcal_delete_event +q063,Send a Discord message,verb_product,discord.discord_send_message,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,371,1.0,1,752,1.0,discord.discord_send_message salesforce.salesforce_get_record filesystem.create_directory salesforce.salesforce_convert_lead github.list_code_scanning_alerts salesforce.salesforce_delete_record filesystem.list_allowed_directories salesforce.salesforce_update_record salesforce.salesforce_create_record salesforce.salesforce_create_lead +q064,Search Elasticsearch,verb_product,elasticsearch.es_search,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,377,1.0,1,700,1.0,elasticsearch.es_search elasticsearch.es_get_doc elasticsearch.es_index_doc slack.slack_search_messages notion.notion_search filesystem.search_files gmail.gmail_search_emails zendesk.zendesk_search linear.linear_search_issues jira.jira_search_issues +q065,Get a Redis key,verb_product,jira.jira_get_issue,0.0,0.0,0.0,0.0000,0.0000,291,97,22977,0.0,59,22447,0.0,30,22206,0.0,jira.jira_get_issue github.create_issue slack.get_reminder github.create_gist jira.get_sprint linear.get_user linear.linear_get_issue gitlab.get_release jira.get_filter slack.slack_get_thread_replies +q066,Find MongoDB documents,verb_product,,0.0,0.0,0.0,0.0000,0.0000,281,94,22397,0.0,57,21868,0.0,29,21583,0.0, +q067,List SQLite tables,verb_product,mysql.mysql_list_tables,0.0,1.0,1.0,0.3333,0.5000,3,1,151,1.0,1,302,1.0,1,651,1.0,mysql.mysql_list_tables postgres.list_tables sqlite.sqlite_list_tables gitlab.list_labels slack.list_files gitlab.list_releases gitlab.list_snippets gitlab.list_pipelines supabase.supabase_list_tables github.list_labels +q068,Describe a postgres table,verb_product,postgres.describe_table,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,358,1.0,1,677,1.0,postgres.describe_table postgres.list_tables slack.list_reminders github.list_starred_repositories github.search_repositories stripe.stripe_create_price supabase.supabase_list_tables elasticsearch.es_search stripe.stripe_create_invoice elasticsearch.es_index_doc +q069,github_create_issue,identifier,github.create_issue,1.0,1.0,1.0,1.0000,0.7602,1,1,324,1.0,1,520,1.0,1,998,1.0,github.create_issue github.create_gist github.create_runner github.create_milestone gitlab.gitlab_create_issue github.search_issues github.create_hook github.create_package github.list_issues github.create_artifact +q070,createIssue,identifier,github.create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,397,1.0,1,579,1.0,1,1039,1.0,github.create_issue jira.jira_create_issue linear.linear_create_issue gitlab.gitlab_create_issue bitbucket.bitbucket_create_issue linear.linear_update_issue sentry.sentry_update_issue linear.create_user jira.jira_update_issue slack.create_file +q071,list-directory,identifier,filesystem.list_directory,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,281,1.0,1,599,1.0,filesystem.list_directory filesystem.directory_tree filesystem.list_directory_with_sizes filesystem.create_directory filesystem.list_allowed_directories stripe.stripe_create_customer github.list_starred_repositories stripe.stripe_get_customer github.list_repository_collaborators github.star_repository +q072,sequential thinking next thought,paraphrase,github.get_secret_scanning_alert,0.0,0.0,0.0,0.0000,0.0000,116,39,9799,0.0,24,9727,0.0,12,9475,0.0,github.get_secret_scanning_alert github.list_secret_scanning_alerts github.get_code_scanning_alert github.get_notification_details github.list_code_scanning_alerts +q073,What time is it in Tokyo,paraphrase,github.actions_get,0.0,0.0,0.0,0.0000,0.0000,119,40,10114,0.0,24,9762,0.0,12,9498,0.0,github.actions_get github.update_issue_comment github.update_milestone github.create_milestone github.add_issue_comment github.assign_copilot_to_issue github.actions_get_job_logs github.actions_run_trigger github.dismiss_notification github.get_notification_details +q074,Convert time between timezones,verb_product,github.get_file_contents,0.0,0.0,0.0,0.0000,0.0000,122,41,10273,0.0,25,10083,0.0,13,10227,0.0,github.get_file_contents notion.get_users_token notion.delete_users_token notion.create_users_token github.get_discussion_comments stripe.stripe_cancel_subscription github.add_comment_to_pending_review github.list_secret_scanning_alerts +q075,Star a GitHub repo,verb_product,github.star_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,227,1.0,1,349,1.0,1,796,1.0,github.star_repository github.unstar_repository github.search_code github.list_releases github.search_users github.create_repository github.label_write github.create_runner github.update_runner github.search_repositories +q076,Fork a repository,resource,playwright.browser_wait_for,0.0,0.0,0.0,0.0000,0.0000,30,10,2448,0.0,6,2372,0.0,3,2315,0.0,playwright.browser_wait_for slack.list_bookmarks playwright.browser_fill_form shopify.shopify_create_order slack.create_bookmark filesystem.create_directory terraform.tf_state_list playwright.browser_click salesforce.salesforce_list_opportunities salesforce.salesforce_get_record +q077,Get the latest GitHub release,verb_product,github.get_latest_release,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,416,1.0,1,810,1.0,github.get_latest_release github.list_releases gitlab.get_release github.get_release_by_tag github.create_milestone github.get_milestone github.create_issue gitlab.delete_release gitlab.create_release github.delete_milestone +q078,Create a gist,resource,github.create_gist,1.0,1.0,1.0,1.0000,1.0000,1,1,290,1.0,1,496,1.0,1,1021,1.0,github.create_gist gitlab.create_job github.create_hook gitlab.create_wiki gitlab.create_snippet gitlab.create_label github.create_issue github.create_runner github.create_branch github.create_artifact +q079,Search GitHub code,verb_product,github.search_code,1.0,1.0,1.0,1.0000,1.0000,1,1,277,1.0,1,369,1.0,1,783,1.0,github.search_code github.search_repositories github.list_code_scanning_alerts +q080,List GitHub notifications,verb_product,github.list_notifications,1.0,1.0,1.0,1.0000,1.0000,1,1,175,1.0,1,325,1.0,1,643,1.0,github.list_notifications github.dismiss_notification github.mark_all_notifications_read github.list_discussions github.list_repository_collaborators stripe.stripe_list_subscriptions stripe.stripe_cancel_subscription sequential_thinking.sequentialthinking +q081,Assign Copilot to an issue,verb_product,github.assign_copilot_to_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,312,1.0,1,418,1.0,1,824,1.0,github.assign_copilot_to_issue github.list_code_scanning_alerts discord.discord_add_reaction salesforce.salesforce_list_opportunities +q082,Get secret scanning alerts,resource,github.list_secret_scanning_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,295,1.0,1,465,1.0,1,884,1.0,github.list_secret_scanning_alerts github.get_secret_scanning_alert github.list_code_scanning_alerts github.get_code_scanning_alert slack.slack_get_channel_history slack.slack_set_channel_topic sentry.sentry_list_events google_calendar.gcal_list_events github.create_environment slack.slack_add_reaction +q083,Create or update a file on GitHub,verb_product,github.get_code_scanning_alert,0.0,0.0,0.0,0.0000,0.0000,20,7,1850,0.0,4,1698,0.0,2,1650,0.0,github.get_code_scanning_alert github.mark_all_notifications_read github.list_code_scanning_alerts +q084,List collaborators on a repo,resource,cloudflare.cf_list_zones,0.0,0.0,0.0,0.0000,0.0000,33,11,2773,0.0,7,2927,0.0,4,3392,0.0,cloudflare.cf_list_zones memory.delete_observations google_calendar.gcal_list_events google_calendar.gcal_list_calendars salesforce.salesforce_list_opportunities playwright.browser_take_screenshot +q085,Reply in a Slack thread,verb_product,slack.get_reminder,0.0,0.0,0.0,0.0000,0.0000,130,44,11132,0.0,26,10591,0.0,13,10318,0.0,slack.get_reminder slack.list_reminders slack.update_reminder slack.create_reminder slack.delete_reminder gitlab.create_pipeline postgres.explain_query github.list_branches github.create_check_run gitlab.create_snippet +q086,Add an emoji reaction on Slack,verb_product,memory.search_nodes,0.0,0.0,0.0,0.0000,0.0000,125,42,10605,0.0,25,10158,0.0,13,10233,0.0,memory.search_nodes pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident +q087,Search Slack messages,verb_product,slack.slack_search_messages,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,349,1.0,1,742,1.0,slack.slack_search_messages memory.search_nodes elasticsearch.es_search filesystem.search_files elasticsearch.es_get_doc github.search_issues elasticsearch.es_index_doc slack.slack_get_channel_history jira.jira_search_issues slack.slack_get_thread_replies +q088,Create a Slack channel,verb_product,slack.slack_create_channel,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,357,1.0,1,686,1.0,slack.slack_create_channel slack.slack_leave_channel slack.slack_list_channels slack.slack_set_channel_topic slack.slack_get_channel_history slack.slack_get_thread_replies discord.discord_list_channels brave_search.brave_local_search salesforce.salesforce_convert_lead google_calendar.gcal_delete_event +q089,Transition a Jira issue,verb_product,jira.jira_transition_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,355,1.0,1,740,1.0,jira.jira_transition_issue jira.list_versions jira.list_sprints jira.update_version stripe.stripe_cancel_subscription github.mark_all_notifications_read discord.discord_add_reaction notion.create_users_token stripe.stripe_create_invoice gitlab.create_milestone +q090,Search Jira with JQL,identifier,jira.list_attachments,0.0,0.0,0.0,0.0000,0.0000,151,51,12663,0.0,31,12344,0.0,16,12438,0.0,jira.list_attachments jira.delete_attachment +q091,Add a worklog to Jira,verb_product,,0.0,0.0,0.0,0.0000,0.0000,155,52,12978,0.0,31,12447,0.0,16,12448,0.0, +q092,List Linear teams,verb_product,linear.linear_list_teams,1.0,1.0,1.0,1.0000,1.0000,1,1,199,1.0,1,345,1.0,1,684,1.0,linear.linear_list_teams linear.list_users linear.list_comments linear.list_cycles linear.list_labels linear.list_attachments linear.get_user linear.create_user linear.delete_user linear.update_user +q093,Comment on a Linear issue,verb_product,linear.list_comments,0.0,0.0,0.0,0.0000,0.0000,146,49,12189,0.0,30,11992,0.0,15,11677,0.0,linear.list_comments linear.update_comment github.create_environment github.list_code_scanning_alerts github.add_comment_to_pending_review github.get_code_scanning_alert salesforce.salesforce_convert_lead salesforce.salesforce_list_opportunities +q094,Move a local file,paraphrase,gitlab.create_milestone,0.0,0.0,0.0,0.0000,0.0000,95,32,8648,0.0,19,8320,0.0,10,8418,0.0,gitlab.create_milestone gmail.gmail_modify_email gitlab.list_milestones google_drive.gdrive_create_file gitlab.get_milestone github.create_milestone filesystem.list_allowed_directories salesforce.salesforce_convert_lead gitlab.delete_milestone gitlab.update_milestone +q095,Recursively search files by pattern,paraphrase,github.list_secret_scanning_alerts,0.0,0.0,0.0,0.0000,0.0000,88,30,8094,0.0,18,7830,0.0,9,7632,0.0,github.list_secret_scanning_alerts stripe.stripe_cancel_subscription github.get_secret_scanning_alert stripe.stripe_list_subscriptions +q096,Edit a file in place with old and new string,paraphrase,github.list_code_scanning_alerts,0.0,0.0,0.0,0.0000,0.0000,81,27,7623,0.0,17,7586,0.0,9,7650,0.0,github.list_code_scanning_alerts discord.discord_list_channels +q097,Read the whole knowledge graph,paraphrase,pagerduty.pagerduty_acknowledge_incident,0.0,0.0,0.0,0.0000,0.0000,105,35,9234,0.0,21,8940,0.0,11,9003,0.0,pagerduty.pagerduty_acknowledge_incident google_drive.gdrive_create_file google_drive.gdrive_share_file google_drive.gdrive_update_file google_calendar.gcal_create_event google_calendar.gcal_list_events google_drive.gdrive_get_file google_calendar.gcal_update_event google_calendar.gcal_delete_event google_drive.gdrive_list_folder +q098,Search memory nodes,verb_product,memory.search_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,348,1.0,1,728,1.0,memory.search_nodes github.search_repositories confluence.confluence_search notion.delete_data_source notion.update_data_source notion.get_data_source grafana.grafana_search_dashboards slack.create_reminder notion.create_data_source salesforce.salesforce_query +q099,Show unstaged git diff,verb_product,,0.0,0.0,0.0,0.0000,0.0000,105,35,8947,0.0,21,8653,0.0,11,8735,0.0, +q100,Stage files with git add,identifier,filesystem.list_allowed_directories,0.0,0.0,0.0,0.0000,0.0000,111,37,9333,0.0,23,9295,0.0,12,9410,0.0,filesystem.list_allowed_directories github.pull_request_read github.list_dependabot_alerts github.list_starred_repositories github.create_or_update_file github.get_notification_details discord.discord_list_channels github.get_dependabot_alert jira.delete_attachment github.mark_all_notifications_read +q101,Create a Shopify product,verb_product,shopify.shopify_create_product,1.0,1.0,1.0,1.0000,1.0000,1,1,286,1.0,1,414,1.0,1,820,1.0,shopify.shopify_create_product hubspot.hubspot_create_deal hubspot.hubspot_create_contact playwright.browser_take_screenshot +q102,Cancel a Stripe subscription,verb_product,stripe.stripe_cancel_subscription,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,400,1.0,1,809,1.0,stripe.stripe_cancel_subscription stripe.stripe_create_invoice slack.slack_add_reaction stripe.stripe_create_payment_intent github.create_milestone hubspot.hubspot_create_contact github.create_environment github.list_secret_scanning_alerts github.get_secret_scanning_alert salesforce.salesforce_list_opportunities +q103,Create a payment intent,resource,notion.create_comment,0.0,1.0,1.0,0.3333,0.5000,3,1,299,1.0,1,489,1.0,1,779,1.0,notion.create_comment linear.create_comment stripe.stripe_create_payment_intent linear.create_attachment jira.create_attachment linear.get_attachment linear.update_attachment jira.get_attachment stripe.stripe_get_payment_intent linear.update_comment +q104,Acknowledge a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,185,1.0,1,343,1.0,1,758,1.0,pagerduty.pagerduty_acknowledge_incident +q105,Create a Datadog monitor,verb_product,datadog.datadog_create_monitor,1.0,1.0,1.0,1.0000,1.0000,1,1,296,1.0,1,423,1.0,1,774,1.0,datadog.datadog_create_monitor notion.create_data_source notion.get_data_source notion.update_data_source notion.delete_data_source github.add_comment_to_pending_review google_calendar.gcal_list_calendars +q106,Resolve a Sentry issue,paraphrase,postgres.explain_query,0.0,0.0,0.0,0.0000,0.0000,232,78,18609,0.0,47,18021,0.0,24,17953,0.0,postgres.explain_query postgres.list_schemas slack.slack_get_user_profile github.search_repositories salesforce.salesforce_query github.list_starred_repositories stripe.stripe_create_customer github.list_secret_scanning_alerts postgres.list_tables notion.create_data_source +q107,Scale a Kubernetes deployment,verb_product,kubernetes.k8s_scale_deployment,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,395,1.0,1,804,1.0,kubernetes.k8s_scale_deployment gitlab.delete_snippet github.create_environment memory.create_entities github.list_dependabot_alerts gitlab.delete_milestone github.update_pull_request github.get_dependabot_alert linear.linear_create_comment playwright.browser_navigate +q108,Apply a Kubernetes manifest,verb_product,gitlab.create_snippet,0.0,0.0,0.0,0.0000,0.0000,200,67,16099,0.0,40,15471,0.0,20,15071,0.0,gitlab.create_snippet gitlab.delete_snippet github.create_milestone memory.create_entities linear.linear_list_teams linear.linear_list_projects gitlab.create_milestone linear.linear_list_issues gitlab.delete_milestone github.update_milestone +q109,Stop a docker container,verb_product,slack.get_reminder,0.0,0.0,0.0,0.0000,0.0000,196,66,15806,0.0,40,15415,0.0,20,15015,0.0,slack.get_reminder notion.create_users_token okta.okta_list_users notion.update_users_token notion.get_data_source slack.create_reminder slack.update_reminder slack.delete_reminder slack.list_reminders notion.create_data_source +q110,Build a docker image,verb_product,github.update_package,0.0,0.0,0.0,0.0000,0.0000,195,65,15591,0.0,39,15071,0.0,20,15055,0.0,github.update_package github.update_check_run notion.list_data_sources slack.slack_get_user_profile github.list_dependabot_alerts notion.delete_data_source github.list_code_scanning_alerts bitbucket.bitbucket_create_pr bitbucket.bitbucket_create_issue github.get_code_scanning_alert +q111,Click an element in Playwright,verb_product,linear.get_attachment,0.0,0.0,0.0,0.0000,0.0000,175,59,14528,0.0,35,13874,0.0,18,13763,0.0,linear.get_attachment linear.list_attachments playwright.browser_take_screenshot pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_create_incident +q112,Type text into a browser input,paraphrase,github.unstar_repository,0.0,0.0,0.0,0.0000,0.0000,169,57,14038,0.0,34,13458,0.0,17,13084,0.0,github.unstar_repository github.star_repository github.create_repository github.search_repositories github.list_starred_repositories github.list_repository_collaborators +q113,List GitLab projects,verb_product,gitlab.gitlab_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,337,1.0,1,653,1.0,gitlab.gitlab_list_projects github.projects_list github.projects_get github.projects_write postgres.list_tables github.list_dependabot_alerts github.list_pull_requests github.list_environments postgres.list_schemas playwright.browser_tab_list +q114,Create a Bitbucket pull request,verb_product,github.create_pull_request,0.0,0.0,0.0,0.0000,0.0000,272,91,21988,0.0,55,21340,0.0,28,20983,0.0,github.create_pull_request github.pull_request_read github.search_pull_requests pagerduty.pagerduty_acknowledge_incident +q115,Create a Gmail draft,verb_product,gmail.gmail_create_draft,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,494,1.0,1,1037,1.0,gmail.gmail_create_draft jira.create_filter gitlab.create_job slack.create_file gitlab.create_wiki github.create_artifact gitlab.create_label gitlab.create_milestone github.create_milestone github.create_issue +q116,Share a Google Drive file,verb_product,google_drive.gdrive_share_file,1.0,1.0,1.0,1.0000,1.0000,1,1,316,1.0,1,409,1.0,1,733,1.0,google_drive.gdrive_share_file postgres.describe_table github.create_or_update_file slack.slack_get_user_profile google_calendar.gcal_list_calendars postgres.list_tables github.list_starred_repositories discord.discord_list_channels notion.list_data_sources gitlab.gitlab_create_issue +q117,List Google Calendar events,verb_product,google_calendar.gcal_list_events,1.0,1.0,1.0,1.0000,1.0000,1,1,202,1.0,1,366,1.0,1,816,1.0,google_calendar.gcal_list_events google_calendar.gcal_list_calendars google_drive.gdrive_search github.list_dependabot_alerts google_drive.gdrive_share_file github.list_code_scanning_alerts github.list_secret_scanning_alerts pagerduty.pagerduty_list_incidents github.get_secret_scanning_alert github.get_code_scanning_alert +q118,Delete a calendar event,resource,google_calendar.gcal_delete_event,1.0,1.0,1.0,1.0000,1.0000,1,1,301,1.0,1,440,1.0,1,742,1.0,google_calendar.gcal_delete_event google_calendar.gcal_list_events google_calendar.gcal_create_event google_calendar.gcal_update_event jira.delete_board slack.delete_reminder slack.delete_canvase slack.delete_bookmark notion.delete_data_source github.list_dependabot_alerts +q119,List Vercel projects,verb_product,vercel.vercel_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,156,1.0,1,296,1.0,1,614,1.0,vercel.vercel_list_projects linear.linear_list_projects sentry.sentry_list_projects vercel.vercel_list_deployments linear.linear_create_project linear.list_comments stripe.stripe_list_customers playwright.browser_tab_close trello.trello_list_cards circleci.circleci_list_pipelines +q120,Select rows from Supabase,verb_product,salesforce.salesforce_list_opportunities,0.0,0.0,0.0,0.0000,0.0000,307,103,24397,0.0,62,23613,0.0,31,22962,0.0,salesforce.salesforce_list_opportunities +q121,Post a Figma comment,verb_product,figma.figma_post_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,289,1.0,1,381,1.0,1,817,1.0,figma.figma_post_comment github.get_code_scanning_alert github.list_code_scanning_alerts +q122,List Okta users,verb_product,okta.okta_list_users,1.0,1.0,1.0,1.0000,1.0000,1,1,233,1.0,1,361,1.0,1,656,1.0,okta.okta_list_users slack.list_canvases github.list_dependabot_alerts postgres.list_tables slack.list_bookmarks playwright.browser_tab_list notion.update_users_token notion.create_users_token terraform.tf_state_list postgres.list_schemas +q123,Range query Prometheus,verb_product,prometheus.prom_query_range,1.0,1.0,1.0,1.0000,1.0000,1,1,266,1.0,1,472,1.0,1,835,1.0,prometheus.prom_query_range linear.create_user okta.okta_create_user slack.create_usergroup notion.create_users_token github.projects_get stripe.stripe_create_customer github.list_environments github.create_check_run playwright.browser_tab_new +q124,Create a Grafana annotation,verb_product,grafana.grafana_create_annotation,1.0,1.0,1.0,1.0000,1.0000,1,1,197,1.0,1,371,1.0,1,728,1.0,grafana.grafana_create_annotation github.mark_all_notifications_read github.get_notification_details pagerduty.pagerduty_create_incident pagerduty.pagerduty_acknowledge_incident +q125,Apply terraform,identifier,terraform.tf_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,171,1.0,1,287,1.0,1,681,1.0,terraform.tf_apply terraform.tf_plan terraform.tf_state_list playwright.browser_wait_for playwright.browser_fill_form trello.trello_create_board stripe.stripe_create_price trello.trello_add_comment github.search_repositories linear.create_comment +q126,List Asana tasks,verb_product,asana.asana_list_tasks,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,395,1.0,1,767,1.0,asana.asana_list_tasks slack.list_canvases slack.list_pins gitlab.list_snippets slack.slack_list_users slack.list_bookmarks slack.slack_list_channels jira.list_attachments github.list_packages gitlab.list_labels +q127,Comment on a Trello card,verb_product,linear.delete_comment,0.0,0.0,0.0,0.0000,0.0000,345,115,27183,0.0,69,26171,0.0,35,25769,0.0,linear.delete_comment github.create_environment kubernetes.k8s_scale_deployment salesforce.salesforce_convert_lead salesforce.salesforce_list_opportunities +q128,Update a ClickUp task,verb_product,clickup.clickup_update_task,1.0,1.0,1.0,1.0000,1.0000,1,1,277,1.0,1,416,1.0,1,840,1.0,clickup.clickup_update_task clickup.clickup_create_task slack.update_canvase slack.update_bookmark github.update_check_run slack.slack_upload_file playwright.browser_click github.list_packages github.create_check_run github.assign_copilot_to_issue +q129,Search Confluence,verb_product,confluence.confluence_search,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,350,1.0,1,757,1.0,confluence.confluence_search cloudflare.cf_purge_cache notion.list_data_sources notion.update_data_source notion.delete_data_source google_calendar.gcal_create_event google_calendar.gcal_list_events google_calendar.gcal_update_event google_calendar.gcal_delete_event google_calendar.gcal_list_calendars +q130,Create a HubSpot deal,verb_product,hubspot.hubspot_create_deal,1.0,1.0,1.0,1.0000,1.0000,1,1,295,1.0,1,536,1.0,1,982,1.0,hubspot.hubspot_create_deal github.create_milestone github.list_dependabot_alerts github.create_pull_request github.get_dependabot_alert github.create_or_update_file github.search_pull_requests github.list_code_scanning_alerts github.get_code_scanning_alert github.get_notification_details +q131,List Zendesk tickets,verb_product,zendesk.zendesk_list_tickets,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,316,1.0,1,644,1.0,zendesk.zendesk_list_tickets kubernetes.k8s_list_services pagerduty.pagerduty_list_incidents kubernetes.k8s_list_deployments notion.delete_users_token github.list_dependabot_alerts +q132,List Discord channels,verb_product,discord.discord_list_channels,1.0,1.0,1.0,1.0000,1.0000,1,1,215,1.0,1,291,1.0,1,596,1.0,discord.discord_list_channels discord.discord_get_channel github.list_code_scanning_alerts google_calendar.gcal_list_calendars github.mark_all_notifications_read salesforce.salesforce_convert_lead +q133,Set a Redis key,verb_product,slack.get_reminder,0.0,0.0,0.0,0.0000,0.0000,293,98,23335,0.0,59,22602,0.0,30,22356,0.0,slack.get_reminder slack.create_pin sentry.sentry_update_issue slack.create_file postgres.list_tables slack.slack_get_thread_replies postgres.describe_table slack.delete_reminder slack.list_reminders slack.create_reminder +q134,Insert MongoDB documents,verb_product,mongodb.mongo_list_collections,0.0,0.0,0.0,0.0000,0.0000,283,95,22628,0.0,57,21868,0.0,29,21583,0.0,mongodb.mongo_list_collections +q135,Run a MySQL query,verb_product,prometheus.prom_query_range,0.0,0.0,0.0,0.0000,0.0000,280,94,22516,0.0,56,21595,0.0,28,21007,0.0,prometheus.prom_query_range memory.search_nodes memory.delete_relations memory.add_observations memory.create_relations pagerduty.pagerduty_list_oncalls memory.create_entities memory.delete_observations pagerduty.pagerduty_resolve_incident sequential_thinking.sequentialthinking +q136,Index a document in Elasticsearch,verb_product,google_calendar.gcal_list_calendars,0.0,0.0,0.0,0.0000,0.0000,291,97,23125,0.0,59,22581,0.0,30,22324,0.0,google_calendar.gcal_list_calendars +q137,Get an S3 object,verb_product,,0.0,0.0,0.0,0.0000,0.0000,295,99,23578,0.0,59,22581,0.0,30,22324,0.0, +q138,List Cloudflare zones,verb_product,cloudflare.cf_list_zones,1.0,1.0,1.0,1.0000,1.0000,1,1,161,1.0,1,321,1.0,1,613,1.0,cloudflare.cf_list_zones cloudflare.cf_list_dns discord.discord_list_channels google_calendar.gcal_list_events notion.update_users_token salesforce.salesforce_update_record google_calendar.gcal_list_calendars salesforce.salesforce_list_opportunities salesforce.salesforce_delete_record github.mark_all_notifications_read +q139,Create a Stripe invoice,verb_product,stripe.stripe_create_invoice,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,401,1.0,1,770,1.0,stripe.stripe_create_invoice notion.notion_create_page github.create_environment shopify.shopify_create_order linear.linear_create_project shopify.shopify_create_product discord.discord_add_reaction playwright.browser_navigate jira.jira_transition_issue notion.update_data_source +q140,List GitHub Actions workflows,verb_product,,0.0,0.0,0.0,0.0000,0.0000,60,20,5782,0.0,12,5614,0.0,6,5488,0.0, +q141,Get GitHub Actions job logs,verb_product,github.actions_get_job_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,366,1.0,1,781,1.0,github.actions_get_job_logs +q142,Create a GitHub repository,verb_product,github.create_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,297,1.0,1,502,1.0,1,1026,1.0,github.create_repository github.search_repositories github.create_milestone github.list_starred_repositories github.create_pull_request github.list_repository_collaborators gitlab.create_snippet github.create_or_update_file github.pull_request_read gitlab.create_milestone +q143,Checkout a git branch,verb_product,github.create_check_run,0.0,0.0,0.0,0.0000,0.0000,117,39,9806,0.0,24,9724,0.0,12,9484,0.0,github.create_check_run github.update_check_run github.search_commits github.search_repositories github.search_pull_requests playwright.browser_take_screenshot pagerduty.pagerduty_acknowledge_incident +q144,Show git log,identifier,github.search_orgs,0.0,0.0,0.0,0.0000,0.0000,119,40,9928,0.0,24,9624,0.0,12,9396,0.0,github.search_orgs github.list_hooks shopify.shopify_get_order shopify.shopify_list_orders shopify.shopify_get_product jira.list_webhooks github.request_copilot_review playwright.browser_navigate shopify.shopify_list_products playwright.browser_wait_for +q145,List postgres schemas,verb_product,postgres.list_schemas,1.0,1.0,1.0,1.0000,1.0000,1,1,207,1.0,1,364,1.0,1,668,1.0,postgres.list_schemas slack.list_reminders github.list_branches github.search_repositories elasticsearch.es_search elasticsearch.es_get_doc discord.discord_list_channels stripe.stripe_create_customer filesystem.search_files discord.discord_list_members +q146,Open nodes in the knowledge graph,verb_product,pagerduty.pagerduty_acknowledge_incident,0.0,0.0,0.0,0.0000,0.0000,100,34,8817,0.0,20,8407,0.0,10,8187,0.0,pagerduty.pagerduty_acknowledge_incident github.add_comment_to_pending_review google_calendar.gcal_list_events google_calendar.gcal_list_calendars +q147,Get file metadata from the filesystem,paraphrase,filesystem.read_media_file,0.0,0.0,0.0,0.0000,0.0000,90,30,8177,0.0,18,7913,0.0,9,7715,0.0,filesystem.read_media_file filesystem.create_directory filesystem.list_allowed_directories salesforce.salesforce_create_lead salesforce.salesforce_update_record +q148,Write a new local file,paraphrase,github.label_write,0.0,0.0,0.0,0.0000,0.0000,88,30,8232,0.0,18,7980,0.0,9,7791,0.0,github.label_write linear.list_users linear.list_labels linear.list_cycles linear.create_label linear.list_comments linear.get_label linear.create_cycle gitlab.create_milestone linear.linear_list_teams +q149,Search GitHub issues,verb_product,github.search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,352,1.0,1,712,1.0,github.search_issues github.search_users github.search_orgs github.search_commits github.search_repositories github.search_pull_requests github.list_branches github.list_secret_scanning_alerts hubspot.hubspot_search_contacts github.list_code_scanning_alerts +q150,Get the authenticated GitHub user,paraphrase,hubspot.hubspot_create_deal,0.0,0.0,0.0,0.0000,0.0000,10,4,1056,0.0,2,907,0.0,1,885,1.0,hubspot.hubspot_create_deal hubspot.hubspot_get_contact github.mark_all_notifications_read github.add_comment_to_pending_review hubspot.hubspot_search_contacts hubspot.hubspot_create_contact sequential_thinking.sequentialthinking bitbucket.bitbucket_create_pr bitbucket.bitbucket_create_issue diff --git a/experiments/mcp-tool-search/results/codedb_find.json b/experiments/mcp-tool-search/results/codedb_find.json new file mode 100644 index 00000000..3ea7d665 --- /dev/null +++ b/experiments/mcp-tool-search/results/codedb_find.json @@ -0,0 +1,6607 @@ +{ + "summary": { + "n": 150, + "index_s": 0.44171834199994464, + "rss_mb": null, + "p50_ms": 10.254846500004078, + "p95_ms": 10.366095449984414, + "recall@1": 0.4866666666666667, + "recall@1_lo": 0.4, + "recall@1_hi": 0.5666666666666667, + "recall@3": 0.5066666666666667, + "recall@3_lo": 0.4266666666666667, + "recall@3_hi": 0.5866666666666667, + "recall@5": 0.5066666666666667, + "recall@5_lo": 0.4266666666666667, + "recall@5_hi": 0.5866666666666667, + "mrr@10": 0.4953968253968254, + "mrr@10_lo": 0.413015873015873, + "mrr@10_hi": 0.5765079365079365, + "ndcg@10": 0.49403029591368397, + "ndcg@10_lo": 0.41143399382506873, + "ndcg@10_hi": 0.5737920498263825 + }, + "meta": { + "name": "codedb_find", + "needs_model": false, + "needs_api": false, + "notes": "codedb_find fuzzy filename search (server__name.py files)" + }, + "rows": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "github.create_hook", + "github.create_gist", + "github.create_runner", + "github.create_branch", + "github.create_package", + "github.create_artifact", + "github.create_milestone", + "github.create_check_run", + "gitlab.gitlab_create_issue" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 326, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 521, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1013, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "kind": "verb_product", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "ranking": [ + "jira.create_board", + "jira.create_sprint", + "jira.create_filter", + "jira.create_version", + "jira.create_webhook", + "jira.create_attachment", + "jira.jira_create_issue", + "jira.jira_create_sprint", + "gitlab.create_job", + "notion.create_comment" + ], + "hit1": "jira.create_board", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.14285714285714285, + "ndcg@10": 0.3333333333333333, + "hit_rank": 7, + "k3_calls": 3, + "k3_tokens": 963, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 1040, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1020, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "kind": "verb_product", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "slack.slack_get_thread_replies", + "slack.slack_get_channel_history", + "elasticsearch.es_get_doc", + "elasticsearch.es_index_doc", + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_update_ticket", + "salesforce.salesforce_get_record", + "salesforce.salesforce_convert_lead", + "salesforce.salesforce_delete_record" + ], + "hit1": "memory.search_nodes", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 130, + "k3_calls": 44, + "k3_tokens": 11040, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 26, + "k5_tokens": 10501, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 13, + "k10_tokens": 10241, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "kind": "resource", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "ranking": [ + "github.update_pull_request_branch", + "cloudflare.cf_list_dns", + "pagerduty.pagerduty_list_oncalls", + "prometheus.prom_list_metrics", + "cloudflare.cf_list_zones", + "kubernetes.k8s_list_deployments", + "pagerduty.pagerduty_list_incidents", + "github.list_code_scanning_alerts", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_acknowledge_incident" + ], + "hit1": "github.update_pull_request_branch", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 47, + "k3_calls": 16, + "k3_tokens": 4347, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 10, + "k5_tokens": 4459, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 5, + "k10_tokens": 4359, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 36, + "k3_calls": 12, + "k3_tokens": 3495, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 8, + "k5_tokens": 3776, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3684, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "kind": "verb_product", + "gold": [ + "postgres.query" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_list_merge_requests", + "github.actions_get_job_logs", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_oncalls", + "github.get_secret_scanning_alert", + "github.list_secret_scanning_alerts", + "salesforce.salesforce_list_opportunities", + "github.list_code_scanning_alerts", + "google_calendar.gcal_list_calendars" + ], + "hit1": "gitlab.gitlab_list_merge_requests", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 103, + "k3_calls": 35, + "k3_tokens": 8997, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 21, + "k5_tokens": 8689, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 11, + "k10_tokens": 8745, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "kind": "paraphrase", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "ranking": [ + "filesystem.list_allowed_directories" + ], + "hit1": "filesystem.list_allowed_directories", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 77, + "k3_calls": 26, + "k3_tokens": 7264, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 16, + "k5_tokens": 7172, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 8, + "k10_tokens": 6996, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "kind": "paraphrase", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "ranking": [ + "github.list_dependabot_alerts", + "github.list_code_scanning_alerts", + "github.get_notification_details", + "github.get_code_scanning_alert" + ], + "hit1": "github.list_dependabot_alerts", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 82, + "k3_calls": 28, + "k3_tokens": 7711, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 17, + "k5_tokens": 7530, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7616, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 89, + "k3_calls": 30, + "k3_tokens": 8108, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7832, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7625, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "kind": "verb_product", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "ranking": [ + "github.create_branch", + "git.git_create_branch", + "playwright.browser_tab_close", + "playwright.browser_take_screenshot", + "playwright.browser_click", + "github.update_pull_request_branch", + "github.list_dependabot_alerts", + "playwright.browser_snapshot", + "playwright.browser_wait_for", + "github.list_repository_collaborators" + ], + "hit1": "github.create_branch", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 124, + "k3_calls": 42, + "k3_tokens": 10467, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 25, + "k5_tokens": 10040, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 13, + "k10_tokens": 10187, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "kind": "verb_product", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones", + "cloudflare.cf_create_dns", + "github.get_notification_details", + "confluence.confluence_get_page", + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "salesforce.salesforce_convert_lead", + "salesforce.salesforce_list_opportunities" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 110, + "k3_calls": 37, + "k3_tokens": 9458, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 22, + "k5_tokens": 9076, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 11, + "k10_tokens": 8834, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q012", + "query": "Show git status", + "kind": "identifier", + "gold": [ + "git.git_status" + ], + "alts": [], + "ranking": [ + "github.search_commits", + "github.search_repositories", + "hubspot.hubspot_list_contacts", + "playwright.browser_tab_list", + "shopify.shopify_list_customers", + "shopify.shopify_list_products", + "playwright.browser_tab_close", + "hubspot.hubspot_get_contact", + "slack.slack_get_channel_history", + "shopify.shopify_create_product" + ], + "hit1": "github.search_commits", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 112, + "k3_calls": 38, + "k3_tokens": 9466, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 23, + "k5_tokens": 9245, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9359, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "kind": "paraphrase", + "gold": [ + "git.git_commit" + ], + "alts": [], + "ranking": [ + "discord.discord_list_channels", + "github.list_code_scanning_alerts", + "discord.discord_get_channel", + "github.get_code_scanning_alert", + "github.add_comment_to_pending_review" + ], + "hit1": "discord.discord_list_channels", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 110, + "k3_calls": 37, + "k3_tokens": 9257, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 22, + "k5_tokens": 8907, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 11, + "k10_tokens": 8698, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_issue", + "linear.create_user", + "linear.linear_get_issue", + "linear.list_users", + "linear.linear_list_issues", + "linear.linear_update_issue", + "linear.get_user", + "linear.linear_assign_issue", + "linear.update_user", + "linear.linear_search_issues" + ], + "hit1": "linear.linear_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 303, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 461, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 820, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "kind": "paraphrase", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 136, + "k3_calls": 46, + "k3_tokens": 11712, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 28, + "k5_tokens": 11391, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 14, + "k10_tokens": 11055, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "ranking": [ + "github.update_issue_comment", + "github.update_environment", + "github.list_code_scanning_alerts", + "github.get_code_scanning_alert" + ], + "hit1": "github.update_issue_comment", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 38, + "k3_calls": 13, + "k3_tokens": 3791, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 8, + "k5_tokens": 3759, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3663, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "kind": "ambiguous", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "ranking": [ + "github.merge_pull_request", + "github.pull_request_read", + "github.create_pull_request", + "github.update_pull_request", + "github.search_pull_requests", + "github.pull_request_review_write", + "github.update_pull_request_branch", + "prometheus.prom_list_metrics", + "pagerduty.pagerduty_list_oncalls", + "pagerduty.pagerduty_list_incidents" + ], + "hit1": "github.merge_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 365, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 560, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 988, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "kind": "verb_product", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 62, + "k3_calls": 21, + "k3_tokens": 6123, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 13, + "k5_tokens": 6050, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 7, + "k10_tokens": 6293, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "kind": "resource", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "ranking": [ + "github.get_dependabot_alert", + "github.list_dependabot_alerts", + "github.delete_environment", + "google_calendar.gcal_list_events", + "vercel.vercel_get_deployment", + "notion.delete_data_source", + "linear.update_comment", + "github.update_environment", + "vercel.vercel_list_deployments", + "google_calendar.gcal_list_calendars" + ], + "hit1": "github.get_dependabot_alert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8597186998521972, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 671, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "kind": "verb_product", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_issues", + "linear.list_users", + "sentry.sentry_list_events", + "sentry.sentry_list_projects", + "sqlite.sqlite_query", + "okta.okta_list_users", + "github.list_runners", + "gitlab.list_snippets", + "slack.slack_list_users", + "notion.list_data_sources" + ], + "hit1": "sentry.sentry_list_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 716, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_customer", + "slack.create_reminder", + "slack.create_usergroup", + "slack.create_bookmark", + "slack.get_reminder", + "slack.list_reminders", + "slack.update_reminder", + "github.create_repository", + "gitlab.create_milestone", + "github.create_milestone" + ], + "hit1": "stripe.stripe_create_customer", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 450, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 892, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "kind": "paraphrase", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_list_oncalls" + ], + "hit1": "pagerduty.pagerduty_get_incident", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 217, + "k3_calls": 73, + "k3_tokens": 17304, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 44, + "k5_tokens": 16814, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 22, + "k10_tokens": 16374, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "kind": "verb_product", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "ranking": [ + "github.mark_all_notifications_read", + "github.get_notification_details", + "github.get_secret_scanning_alert", + "github.list_code_scanning_alerts", + "github.list_secret_scanning_alerts", + "playwright.browser_navigate", + "discord.discord_get_channel", + "sequential_thinking.sequentialthinking", + "pagerduty.pagerduty_acknowledge_incident" + ], + "hit1": "github.mark_all_notifications_read", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 245, + "k3_calls": 82, + "k3_tokens": 19583, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 49, + "k5_tokens": 18819, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 25, + "k10_tokens": 18648, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "kind": "verb_product", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "ranking": [ + "github.search_commits" + ], + "hit1": "github.search_commits", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 245, + "k3_calls": 82, + "k3_tokens": 19506, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 49, + "k5_tokens": 18792, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 25, + "k10_tokens": 18630, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "kind": "paraphrase", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "ranking": [ + "github.list_code_scanning_alerts", + "playwright.browser_evaluate", + "github.list_starred_repositories", + "playwright.browser_tab_close", + "salesforce.salesforce_delete_record", + "github.mark_all_notifications_read" + ], + "hit1": "github.list_code_scanning_alerts", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 253, + "k3_calls": 85, + "k3_tokens": 20366, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 51, + "k5_tokens": 19618, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 26, + "k10_tokens": 19523, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_create_event", + "google_calendar.gcal_list_events", + "github.create_environment", + "google_calendar.gcal_update_event", + "google_calendar.gcal_delete_event", + "slack.create_reminder", + "jira.create_filter", + "github.create_artifact", + "github.create_runner", + "slack.create_canvase" + ], + "hit1": "google_calendar.gcal_create_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 324, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 484, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 984, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_list_pods", + "kubernetes.k8s_list_deployments", + "github.projects_list", + "github.list_starred_repositories", + "linear.linear_list_projects", + "github.search_repositories", + "github.delete_repository", + "vercel.vercel_list_projects", + "github.list_code_scanning_alerts", + "cloudflare.cf_list_zones" + ], + "hit1": "kubernetes.k8s_list_pods", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 198, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 630, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "ranking": [ + "docker.docker_images" + ], + "hit1": "docker.docker_images", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 193, + "k3_calls": 65, + "k3_tokens": 15818, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 39, + "k5_tokens": 15194, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 20, + "k10_tokens": 15064, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "ranking": [ + "slack.get_reminder", + "slack.create_reminder", + "notion.create_users_token", + "slack.update_reminder", + "slack.delete_reminder", + "slack.list_reminders", + "notion.update_users_token", + "github.get_code_scanning_alert", + "notion.create_data_source", + "confluence.confluence_search" + ], + "hit1": "slack.get_reminder", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 194, + "k3_calls": 65, + "k3_tokens": 15640, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 39, + "k5_tokens": 15120, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 20, + "k10_tokens": 15019, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "kind": "verb_product", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "ranking": [ + "slack.slack_get_channel_history" + ], + "hit1": "slack.slack_get_channel_history", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 184, + "k3_calls": 62, + "k3_tokens": 14888, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 37, + "k5_tokens": 14342, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 19, + "k10_tokens": 14250, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "kind": "ambiguous", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "ranking": [ + "playwright.browser_evaluate", + "playwright.browser_click", + "playwright.browser_tab_list", + "playwright.browser_tab_close", + "playwright.browser_fill_form", + "github.list_code_scanning_alerts", + "github.get_code_scanning_alert", + "github.mark_all_notifications_read" + ], + "hit1": "playwright.browser_evaluate", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 171, + "k3_calls": 57, + "k3_tokens": 13984, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 35, + "k5_tokens": 13740, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 18, + "k10_tokens": 13672, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "kind": "ambiguous", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "ranking": [ + "github.create_or_update_file" + ], + "hit1": "github.create_or_update_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 171, + "k3_calls": 57, + "k3_tokens": 14038, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 35, + "k5_tokens": 13740, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 18, + "k10_tokens": 13672, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_create_merge_request", + "gitlab.create_milestone", + "github.create_pull_request", + "github.pull_request_read", + "github.create_milestone", + "github.create_repository", + "github.get_release_by_tag", + "github.search_pull_requests", + "postgres.describe_table", + "sqlite.sqlite_describe_table" + ], + "hit1": "gitlab.gitlab_create_merge_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 397, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 592, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 930, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "kind": "verb_product", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "ranking": [ + "notion.get_data_source", + "notion.update_data_source", + "notion.create_data_source", + "notion.list_data_sources", + "postgres.explain_query", + "linear.get_comment", + "hubspot.hubspot_search_contacts", + "pagerduty.pagerduty_list_oncalls", + "stripe.stripe_cancel_subscription", + "linear.list_comments" + ], + "hit1": "notion.get_data_source", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 165, + "k3_calls": 55, + "k3_tokens": 13537, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 33, + "k5_tokens": 13097, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 17, + "k10_tokens": 13110, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "kind": "verb_product", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "ranking": [ + "notion.notion_create_page", + "grafana.grafana_create_annotation", + "jira.get_version", + "notion.update_data_source", + "datadog.datadog_create_monitor", + "jira.update_version", + "stripe.stripe_create_payment_intent", + "hubspot.hubspot_create_contact", + "linear.get_comment", + "postgres.explain_query" + ], + "hit1": "notion.notion_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 430, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 773, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "kind": "verb_product", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "ranking": [ + "notion.notion_query_database", + "notion.update_data_source", + "notion.get_data_source", + "notion.list_data_sources", + "notion.create_data_source", + "notion.delete_data_source", + "notion.update_users_token", + "notion.notion_update_page", + "github.actions_get", + "memory.delete_observations" + ], + "hit1": "notion.notion_query_database", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 233, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 752, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_update_ticket", + "github.get_notification_details", + "kubernetes.k8s_scale_deployment", + "github.list_dependabot_alerts", + "pagerduty.pagerduty_list_incidents" + ], + "hit1": "zendesk.zendesk_create_ticket", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 756, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_contact", + "github.list_code_scanning_alerts", + "github.get_code_scanning_alert", + "github.add_comment_to_pending_review", + "github.get_notification_details", + "playwright.browser_take_screenshot" + ], + "hit1": "hubspot.hubspot_create_contact", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 301, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 508, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 832, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "kind": "verb_product", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_create_lead", + "salesforce.salesforce_create_record", + "salesforce.salesforce_convert_lead", + "salesforce.salesforce_update_record", + "notion.create_data_source", + "salesforce.salesforce_query", + "salesforce.salesforce_get_record", + "salesforce.salesforce_delete_record", + "linear.linear_create_issue", + "gitlab.create_milestone" + ], + "hit1": "salesforce.salesforce_create_lead", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 445, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 895, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "kind": "identifier", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query_range", + "salesforce.salesforce_query", + "pagerduty.pagerduty_list_oncalls", + "salesforce.salesforce_update_record", + "pagerduty.pagerduty_resolve_incident", + "sequential_thinking.sequentialthinking", + "salesforce.salesforce_list_opportunities", + "github.assign_copilot_to_issue", + "github.list_dependabot_alerts", + "github.actions_get_job_logs" + ], + "hit1": "prometheus.prom_query_range", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 398, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 837, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "kind": "verb_product", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "ranking": [ + "shopify.shopify_list_orders", + "shopify.shopify_get_order", + "shopify.shopify_create_order", + "shopify.shopify_list_customers", + "shopify.shopify_list_products", + "shopify.shopify_create_product", + "shopify.shopify_update_product", + "playwright.browser_snapshot", + "playwright.browser_take_screenshot", + "github.fork_repository" + ], + "hit1": "shopify.shopify_list_orders", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 210, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 692, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_acknowledge_incident", + "jira.update_sprint", + "github.list_dependabot_alerts", + "google_calendar.gcal_delete_event", + "linear.linear_update_issue", + "google_calendar.gcal_update_event" + ], + "hit1": "pagerduty.pagerduty_create_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 823, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "kind": "paraphrase", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "ranking": [ + "github.get_code_scanning_alert", + "github.list_code_scanning_alerts" + ], + "hit1": "github.get_code_scanning_alert", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 243, + "k3_calls": 81, + "k3_tokens": 19511, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 49, + "k5_tokens": 18939, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 25, + "k10_tokens": 18705, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "kind": "verb_product", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "ranking": [ + "datadog.datadog_query_metrics", + "elasticsearch.es_get_doc", + "elasticsearch.es_index_doc", + "pagerduty.pagerduty_list_oncalls", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_acknowledge_incident", + "github.add_comment_to_pending_review", + "pagerduty.pagerduty_create_incident", + "google_calendar.gcal_list_calendars" + ], + "hit1": "datadog.datadog_query_metrics", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 277, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 835, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "kind": "verb_product", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "ranking": [ + "notion.get_data_source", + "notion.list_data_sources", + "pagerduty.pagerduty_list_oncalls", + "google_calendar.gcal_list_calendars" + ], + "hit1": "notion.get_data_source", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 241, + "k3_calls": 81, + "k3_tokens": 19283, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 49, + "k5_tokens": 18785, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 25, + "k10_tokens": 18646, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_list_buckets" + ], + "hit1": "aws_s3.s3_list_buckets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 153, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 311, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "kind": "paraphrase", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 296, + "k3_calls": 99, + "k3_tokens": 23677, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 60, + "k5_tokens": 23014, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 30, + "k10_tokens": 22354, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_purge_cache", + "cloudflare.cf_create_dns", + "cloudflare.cf_list_zones", + "cloudflare.cf_list_dns", + "slack.update_reminder", + "salesforce.salesforce_update_record", + "github.create_or_update_file", + "notion.update_data_source", + "clickup.clickup_create_task", + "confluence.confluence_create_page" + ], + "hit1": "cloudflare.cf_purge_cache", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 844, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "cloudflare.cf_list_zones", + "discord.discord_add_reaction", + "discord.discord_list_channels", + "google_calendar.gcal_delete_event", + "github.mark_all_notifications_read", + "google_calendar.gcal_update_event", + "google_calendar.gcal_list_calendars", + "salesforce.salesforce_list_opportunities" + ], + "hit1": "cloudflare.cf_create_dns", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 642, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_deployments", + "kubernetes.k8s_list_deployments", + "linear.list_comments", + "linear.delete_comment", + "linear.list_attachments", + "memory.delete_entities", + "linear.update_comment", + "stripe.stripe_list_customers", + "linear.delete_attachment", + "pagerduty.pagerduty_list_incidents" + ], + "hit1": "vercel.vercel_list_deployments", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 336, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "ranking": [ + "playwright.browser_type", + "notion.update_users_token", + "playwright.browser_tab_close", + "playwright.browser_tab_new", + "playwright.browser_tab_list", + "playwright.browser_wait_for", + "playwright.browser_evaluate", + "playwright.browser_navigate", + "playwright.browser_snapshot", + "playwright.browser_click" + ], + "hit1": "playwright.browser_type", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 309, + "k3_calls": 103, + "k3_tokens": 24489, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 62, + "k5_tokens": 23669, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 31, + "k10_tokens": 22987, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "kind": "paraphrase", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "ranking": [ + "terraform.tf_state_list", + "stripe.stripe_get_customer", + "filesystem.list_allowed_directories", + "filesystem.list_directory_with_sizes" + ], + "hit1": "terraform.tf_state_list", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 316, + "k3_calls": 106, + "k3_tokens": 25071, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 64, + "k5_tokens": 24328, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 32, + "k10_tokens": 23656, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "kind": "verb_product", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "ranking": [ + "github.list_code_scanning_alerts", + "github.mark_all_notifications_read", + "sequential_thinking.sequentialthinking" + ], + "hit1": "github.list_code_scanning_alerts", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 316, + "k3_calls": 106, + "k3_tokens": 25217, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 64, + "k5_tokens": 24398, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 32, + "k10_tokens": 23694, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "kind": "verb_product", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "ranking": [ + "notion.create_users_token", + "hubspot.hubspot_create_contact", + "slack.update_canvase", + "playwright.browser_tab_new", + "postgres.explain_query", + "prometheus.prom_query_range", + "memory.delete_observations", + "kubernetes.k8s_scale_deployment", + "playwright.browser_take_screenshot", + "salesforce.salesforce_convert_lead" + ], + "hit1": "notion.create_users_token", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 325, + "k3_calls": 109, + "k3_tokens": 25800, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 65, + "k5_tokens": 24768, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 33, + "k10_tokens": 24366, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "ranking": [ + "linear.list_comments", + "notion.create_users_token", + "notion.delete_users_token", + "github.list_environments", + "stripe.stripe_create_customer", + "github.update_environment", + "playwright.browser_take_screenshot", + "playwright.browser_tab_new", + "hubspot.hubspot_search_contacts", + "stripe.stripe_create_refund" + ], + "hit1": "linear.list_comments", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 329, + "k3_calls": 110, + "k3_tokens": 25880, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 66, + "k5_tokens": 25000, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 33, + "k10_tokens": 24340, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "kind": "verb_product", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "ranking": [ + "grafana.grafana_search_dashboards", + "google_calendar.gcal_list_calendars", + "salesforce.salesforce_create_lead", + "salesforce.salesforce_convert_lead", + "pagerduty.pagerduty_list_oncalls" + ], + "hit1": "grafana.grafana_search_dashboards", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 225, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "kind": "verb_product", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "ranking": [ + "slack.list_reminders", + "stripe.stripe_create_invoice", + "playwright.browser_click", + "stripe.stripe_cancel_subscription", + "stripe.stripe_create_price", + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_customer", + "stripe.stripe_create_product", + "playwright.browser_tab_close", + "github.get_secret_scanning_alert" + ], + "hit1": "slack.list_reminders", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 332, + "k3_calls": 111, + "k3_tokens": 26129, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 67, + "k5_tokens": 25314, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 34, + "k10_tokens": 25005, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "kind": "identifier", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "ranking": [ + "terraform.tf_plan", + "stripe.stripe_create_refund", + "linear.create_comment", + "trello.trello_add_comment", + "linear.linear_create_comment", + "stripe.stripe_create_invoice", + "github.mark_all_notifications_read", + "salesforce.salesforce_list_opportunities", + "stripe.stripe_create_payment_intent", + "salesforce.salesforce_convert_lead" + ], + "hit1": "terraform.tf_plan", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 365, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "kind": "verb_product", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "ranking": [ + "asana.asana_create_task", + "slack.create_canvase", + "gitlab.create_snippet", + "slack.slack_create_channel", + "slack.get_canvase", + "slack.slack_add_reaction", + "memory.create_relations", + "slack.update_canvase", + "notion.create_users_token", + "slack.delete_canvase" + ], + "hit1": "asana.asana_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 337, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 467, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 843, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "ranking": [ + "trello.trello_create_card", + "trello.trello_create_board", + "trello.trello_list_cards", + "memory.create_relations", + "gitlab.create_release", + "github.create_repository", + "github.create_pull_request", + "github.create_or_update_file", + "stripe.stripe_create_product", + "vercel.vercel_create_project" + ], + "hit1": "trello.trello_create_card", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 394, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 895, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_create_task", + "clickup.clickup_update_task", + "github.create_check_run", + "slack.slack_create_channel", + "gitlab.gitlab_create_issue", + "slack.slack_get_users", + "slack.slack_add_reaction", + "bitbucket.bitbucket_create_issue", + "slack.update_canvase", + "slack.slack_list_users" + ], + "hit1": "clickup.clickup_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 304, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 467, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 854, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "kind": "verb_product", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "ranking": [ + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "confluence.confluence_get_page", + "cloudflare.cf_purge_cache", + "google_calendar.gcal_create_event", + "google_calendar.gcal_list_events", + "notion.update_data_source", + "github.actions_run_trigger", + "google_calendar.gcal_update_event", + "google_calendar.gcal_delete_event" + ], + "hit1": "confluence.confluence_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 450, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 889, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "kind": "verb_product", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "salesforce.salesforce_get_record", + "filesystem.create_directory", + "salesforce.salesforce_convert_lead", + "github.list_code_scanning_alerts", + "salesforce.salesforce_delete_record", + "filesystem.list_allowed_directories", + "salesforce.salesforce_update_record", + "salesforce.salesforce_create_record", + "salesforce.salesforce_create_lead" + ], + "hit1": "discord.discord_send_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 752, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_search", + "elasticsearch.es_get_doc", + "elasticsearch.es_index_doc", + "slack.slack_search_messages", + "notion.notion_search", + "filesystem.search_files", + "gmail.gmail_search_emails", + "zendesk.zendesk_search", + "linear.linear_search_issues", + "jira.jira_search_issues" + ], + "hit1": "elasticsearch.es_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 700, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "ranking": [ + "jira.jira_get_issue", + "github.create_issue", + "slack.get_reminder", + "github.create_gist", + "jira.get_sprint", + "linear.get_user", + "linear.linear_get_issue", + "gitlab.get_release", + "jira.get_filter", + "slack.slack_get_thread_replies" + ], + "hit1": "jira.jira_get_issue", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 291, + "k3_calls": 97, + "k3_tokens": 22977, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 59, + "k5_tokens": 22447, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 30, + "k10_tokens": 22206, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 281, + "k3_calls": 94, + "k3_tokens": 22397, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 57, + "k5_tokens": 21868, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 29, + "k10_tokens": 21583, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "kind": "verb_product", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "ranking": [ + "mysql.mysql_list_tables", + "postgres.list_tables", + "sqlite.sqlite_list_tables", + "gitlab.list_labels", + "slack.list_files", + "gitlab.list_releases", + "gitlab.list_snippets", + "gitlab.list_pipelines", + "supabase.supabase_list_tables", + "github.list_labels" + ], + "hit1": "mysql.mysql_list_tables", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 651, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "kind": "verb_product", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "ranking": [ + "postgres.describe_table", + "postgres.list_tables", + "slack.list_reminders", + "github.list_starred_repositories", + "github.search_repositories", + "stripe.stripe_create_price", + "supabase.supabase_list_tables", + "elasticsearch.es_search", + "stripe.stripe_create_invoice", + "elasticsearch.es_index_doc" + ], + "hit1": "postgres.describe_table", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 677, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "github.create_gist", + "github.create_runner", + "github.create_milestone", + "gitlab.gitlab_create_issue", + "github.search_issues", + "github.create_hook", + "github.create_package", + "github.list_issues", + "github.create_artifact" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 324, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 520, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 998, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "ranking": [ + "github.create_issue", + "jira.jira_create_issue", + "linear.linear_create_issue", + "gitlab.gitlab_create_issue", + "bitbucket.bitbucket_create_issue", + "linear.linear_update_issue", + "sentry.sentry_update_issue", + "linear.create_user", + "jira.jira_update_issue", + "slack.create_file" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 397, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 579, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1039, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "kind": "identifier", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "ranking": [ + "filesystem.list_directory", + "filesystem.directory_tree", + "filesystem.list_directory_with_sizes", + "filesystem.create_directory", + "filesystem.list_allowed_directories", + "stripe.stripe_create_customer", + "github.list_starred_repositories", + "stripe.stripe_get_customer", + "github.list_repository_collaborators", + "github.star_repository" + ], + "hit1": "filesystem.list_directory", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 599, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "kind": "paraphrase", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "ranking": [ + "github.get_secret_scanning_alert", + "github.list_secret_scanning_alerts", + "github.get_code_scanning_alert", + "github.get_notification_details", + "github.list_code_scanning_alerts" + ], + "hit1": "github.get_secret_scanning_alert", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 116, + "k3_calls": 39, + "k3_tokens": 9799, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 24, + "k5_tokens": 9727, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9475, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "kind": "paraphrase", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "ranking": [ + "github.actions_get", + "github.update_issue_comment", + "github.update_milestone", + "github.create_milestone", + "github.add_issue_comment", + "github.assign_copilot_to_issue", + "github.actions_get_job_logs", + "github.actions_run_trigger", + "github.dismiss_notification", + "github.get_notification_details" + ], + "hit1": "github.actions_get", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 119, + "k3_calls": 40, + "k3_tokens": 10114, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 24, + "k5_tokens": 9762, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9498, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "kind": "verb_product", + "gold": [ + "time.convert_time" + ], + "alts": [], + "ranking": [ + "github.get_file_contents", + "notion.get_users_token", + "notion.delete_users_token", + "notion.create_users_token", + "github.get_discussion_comments", + "stripe.stripe_cancel_subscription", + "github.add_comment_to_pending_review", + "github.list_secret_scanning_alerts" + ], + "hit1": "github.get_file_contents", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 122, + "k3_calls": 41, + "k3_tokens": 10273, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 25, + "k5_tokens": 10083, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 13, + "k10_tokens": 10227, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "kind": "verb_product", + "gold": [ + "github.star_repository" + ], + "alts": [], + "ranking": [ + "github.star_repository", + "github.unstar_repository", + "github.search_code", + "github.list_releases", + "github.search_users", + "github.create_repository", + "github.label_write", + "github.create_runner", + "github.update_runner", + "github.search_repositories" + ], + "hit1": "github.star_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "kind": "resource", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "ranking": [ + "playwright.browser_wait_for", + "slack.list_bookmarks", + "playwright.browser_fill_form", + "shopify.shopify_create_order", + "slack.create_bookmark", + "filesystem.create_directory", + "terraform.tf_state_list", + "playwright.browser_click", + "salesforce.salesforce_list_opportunities", + "salesforce.salesforce_get_record" + ], + "hit1": "playwright.browser_wait_for", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 30, + "k3_calls": 10, + "k3_tokens": 2448, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 6, + "k5_tokens": 2372, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 3, + "k10_tokens": 2315, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "kind": "verb_product", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "ranking": [ + "github.get_latest_release", + "github.list_releases", + "gitlab.get_release", + "github.get_release_by_tag", + "github.create_milestone", + "github.get_milestone", + "github.create_issue", + "gitlab.delete_release", + "gitlab.create_release", + "github.delete_milestone" + ], + "hit1": "github.get_latest_release", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 810, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "kind": "resource", + "gold": [ + "github.create_gist" + ], + "alts": [], + "ranking": [ + "github.create_gist", + "gitlab.create_job", + "github.create_hook", + "gitlab.create_wiki", + "gitlab.create_snippet", + "gitlab.create_label", + "github.create_issue", + "github.create_runner", + "github.create_branch", + "github.create_artifact" + ], + "hit1": "github.create_gist", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 290, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 496, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1021, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "kind": "verb_product", + "gold": [ + "github.search_code" + ], + "alts": [], + "ranking": [ + "github.search_code", + "github.search_repositories", + "github.list_code_scanning_alerts" + ], + "hit1": "github.search_code", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 277, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 369, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 783, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "kind": "verb_product", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "ranking": [ + "github.list_notifications", + "github.dismiss_notification", + "github.mark_all_notifications_read", + "github.list_discussions", + "github.list_repository_collaborators", + "stripe.stripe_list_subscriptions", + "stripe.stripe_cancel_subscription", + "sequential_thinking.sequentialthinking" + ], + "hit1": "github.list_notifications", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 175, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 325, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 643, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "kind": "verb_product", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "ranking": [ + "github.assign_copilot_to_issue", + "github.list_code_scanning_alerts", + "discord.discord_add_reaction", + "salesforce.salesforce_list_opportunities" + ], + "hit1": "github.assign_copilot_to_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 312, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 824, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "kind": "resource", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "ranking": [ + "github.list_secret_scanning_alerts", + "github.get_secret_scanning_alert", + "github.list_code_scanning_alerts", + "github.get_code_scanning_alert", + "slack.slack_get_channel_history", + "slack.slack_set_channel_topic", + "sentry.sentry_list_events", + "google_calendar.gcal_list_events", + "github.create_environment", + "slack.slack_add_reaction" + ], + "hit1": "github.list_secret_scanning_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 465, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 884, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "ranking": [ + "github.get_code_scanning_alert", + "github.mark_all_notifications_read", + "github.list_code_scanning_alerts" + ], + "hit1": "github.get_code_scanning_alert", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 20, + "k3_calls": 7, + "k3_tokens": 1850, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 4, + "k5_tokens": 1698, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 2, + "k10_tokens": 1650, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "kind": "resource", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones", + "memory.delete_observations", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "salesforce.salesforce_list_opportunities", + "playwright.browser_take_screenshot" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 33, + "k3_calls": 11, + "k3_tokens": 2773, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 7, + "k5_tokens": 2927, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3392, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "kind": "verb_product", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "ranking": [ + "slack.get_reminder", + "slack.list_reminders", + "slack.update_reminder", + "slack.create_reminder", + "slack.delete_reminder", + "gitlab.create_pipeline", + "postgres.explain_query", + "github.list_branches", + "github.create_check_run", + "gitlab.create_snippet" + ], + "hit1": "slack.get_reminder", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 130, + "k3_calls": 44, + "k3_tokens": 11132, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 26, + "k5_tokens": 10591, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 13, + "k10_tokens": 10318, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "kind": "verb_product", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "ranking": [ + "memory.search_nodes", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident" + ], + "hit1": "memory.search_nodes", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 125, + "k3_calls": 42, + "k3_tokens": 10605, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 25, + "k5_tokens": 10158, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 13, + "k10_tokens": 10233, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "kind": "verb_product", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "ranking": [ + "slack.slack_search_messages", + "memory.search_nodes", + "elasticsearch.es_search", + "filesystem.search_files", + "elasticsearch.es_get_doc", + "github.search_issues", + "elasticsearch.es_index_doc", + "slack.slack_get_channel_history", + "jira.jira_search_issues", + "slack.slack_get_thread_replies" + ], + "hit1": "slack.slack_search_messages", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "kind": "verb_product", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "ranking": [ + "slack.slack_create_channel", + "slack.slack_leave_channel", + "slack.slack_list_channels", + "slack.slack_set_channel_topic", + "slack.slack_get_channel_history", + "slack.slack_get_thread_replies", + "discord.discord_list_channels", + "brave_search.brave_local_search", + "salesforce.salesforce_convert_lead", + "google_calendar.gcal_delete_event" + ], + "hit1": "slack.slack_create_channel", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 357, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "kind": "verb_product", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_transition_issue", + "jira.list_versions", + "jira.list_sprints", + "jira.update_version", + "stripe.stripe_cancel_subscription", + "github.mark_all_notifications_read", + "discord.discord_add_reaction", + "notion.create_users_token", + "stripe.stripe_create_invoice", + "gitlab.create_milestone" + ], + "hit1": "jira.jira_transition_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "kind": "identifier", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "ranking": [ + "jira.list_attachments", + "jira.delete_attachment" + ], + "hit1": "jira.list_attachments", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 151, + "k3_calls": 51, + "k3_tokens": 12663, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 31, + "k5_tokens": 12344, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 16, + "k10_tokens": 12438, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "kind": "verb_product", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 155, + "k3_calls": 52, + "k3_tokens": 12978, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 31, + "k5_tokens": 12447, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 16, + "k10_tokens": 12448, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "kind": "verb_product", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "ranking": [ + "linear.linear_list_teams", + "linear.list_users", + "linear.list_comments", + "linear.list_cycles", + "linear.list_labels", + "linear.list_attachments", + "linear.get_user", + "linear.create_user", + "linear.delete_user", + "linear.update_user" + ], + "hit1": "linear.linear_list_teams", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 199, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "ranking": [ + "linear.list_comments", + "linear.update_comment", + "github.create_environment", + "github.list_code_scanning_alerts", + "github.add_comment_to_pending_review", + "github.get_code_scanning_alert", + "salesforce.salesforce_convert_lead", + "salesforce.salesforce_list_opportunities" + ], + "hit1": "linear.list_comments", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 146, + "k3_calls": 49, + "k3_tokens": 12189, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 30, + "k5_tokens": 11992, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11677, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q094", + "query": "Move a local file", + "kind": "paraphrase", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "ranking": [ + "gitlab.create_milestone", + "gmail.gmail_modify_email", + "gitlab.list_milestones", + "google_drive.gdrive_create_file", + "gitlab.get_milestone", + "github.create_milestone", + "filesystem.list_allowed_directories", + "salesforce.salesforce_convert_lead", + "gitlab.delete_milestone", + "gitlab.update_milestone" + ], + "hit1": "gitlab.create_milestone", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 95, + "k3_calls": 32, + "k3_tokens": 8648, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 19, + "k5_tokens": 8320, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 10, + "k10_tokens": 8418, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "kind": "paraphrase", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "ranking": [ + "github.list_secret_scanning_alerts", + "stripe.stripe_cancel_subscription", + "github.get_secret_scanning_alert", + "stripe.stripe_list_subscriptions" + ], + "hit1": "github.list_secret_scanning_alerts", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 88, + "k3_calls": 30, + "k3_tokens": 8094, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7830, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7632, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "kind": "paraphrase", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "ranking": [ + "github.list_code_scanning_alerts", + "discord.discord_list_channels" + ], + "hit1": "github.list_code_scanning_alerts", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 81, + "k3_calls": 27, + "k3_tokens": 7623, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 17, + "k5_tokens": 7586, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7650, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "google_drive.gdrive_create_file", + "google_drive.gdrive_share_file", + "google_drive.gdrive_update_file", + "google_calendar.gcal_create_event", + "google_calendar.gcal_list_events", + "google_drive.gdrive_get_file", + "google_calendar.gcal_update_event", + "google_calendar.gcal_delete_event", + "google_drive.gdrive_list_folder" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 105, + "k3_calls": 35, + "k3_tokens": 9234, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 21, + "k5_tokens": 8940, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 11, + "k10_tokens": 9003, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "kind": "verb_product", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "github.search_repositories", + "confluence.confluence_search", + "notion.delete_data_source", + "notion.update_data_source", + "notion.get_data_source", + "grafana.grafana_search_dashboards", + "slack.create_reminder", + "notion.create_data_source", + "salesforce.salesforce_query" + ], + "hit1": "memory.search_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "kind": "verb_product", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 105, + "k3_calls": 35, + "k3_tokens": 8947, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 21, + "k5_tokens": 8653, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 11, + "k10_tokens": 8735, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "kind": "identifier", + "gold": [ + "git.git_add" + ], + "alts": [], + "ranking": [ + "filesystem.list_allowed_directories", + "github.pull_request_read", + "github.list_dependabot_alerts", + "github.list_starred_repositories", + "github.create_or_update_file", + "github.get_notification_details", + "discord.discord_list_channels", + "github.get_dependabot_alert", + "jira.delete_attachment", + "github.mark_all_notifications_read" + ], + "hit1": "filesystem.list_allowed_directories", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 111, + "k3_calls": 37, + "k3_tokens": 9333, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 23, + "k5_tokens": 9295, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9410, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "kind": "verb_product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "ranking": [ + "shopify.shopify_create_product", + "hubspot.hubspot_create_deal", + "hubspot.hubspot_create_contact", + "playwright.browser_take_screenshot" + ], + "hit1": "shopify.shopify_create_product", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 286, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 414, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 820, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "kind": "verb_product", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "ranking": [ + "stripe.stripe_cancel_subscription", + "stripe.stripe_create_invoice", + "slack.slack_add_reaction", + "stripe.stripe_create_payment_intent", + "github.create_milestone", + "hubspot.hubspot_create_contact", + "github.create_environment", + "github.list_secret_scanning_alerts", + "github.get_secret_scanning_alert", + "salesforce.salesforce_list_opportunities" + ], + "hit1": "stripe.stripe_cancel_subscription", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 400, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 809, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "kind": "resource", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "ranking": [ + "notion.create_comment", + "linear.create_comment", + "stripe.stripe_create_payment_intent", + "linear.create_attachment", + "jira.create_attachment", + "linear.get_attachment", + "linear.update_attachment", + "jira.get_attachment", + "stripe.stripe_get_payment_intent", + "linear.update_comment" + ], + "hit1": "notion.create_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 299, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 489, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 779, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "kind": "verb_product", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "ranking": [ + "datadog.datadog_create_monitor", + "notion.create_data_source", + "notion.get_data_source", + "notion.update_data_source", + "notion.delete_data_source", + "github.add_comment_to_pending_review", + "google_calendar.gcal_list_calendars" + ], + "hit1": "datadog.datadog_create_monitor", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 296, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 423, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "kind": "paraphrase", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "ranking": [ + "postgres.explain_query", + "postgres.list_schemas", + "slack.slack_get_user_profile", + "github.search_repositories", + "salesforce.salesforce_query", + "github.list_starred_repositories", + "stripe.stripe_create_customer", + "github.list_secret_scanning_alerts", + "postgres.list_tables", + "notion.create_data_source" + ], + "hit1": "postgres.explain_query", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 232, + "k3_calls": 78, + "k3_tokens": 18609, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 47, + "k5_tokens": 18021, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 24, + "k10_tokens": 17953, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_scale_deployment", + "gitlab.delete_snippet", + "github.create_environment", + "memory.create_entities", + "github.list_dependabot_alerts", + "gitlab.delete_milestone", + "github.update_pull_request", + "github.get_dependabot_alert", + "linear.linear_create_comment", + "playwright.browser_navigate" + ], + "hit1": "kubernetes.k8s_scale_deployment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 395, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 804, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "ranking": [ + "gitlab.create_snippet", + "gitlab.delete_snippet", + "github.create_milestone", + "memory.create_entities", + "linear.linear_list_teams", + "linear.linear_list_projects", + "gitlab.create_milestone", + "linear.linear_list_issues", + "gitlab.delete_milestone", + "github.update_milestone" + ], + "hit1": "gitlab.create_snippet", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 200, + "k3_calls": 67, + "k3_tokens": 16099, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 40, + "k5_tokens": 15471, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 20, + "k10_tokens": 15071, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "ranking": [ + "slack.get_reminder", + "notion.create_users_token", + "okta.okta_list_users", + "notion.update_users_token", + "notion.get_data_source", + "slack.create_reminder", + "slack.update_reminder", + "slack.delete_reminder", + "slack.list_reminders", + "notion.create_data_source" + ], + "hit1": "slack.get_reminder", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 196, + "k3_calls": 66, + "k3_tokens": 15806, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 40, + "k5_tokens": 15415, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 20, + "k10_tokens": 15015, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "kind": "verb_product", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "ranking": [ + "github.update_package", + "github.update_check_run", + "notion.list_data_sources", + "slack.slack_get_user_profile", + "github.list_dependabot_alerts", + "notion.delete_data_source", + "github.list_code_scanning_alerts", + "bitbucket.bitbucket_create_pr", + "bitbucket.bitbucket_create_issue", + "github.get_code_scanning_alert" + ], + "hit1": "github.update_package", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 195, + "k3_calls": 65, + "k3_tokens": 15591, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 39, + "k5_tokens": 15071, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 20, + "k10_tokens": 15055, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "kind": "verb_product", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "ranking": [ + "linear.get_attachment", + "linear.list_attachments", + "playwright.browser_take_screenshot", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_create_incident" + ], + "hit1": "linear.get_attachment", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 175, + "k3_calls": 59, + "k3_tokens": 14528, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 35, + "k5_tokens": 13874, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 18, + "k10_tokens": 13763, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "kind": "paraphrase", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "ranking": [ + "github.unstar_repository", + "github.star_repository", + "github.create_repository", + "github.search_repositories", + "github.list_starred_repositories", + "github.list_repository_collaborators" + ], + "hit1": "github.unstar_repository", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 169, + "k3_calls": 57, + "k3_tokens": 14038, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 34, + "k5_tokens": 13458, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 17, + "k10_tokens": 13084, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_list_projects", + "github.projects_list", + "github.projects_get", + "github.projects_write", + "postgres.list_tables", + "github.list_dependabot_alerts", + "github.list_pull_requests", + "github.list_environments", + "postgres.list_schemas", + "playwright.browser_tab_list" + ], + "hit1": "gitlab.gitlab_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 653, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "kind": "verb_product", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "ranking": [ + "github.create_pull_request", + "github.pull_request_read", + "github.search_pull_requests", + "pagerduty.pagerduty_acknowledge_incident" + ], + "hit1": "github.create_pull_request", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 272, + "k3_calls": 91, + "k3_tokens": 21988, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 55, + "k5_tokens": 21340, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 28, + "k10_tokens": 20983, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "kind": "verb_product", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "ranking": [ + "gmail.gmail_create_draft", + "jira.create_filter", + "gitlab.create_job", + "slack.create_file", + "gitlab.create_wiki", + "github.create_artifact", + "gitlab.create_label", + "gitlab.create_milestone", + "github.create_milestone", + "github.create_issue" + ], + "hit1": "gmail.gmail_create_draft", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 494, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1037, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "kind": "verb_product", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_share_file", + "postgres.describe_table", + "github.create_or_update_file", + "slack.slack_get_user_profile", + "google_calendar.gcal_list_calendars", + "postgres.list_tables", + "github.list_starred_repositories", + "discord.discord_list_channels", + "notion.list_data_sources", + "gitlab.gitlab_create_issue" + ], + "hit1": "google_drive.gdrive_share_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 316, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "kind": "verb_product", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "google_drive.gdrive_search", + "github.list_dependabot_alerts", + "google_drive.gdrive_share_file", + "github.list_code_scanning_alerts", + "github.list_secret_scanning_alerts", + "pagerduty.pagerduty_list_incidents", + "github.get_secret_scanning_alert", + "github.get_code_scanning_alert" + ], + "hit1": "google_calendar.gcal_list_events", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 816, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_create_event", + "google_calendar.gcal_update_event", + "jira.delete_board", + "slack.delete_reminder", + "slack.delete_canvase", + "slack.delete_bookmark", + "notion.delete_data_source", + "github.list_dependabot_alerts" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 301, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 440, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_projects", + "linear.linear_list_projects", + "sentry.sentry_list_projects", + "vercel.vercel_list_deployments", + "linear.linear_create_project", + "linear.list_comments", + "stripe.stripe_list_customers", + "playwright.browser_tab_close", + "trello.trello_list_cards", + "circleci.circleci_list_pipelines" + ], + "hit1": "vercel.vercel_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 156, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 614, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_list_opportunities" + ], + "hit1": "salesforce.salesforce_list_opportunities", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 307, + "k3_calls": 103, + "k3_tokens": 24397, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 62, + "k5_tokens": 23613, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 31, + "k10_tokens": 22962, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "kind": "verb_product", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "github.get_code_scanning_alert", + "github.list_code_scanning_alerts" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 289, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 817, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "kind": "verb_product", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "ranking": [ + "okta.okta_list_users", + "slack.list_canvases", + "github.list_dependabot_alerts", + "postgres.list_tables", + "slack.list_bookmarks", + "playwright.browser_tab_list", + "notion.update_users_token", + "notion.create_users_token", + "terraform.tf_state_list", + "postgres.list_schemas" + ], + "hit1": "okta.okta_list_users", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 233, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 361, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 656, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query_range", + "linear.create_user", + "okta.okta_create_user", + "slack.create_usergroup", + "notion.create_users_token", + "github.projects_get", + "stripe.stripe_create_customer", + "github.list_environments", + "github.create_check_run", + "playwright.browser_tab_new" + ], + "hit1": "prometheus.prom_query_range", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 266, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 472, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 835, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "kind": "verb_product", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "ranking": [ + "grafana.grafana_create_annotation", + "github.mark_all_notifications_read", + "github.get_notification_details", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_acknowledge_incident" + ], + "hit1": "grafana.grafana_create_annotation", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 197, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "kind": "identifier", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "ranking": [ + "terraform.tf_apply", + "terraform.tf_plan", + "terraform.tf_state_list", + "playwright.browser_wait_for", + "playwright.browser_fill_form", + "trello.trello_create_board", + "stripe.stripe_create_price", + "trello.trello_add_comment", + "github.search_repositories", + "linear.create_comment" + ], + "hit1": "terraform.tf_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 287, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 681, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "kind": "verb_product", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "ranking": [ + "asana.asana_list_tasks", + "slack.list_canvases", + "slack.list_pins", + "gitlab.list_snippets", + "slack.slack_list_users", + "slack.list_bookmarks", + "slack.slack_list_channels", + "jira.list_attachments", + "github.list_packages", + "gitlab.list_labels" + ], + "hit1": "asana.asana_list_tasks", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 395, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 767, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "ranking": [ + "linear.delete_comment", + "github.create_environment", + "kubernetes.k8s_scale_deployment", + "salesforce.salesforce_convert_lead", + "salesforce.salesforce_list_opportunities" + ], + "hit1": "linear.delete_comment", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 345, + "k3_calls": 115, + "k3_tokens": 27183, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 69, + "k5_tokens": 26171, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 35, + "k10_tokens": 25769, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_update_task", + "clickup.clickup_create_task", + "slack.update_canvase", + "slack.update_bookmark", + "github.update_check_run", + "slack.slack_upload_file", + "playwright.browser_click", + "github.list_packages", + "github.create_check_run", + "github.assign_copilot_to_issue" + ], + "hit1": "clickup.clickup_update_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 277, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 840, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "kind": "verb_product", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "ranking": [ + "confluence.confluence_search", + "cloudflare.cf_purge_cache", + "notion.list_data_sources", + "notion.update_data_source", + "notion.delete_data_source", + "google_calendar.gcal_create_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_update_event", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_list_calendars" + ], + "hit1": "confluence.confluence_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_deal", + "github.create_milestone", + "github.list_dependabot_alerts", + "github.create_pull_request", + "github.get_dependabot_alert", + "github.create_or_update_file", + "github.search_pull_requests", + "github.list_code_scanning_alerts", + "github.get_code_scanning_alert", + "github.get_notification_details" + ], + "hit1": "hubspot.hubspot_create_deal", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 536, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 982, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_list_tickets", + "kubernetes.k8s_list_services", + "pagerduty.pagerduty_list_incidents", + "kubernetes.k8s_list_deployments", + "notion.delete_users_token", + "github.list_dependabot_alerts" + ], + "hit1": "zendesk.zendesk_list_tickets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 644, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "kind": "verb_product", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "ranking": [ + "discord.discord_list_channels", + "discord.discord_get_channel", + "github.list_code_scanning_alerts", + "google_calendar.gcal_list_calendars", + "github.mark_all_notifications_read", + "salesforce.salesforce_convert_lead" + ], + "hit1": "discord.discord_list_channels", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 596, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "ranking": [ + "slack.get_reminder", + "slack.create_pin", + "sentry.sentry_update_issue", + "slack.create_file", + "postgres.list_tables", + "slack.slack_get_thread_replies", + "postgres.describe_table", + "slack.delete_reminder", + "slack.list_reminders", + "slack.create_reminder" + ], + "hit1": "slack.get_reminder", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 293, + "k3_calls": 98, + "k3_tokens": 23335, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 59, + "k5_tokens": 22602, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 30, + "k10_tokens": 22356, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_list_collections" + ], + "hit1": "mongodb.mongo_list_collections", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 283, + "k3_calls": 95, + "k3_tokens": 22628, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 57, + "k5_tokens": 21868, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 29, + "k10_tokens": 21583, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "kind": "verb_product", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query_range", + "memory.search_nodes", + "memory.delete_relations", + "memory.add_observations", + "memory.create_relations", + "pagerduty.pagerduty_list_oncalls", + "memory.create_entities", + "memory.delete_observations", + "pagerduty.pagerduty_resolve_incident", + "sequential_thinking.sequentialthinking" + ], + "hit1": "prometheus.prom_query_range", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 280, + "k3_calls": 94, + "k3_tokens": 22516, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 56, + "k5_tokens": 21595, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 28, + "k10_tokens": 21007, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_list_calendars" + ], + "hit1": "google_calendar.gcal_list_calendars", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 291, + "k3_calls": 97, + "k3_tokens": 23125, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 59, + "k5_tokens": 22581, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 30, + "k10_tokens": 22324, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 295, + "k3_calls": 99, + "k3_tokens": 23578, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 59, + "k5_tokens": 22581, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 30, + "k10_tokens": 22324, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones", + "cloudflare.cf_list_dns", + "discord.discord_list_channels", + "google_calendar.gcal_list_events", + "notion.update_users_token", + "salesforce.salesforce_update_record", + "google_calendar.gcal_list_calendars", + "salesforce.salesforce_list_opportunities", + "salesforce.salesforce_delete_record", + "github.mark_all_notifications_read" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 161, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 613, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "notion.notion_create_page", + "github.create_environment", + "shopify.shopify_create_order", + "linear.linear_create_project", + "shopify.shopify_create_product", + "discord.discord_add_reaction", + "playwright.browser_navigate", + "jira.jira_transition_issue", + "notion.update_data_source" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 770, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "kind": "verb_product", + "gold": [ + "github.actions_list" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 60, + "k3_calls": 20, + "k3_tokens": 5782, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 12, + "k5_tokens": 5614, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 6, + "k10_tokens": 5488, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "kind": "verb_product", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "ranking": [ + "github.actions_get_job_logs" + ], + "hit1": "github.actions_get_job_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "kind": "verb_product", + "gold": [ + "github.create_repository" + ], + "alts": [], + "ranking": [ + "github.create_repository", + "github.search_repositories", + "github.create_milestone", + "github.list_starred_repositories", + "github.create_pull_request", + "github.list_repository_collaborators", + "gitlab.create_snippet", + "github.create_or_update_file", + "github.pull_request_read", + "gitlab.create_milestone" + ], + "hit1": "github.create_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 502, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1026, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "kind": "verb_product", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "ranking": [ + "github.create_check_run", + "github.update_check_run", + "github.search_commits", + "github.search_repositories", + "github.search_pull_requests", + "playwright.browser_take_screenshot", + "pagerduty.pagerduty_acknowledge_incident" + ], + "hit1": "github.create_check_run", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 117, + "k3_calls": 39, + "k3_tokens": 9806, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 24, + "k5_tokens": 9724, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9484, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q144", + "query": "Show git log", + "kind": "identifier", + "gold": [ + "git.git_log" + ], + "alts": [], + "ranking": [ + "github.search_orgs", + "github.list_hooks", + "shopify.shopify_get_order", + "shopify.shopify_list_orders", + "shopify.shopify_get_product", + "jira.list_webhooks", + "github.request_copilot_review", + "playwright.browser_navigate", + "shopify.shopify_list_products", + "playwright.browser_wait_for" + ], + "hit1": "github.search_orgs", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 119, + "k3_calls": 40, + "k3_tokens": 9928, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 24, + "k5_tokens": 9624, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9396, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "kind": "verb_product", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "ranking": [ + "postgres.list_schemas", + "slack.list_reminders", + "github.list_branches", + "github.search_repositories", + "elasticsearch.es_search", + "elasticsearch.es_get_doc", + "discord.discord_list_channels", + "stripe.stripe_create_customer", + "filesystem.search_files", + "discord.discord_list_members" + ], + "hit1": "postgres.list_schemas", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 207, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 668, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "kind": "verb_product", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "github.add_comment_to_pending_review", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 100, + "k3_calls": 34, + "k3_tokens": 8817, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 20, + "k5_tokens": 8407, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 10, + "k10_tokens": 8187, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "kind": "paraphrase", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "ranking": [ + "filesystem.read_media_file", + "filesystem.create_directory", + "filesystem.list_allowed_directories", + "salesforce.salesforce_create_lead", + "salesforce.salesforce_update_record" + ], + "hit1": "filesystem.read_media_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 90, + "k3_calls": 30, + "k3_tokens": 8177, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7913, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7715, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "kind": "paraphrase", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "ranking": [ + "github.label_write", + "linear.list_users", + "linear.list_labels", + "linear.list_cycles", + "linear.create_label", + "linear.list_comments", + "linear.get_label", + "linear.create_cycle", + "gitlab.create_milestone", + "linear.linear_list_teams" + ], + "hit1": "github.label_write", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 88, + "k3_calls": 30, + "k3_tokens": 8232, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7980, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7791, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "kind": "verb_product", + "gold": [ + "github.search_issues" + ], + "alts": [], + "ranking": [ + "github.search_issues", + "github.search_users", + "github.search_orgs", + "github.search_commits", + "github.search_repositories", + "github.search_pull_requests", + "github.list_branches", + "github.list_secret_scanning_alerts", + "hubspot.hubspot_search_contacts", + "github.list_code_scanning_alerts" + ], + "hit1": "github.search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 712, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "kind": "paraphrase", + "gold": [ + "github.get_me" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_deal", + "hubspot.hubspot_get_contact", + "github.mark_all_notifications_read", + "github.add_comment_to_pending_review", + "hubspot.hubspot_search_contacts", + "hubspot.hubspot_create_contact", + "sequential_thinking.sequentialthinking", + "bitbucket.bitbucket_create_pr", + "bitbucket.bitbucket_create_issue" + ], + "hit1": "hubspot.hubspot_create_deal", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 10, + "k3_calls": 4, + "k3_tokens": 1056, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 2, + "k5_tokens": 907, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 885, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "cost": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 28.313333333333333, + "p95_calls": 101.19999999999993, + "success_within_1": 0.5066666666666667, + "success_within_2": 0.5066666666666667, + "success_within_3": 0.5133333333333333, + "mean_tokens": 6934.52, + "p95_tokens": 24072.999999999985, + "total_tokens": 1040178, + "tokens_per_successful_query": 6934.52, + "total_tokens_successful": 1040178 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 17.26, + "p95_calls": 61.099999999999966, + "success_within_1": 0.5066666666666667, + "success_within_2": 0.52, + "success_within_3": 0.52, + "mean_tokens": 6801.473333333333, + "p95_tokens": 23343.44999999999, + "total_tokens": 1020221, + "tokens_per_successful_query": 6801.473333333333, + "total_tokens_successful": 1020221 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 9.013333333333334, + "p95_calls": 30.549999999999983, + "success_within_1": 0.52, + "success_within_2": 0.5266666666666666, + "success_within_3": 0.5333333333333333, + "mean_tokens": 6907.233333333334, + "p95_tokens": 22689.29999999999, + "total_tokens": 1036085, + "tokens_per_successful_query": 6907.233333333334, + "total_tokens_successful": 1036085 + } + } +} diff --git a/experiments/mcp-tool-search/results/codedb_search.csv b/experiments/mcp-tool-search/results/codedb_search.csv new file mode 100644 index 00000000..f8ec56d6 --- /dev/null +++ b/experiments/mcp-tool-search/results/codedb_search.csv @@ -0,0 +1,151 @@ +id,query,kind,hit1,recall@1,recall@3,recall@5,mrr@10,ndcg@10,hit_rank,k3_calls,k3_tokens,k3_success1,k5_calls,k5_tokens,k5_success1,k10_calls,k10_tokens,k10_success1,ranking +q001,Create a GitHub issue,verb_product,github.create_issue,1.0,1.0,1.0,1.0000,0.8956,1,1,335,1.0,1,582,1.0,1,1111,1.0,github.create_issue gitlab.gitlab_create_issue bitbucket.bitbucket_create_issue linear.linear_create_issue jira.jira_create_issue github.issue_write git.git_create_branch github.issue_read github.add_issue_comment github.update_issue_comment +q002,Create a Jira ticket,verb_product,jira.jira_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,333,1.0,1,481,1.0,1,966,1.0,jira.jira_create_issue zendesk.zendesk_create_ticket jira.jira_create_sprint zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket jira.create_attachment jira.create_webhook jira.create_sprint jira.create_filter jira.create_version +q003,Send a Slack message,verb_product,discord.discord_send_message,0.0,1.0,1.0,0.5000,0.6309,2,1,247,1.0,1,427,1.0,1,727,1.0,discord.discord_send_message slack.slack_post_message slack.slack_add_reaction twilio.twilio_send_sms gmail.gmail_send_email slack.slack_leave_channel slack.slack_join_channel slack.slack_get_user_profile slack.slack_search_messages slack.slack_create_channel +q004,List open pull requests,resource,github.list_pull_requests,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,460,1.0,1,836,1.0,github.list_pull_requests github.search_pull_requests gitlab.gitlab_list_merge_requests bitbucket.bitbucket_list_prs github.create_pull_request playwright.browser_tab_list memory.open_nodes github.merge_pull_request github.pull_request_read github.update_pull_request_branch +q005,Open a pull request on GitHub,verb_product,github.create_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,368,1.0,1,653,1.0,1,1144,1.0,github.create_pull_request github.merge_pull_request github.pull_request_read github.pull_request_review_write github.update_pull_request github.update_pull_request_branch github.request_copilot_review github.add_comment_to_pending_review github.search_pull_requests github.list_pull_requests +q006,Run a SQL query against postgres,verb_product,postgres.query,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,295,1.0,1,553,1.0,postgres.query mysql.mysql_query sqlite.sqlite_query postgres.explain_query salesforce.salesforce_query postgres.list_schemas postgres.list_tables postgres.describe_table github.delete_check_run github.get_check_run +q007,Read a local file from disk,paraphrase,filesystem.read_media_file,0.0,1.0,1.0,0.5000,0.6697,2,1,223,1.0,1,390,1.0,1,759,1.0,filesystem.read_media_file filesystem.read_file filesystem.read_text_file figma.figma_get_file_nodes gitlab.gitlab_get_file github.delete_file github.get_file_contents brave_search.brave_local_search memory.read_graph github.mark_all_notifications_read +q008,What files are in this folder,paraphrase,google_drive.gdrive_list_folder,0.0,0.0,0.0,0.1667,0.4244,6,2,482,0.0,2,712,0.0,1,690,1.0,google_drive.gdrive_list_folder slack.list_files github.push_files filesystem.search_files cloudflare.cf_purge_cache filesystem.list_directory filesystem.move_file filesystem.directory_tree filesystem.list_directory_with_sizes google_drive.gdrive_search +q009,Remember this fact in the knowledge graph,paraphrase,memory.read_graph,0.0,1.0,1.0,0.5000,0.5672,2,1,156,1.0,1,262,1.0,1,493,1.0,memory.read_graph memory.create_entities memory.create_relations memory.add_observations memory.open_nodes memory.search_nodes memory.delete_relations memory.delete_entities postgres.list_schemas sqlite.sqlite_list_tables +q010,Search the web with Brave,verb_product,brave_search.brave_web_search,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,374,1.0,1,687,1.0,brave_search.brave_web_search brave_search.brave_local_search jira.jira_search_issues memory.search_nodes filesystem.list_directory_with_sizes github.update_pull_request_branch grafana.grafana_search_dashboards github.search_users hubspot.hubspot_search_contacts github.search_orgs +q011,Fetch the contents of a URL,verb_product,fetch.fetch,1.0,1.0,1.0,1.0000,1.0000,1,1,264,1.0,1,380,1.0,1,672,1.0,fetch.fetch filesystem.read_file github.get_file_contents playwright.browser_navigate git.git_add playwright.browser_snapshot slack.slack_set_channel_topic git.git_status playwright.browser_take_screenshot github.get_me +q012,Show git status,identifier,git.git_status,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,299,1.0,1,684,1.0,git.git_status git.git_show git.git_diff_unstaged git.git_diff_staged git.git_log git.git_diff gitlab.list_releases gitlab.list_milestones github.list_milestones gitlab.list_labels +q013,Commit staged changes,paraphrase,git.git_commit,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,327,1.0,1,596,1.0,git.git_commit git.git_reset git.git_diff_staged github.get_commit git.git_show git.git_log github.push_files id_map.json +q014,Create a Linear issue,verb_product,linear.linear_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,293,1.0,1,463,1.0,1,974,1.0,linear.linear_create_issue linear.linear_create_comment linear.linear_assign_issue linear.linear_get_issue linear.linear_update_issue bitbucket.bitbucket_create_issue gitlab.gitlab_create_issue jira.jira_create_issue github.create_issue linear.linear_create_project +q015,Make a task in my project tracker Linear,paraphrase,linear.linear_create_project,0.0,0.0,0.0,0.0000,0.0000,146,49,12453,0.0,30,12193,0.0,15,11833,0.0,linear.linear_create_project twilio.twilio_make_call clickup.clickup_get_task asana.asana_get_task clickup.clickup_update_task asana.asana_update_task clickup.clickup_create_task asana.asana_create_task linear.get_attachment linear.delete_attachment +q016,Post a comment on a GitHub issue,verb_product,github.update_issue_comment,0.0,1.0,1.0,0.3333,0.5000,3,1,307,1.0,1,510,1.0,1,1103,1.0,github.update_issue_comment figma.figma_post_comment github.add_issue_comment github.assign_copilot_to_issue github.issue_read github.add_comment_to_pending_review github.list_repository_collaborators github.create_issue github.get_discussion_comments github.issue_write +q017,Merge the pull request,ambiguous,github.merge_pull_request,1.0,1.0,1.0,1.0000,0.9239,1,1,299,1.0,1,495,1.0,1,1121,1.0,github.merge_pull_request github.update_pull_request_branch gitlab.gitlab_get_merge_request gitlab.gitlab_merge_merge_request gitlab.gitlab_create_merge_request github.pull_request_read github.pull_request_review_write github.update_pull_request github.create_pull_request github.request_copilot_review +q018,Trigger a GitHub Actions workflow,verb_product,github.actions_run_trigger,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,458,1.0,1,733,1.0,github.actions_run_trigger github.actions_get github.actions_list github.actions_get_job_logs circleci.circleci_trigger_pipeline github.search_users github.search_orgs github.search_commits github.search_pull_requests github.star_repository +q019,Get Dependabot alerts,resource,github.list_dependabot_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,367,1.0,1,642,1.0,github.list_dependabot_alerts github.get_dependabot_alert github.list_secret_scanning_alerts github.list_code_scanning_alerts id_map.json discord.discord_get_channel vercel.vercel_get_deployment confluence.confluence_get_page redis.redis_get pagerduty.pagerduty_get_incident +q020,List Sentry issues,verb_product,sentry.sentry_list_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,395,1.0,1,796,1.0,sentry.sentry_list_issues sentry.sentry_list_projects sentry.sentry_list_events gitlab.gitlab_list_issues linear.linear_list_issues github.list_issues sentry.sentry_get_issue sentry.sentry_update_issue linear.linear_search_issues github.search_issues +q021,Create a Stripe customer,verb_product,stripe.stripe_create_customer,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,341,1.0,1,662,1.0,stripe.stripe_create_customer stripe.stripe_create_invoice stripe.stripe_create_payment_intent stripe.stripe_get_customer stripe.stripe_create_product stripe.stripe_create_refund stripe.stripe_create_price stripe.stripe_list_invoices stripe.stripe_list_subscriptions stripe.stripe_get_payment_intent +q022,Refund a Stripe payment,paraphrase,stripe.stripe_create_refund,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,332,1.0,1,629,1.0,stripe.stripe_create_refund stripe.stripe_get_payment_intent stripe.stripe_create_payment_intent stripe.stripe_create_invoice stripe.stripe_get_customer stripe.stripe_cancel_subscription stripe.stripe_list_invoices stripe.stripe_list_customers stripe.stripe_create_product stripe.stripe_list_subscriptions +q023,Send an email with Gmail,verb_product,gmail.gmail_send_email,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,405,1.0,1,716,1.0,gmail.gmail_send_email gmail.gmail_trash_email gmail.gmail_get_email gmail.gmail_modify_email twilio.twilio_send_sms discord.discord_send_message gmail.gmail_list_labels gmail.gmail_search_emails gmail.gmail_create_draft filesystem.list_directory_with_sizes +q024,Search my Gmail inbox,verb_product,gmail.gmail_search_emails,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,315,1.0,1,668,1.0,gmail.gmail_search_emails gmail.gmail_list_labels gmail.gmail_trash_email gmail.gmail_get_email gmail.gmail_create_draft gmail.gmail_send_email gmail.gmail_modify_email grafana.grafana_search_dashboards github.search_users hubspot.hubspot_search_contacts +q025,Upload a file to Google Drive,paraphrase,slack.slack_upload_file,0.0,0.0,1.0,0.2000,0.3869,5,2,523,0.0,1,446,1.0,1,774,1.0,slack.slack_upload_file google_drive.gdrive_get_file google_drive.gdrive_update_file google_drive.gdrive_share_file google_drive.gdrive_create_file google_drive.gdrive_search google_drive.gdrive_list_folder filesystem.edit_file aws_s3.s3_put_object google_calendar.gcal_list_calendars +q026,Create a calendar event,resource,google_calendar.gcal_create_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,427,1.0,1,733,1.0,google_calendar.gcal_create_event google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars github.pull_request_review_write id_map.json trello.trello_create_board stripe.stripe_create_invoice vercel.vercel_create_project +q027,List Kubernetes pods,verb_product,kubernetes.k8s_list_pods,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,303,1.0,1,665,1.0,kubernetes.k8s_list_pods kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_apply kubernetes.k8s_get_pod kubernetes.k8s_scale_deployment kubernetes.k8s_delete kubernetes.k8s_pod_logs id_map.json +q028,Get logs from a k8s pod,verb_product,kubernetes.k8s_pod_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,455,1.0,1,836,1.0,kubernetes.k8s_pod_logs kubernetes.k8s_get_pod github.actions_get_job_logs datadog.datadog_list_logs docker.docker_logs elasticsearch.es_get_doc aws_s3.s3_get_object slack.slack_get_channel_history figma.figma_get_file_nodes gitlab.gitlab_get_file +q029,Run a docker container,verb_product,docker.docker_run,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,345,1.0,1,597,1.0,docker.docker_run docker.docker_exec docker.docker_stop docker.docker_rm docker.docker_logs docker.docker_images docker.docker_ps docker.docker_build github.delete_check_run github.get_check_run +q030,Show docker container logs,verb_product,docker.docker_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,350,1.0,1,759,1.0,docker.docker_logs docker.docker_stop docker.docker_rm docker.docker_exec kubernetes.k8s_pod_logs docker.docker_run git.git_log datadog.datadog_list_logs github.actions_get_job_logs git.git_show +q031,Navigate the browser to a URL,ambiguous,playwright.browser_navigate,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,282,1.0,1,603,1.0,playwright.browser_navigate puppeteer.puppeteer_navigate playwright.browser_tab_new playwright.browser_evaluate playwright.browser_press_key playwright.browser_snapshot playwright.browser_click playwright.browser_wait_for playwright.browser_take_screenshot fetch.fetch +q032,Take a screenshot of the page,ambiguous,playwright.browser_take_screenshot,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,334,1.0,1,711,1.0,playwright.browser_take_screenshot puppeteer.puppeteer_screenshot playwright.browser_snapshot notion.notion_get_page notion.notion_update_page confluence.confluence_get_page confluence.confluence_create_page confluence.confluence_update_page notion.notion_create_page playwright.browser_evaluate +q033,Create a GitLab merge request,verb_product,gitlab.gitlab_create_merge_request,1.0,1.0,1.0,1.0000,1.0000,1,1,297,1.0,1,475,1.0,1,922,1.0,gitlab.gitlab_create_merge_request gitlab.gitlab_get_merge_request gitlab.gitlab_merge_merge_request gitlab.gitlab_list_merge_requests github.merge_pull_request gitlab.gitlab_create_pipeline gitlab.gitlab_create_issue github.create_pull_request git.git_create_branch gitlab.gitlab_list_projects +q034,Search Notion pages,verb_product,notion.notion_search,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,292,1.0,1,558,1.0,notion.notion_search notion.notion_list_users notion.notion_get_block notion.notion_get_page grafana.grafana_search_dashboards github.search_users hubspot.hubspot_search_contacts github.search_orgs confluence.confluence_search github.search_commits +q035,Create a Notion page,verb_product,notion.notion_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,230,1.0,1,407,1.0,1,822,1.0,notion.notion_create_page notion.notion_get_page notion.notion_update_page confluence.confluence_create_page notion.notion_create_database notion.notion_get_comments notion.create_comment notion.create_data_source notion.create_users_token confluence.confluence_get_page +q036,Query a Notion database,verb_product,notion.notion_query_database,1.0,1.0,1.0,1.0000,1.0000,1,1,279,1.0,1,401,1.0,1,669,1.0,notion.notion_query_database notion.notion_create_database notion.notion_create_page notion.notion_search postgres.query notion.notion_list_users notion.notion_get_block notion.notion_get_page notion.notion_get_comments notion.notion_update_page +q037,Create a Zendesk ticket,verb_product,zendesk.zendesk_create_ticket,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,426,1.0,1,646,1.0,zendesk.zendesk_create_ticket zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket jira.jira_create_issue zendesk.zendesk_list_tickets zendesk.zendesk_search id_map.json trello.trello_create_board stripe.stripe_create_invoice vercel.vercel_create_project +q038,Create a HubSpot contact,verb_product,hubspot.hubspot_create_contact,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,899,1.0,hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_create_deal hubspot.hubspot_search_contacts hubspot.hubspot_list_contacts github.create_gist github.create_repository github.create_branch github.create_issue github.create_or_update_file +q039,Create a Salesforce lead,verb_product,salesforce.salesforce_create_lead,1.0,1.0,1.0,1.0000,1.0000,1,1,252,1.0,1,378,1.0,1,661,1.0,salesforce.salesforce_create_lead salesforce.salesforce_convert_lead salesforce.salesforce_create_record salesforce.salesforce_list_opportunities salesforce.salesforce_delete_record salesforce.salesforce_query salesforce.salesforce_update_record salesforce.salesforce_get_record id_map.json trello.trello_create_board +q040,Run a SOQL query,identifier,salesforce.salesforce_query,1.0,1.0,1.0,1.0000,1.0000,1,1,187,1.0,1,293,1.0,1,729,1.0,salesforce.salesforce_query postgres.query mysql.mysql_query sqlite.sqlite_query github.delete_check_run github.get_check_run github.update_check_run docker.docker_run github.create_check_run github.actions_run_trigger +q041,List Shopify orders,verb_product,shopify.shopify_list_orders,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,320,1.0,1,606,1.0,shopify.shopify_list_orders shopify.shopify_list_customers shopify.shopify_list_products shopify.shopify_get_product shopify.shopify_get_order shopify.shopify_update_product shopify.shopify_create_order shopify.shopify_create_product id_map.json clickup.clickup_list_tasks +q042,Create a PagerDuty incident,verb_product,pagerduty.pagerduty_create_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,403,1.0,1,628,1.0,pagerduty.pagerduty_create_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls id_map.json trello.trello_create_board stripe.stripe_create_invoice vercel.vercel_create_project +q043,Who is on call in PagerDuty,paraphrase,pagerduty.pagerduty_list_oncalls,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,363,1.0,1,724,1.0,pagerduty.pagerduty_list_oncalls pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_create_incident twilio.twilio_make_call slack.slack_create_channel filesystem.list_allowed_directories memory.search_nodes +q044,Query Datadog metrics,verb_product,datadog.datadog_query_metrics,1.0,1.0,1.0,1.0000,1.0000,1,1,284,1.0,1,449,1.0,1,767,1.0,datadog.datadog_query_metrics datadog.datadog_list_monitors datadog.datadog_list_logs datadog.datadog_create_monitor prometheus.prom_list_metrics datadog.datadog_get_monitor postgres.query prometheus.prom_query postgres.explain_query prometheus.prom_query_range +q045,Search Datadog logs,verb_product,datadog.datadog_list_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,458,1.0,1,821,1.0,datadog.datadog_list_logs datadog.datadog_list_monitors datadog.datadog_get_monitor datadog.datadog_query_metrics datadog.datadog_create_monitor docker.docker_logs kubernetes.k8s_pod_logs github.actions_get_job_logs grafana.grafana_search_dashboards github.search_users +q046,List S3 buckets,verb_product,aws_s3.s3_list_buckets,1.0,1.0,1.0,1.0000,1.0000,1,1,203,1.0,1,361,1.0,1,523,1.0,aws_s3.s3_list_buckets aws_s3.s3_list_objects aws_s3.s3_delete_object aws_s3.s3_get_object aws_s3.s3_put_object clickup.clickup_list_tasks id_map.json vercel.vercel_list_projects jira.jira_list_projects cloudflare.cf_list_zones +q047,Upload an object to S3,paraphrase,aws_s3.s3_put_object,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,405,1.0,1,820,1.0,aws_s3.s3_put_object aws_s3.s3_delete_object aws_s3.s3_get_object slack.slack_upload_file aws_s3.s3_list_objects github.assign_copilot_to_issue twilio.twilio_send_sms gmail.gmail_send_email slack.slack_add_reaction aws_s3.s3_list_buckets +q048,Purge Cloudflare cache,verb_product,cloudflare.cf_purge_cache,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,286,1.0,1,643,1.0,cloudflare.cf_purge_cache cloudflare.cf_list_zones cloudflare.cf_list_dns cloudflare.cf_create_dns id_map.json +q049,Create a Cloudflare DNS record,verb_product,cloudflare.cf_create_dns,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,356,1.0,1,660,1.0,cloudflare.cf_create_dns salesforce.salesforce_create_record cloudflare.cf_list_dns cloudflare.cf_list_zones cloudflare.cf_purge_cache salesforce.salesforce_delete_record salesforce.salesforce_update_record salesforce.salesforce_get_record id_map.json git.git_commit +q050,List Vercel deployments,verb_product,vercel.vercel_list_deployments,1.0,1.0,1.0,1.0000,1.0000,1,1,173,1.0,1,291,1.0,1,449,1.0,vercel.vercel_list_deployments kubernetes.k8s_list_deployments vercel.vercel_list_projects vercel.vercel_get_deployment vercel.vercel_create_project id_map.json clickup.clickup_list_tasks jira.jira_list_projects cloudflare.cf_list_zones linear.linear_list_teams +q051,Insert a row into Supabase,verb_product,supabase.supabase_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,203,1.0,1,347,1.0,1,598,1.0,supabase.supabase_insert mongodb.mongo_insert supabase.supabase_list_tables supabase.supabase_invoke_fn supabase.supabase_select supabase.supabase_update playwright.browser_type id_map.json +q052,Export images from Figma,paraphrase,figma.figma_get_images,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,328,1.0,1,673,1.0,figma.figma_get_images figma.figma_get_file_nodes docker.docker_images figma.figma_get_file figma.figma_get_comments figma.figma_post_comment memory.delete_observations elasticsearch.es_get_doc datadog.datadog_list_logs memory.delete_relations +q053,Send an SMS with Twilio,verb_product,twilio.twilio_send_sms,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,409,1.0,1,647,1.0,twilio.twilio_send_sms twilio.twilio_list_messages gmail.gmail_send_email twilio.twilio_make_call discord.discord_send_message filesystem.list_directory_with_sizes puppeteer.puppeteer_click puppeteer.puppeteer_fill id_map.json puppeteer.puppeteer_evaluate +q054,Create an Okta user,verb_product,okta.okta_create_user,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,329,1.0,1,743,1.0,okta.okta_create_user okta.okta_deactivate_user okta.okta_get_user linear.create_user okta.okta_list_users asana.asana_create_project asana.asana_create_task github.create_issue slack.slack_get_user_profile linear.get_user +q055,Instant query Prometheus,verb_product,prometheus.prom_query,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,375,1.0,1,680,1.0,prometheus.prom_query prometheus.prom_query_range prometheus.prom_list_metrics postgres.query datadog.datadog_query_metrics postgres.explain_query mysql.mysql_query sqlite.sqlite_query salesforce.salesforce_query notion.notion_query_database +q056,Search Grafana dashboards,verb_product,grafana.grafana_search_dashboards,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,324,1.0,1,605,1.0,grafana.grafana_search_dashboards grafana.grafana_get_dashboard grafana.grafana_create_annotation github.search_users hubspot.hubspot_search_contacts github.search_orgs confluence.confluence_search github.search_commits github.search_pull_requests slack.slack_search_messages +q057,Trigger a CircleCI pipeline,verb_product,circleci.circleci_trigger_pipeline,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,390,1.0,1,664,1.0,circleci.circleci_trigger_pipeline circleci.circleci_get_pipeline circleci.circleci_list_pipelines gitlab.gitlab_create_pipeline github.actions_run_trigger gitlab.get_pipeline gitlab.delete_pipeline gitlab.update_pipeline gitlab.create_pipeline id_map.json +q058,Run terraform plan,identifier,terraform.tf_plan,1.0,1.0,1.0,1.0000,1.0000,1,1,172,1.0,1,280,1.0,1,717,1.0,terraform.tf_plan terraform.tf_apply terraform.tf_state_list github.delete_check_run github.get_check_run github.update_check_run docker.docker_run github.create_check_run github.actions_run_trigger postgres.explain_query +q059,Create an Asana task,verb_product,asana.asana_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,429,1.0,1,825,1.0,asana.asana_create_task asana.asana_get_task asana.asana_update_task asana.asana_create_project clickup.clickup_create_task asana.asana_list_tasks clickup.clickup_get_task clickup.clickup_update_task okta.okta_create_user github.create_issue +q060,Create a Trello card,verb_product,trello.trello_create_card,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,316,1.0,1,615,1.0,trello.trello_create_card trello.trello_create_board trello.trello_add_comment trello.trello_list_cards id_map.json stripe.stripe_create_invoice vercel.vercel_create_project okta.okta_create_user slack.slack_create_channel shopify.shopify_create_order +q061,Create a ClickUp task,verb_product,clickup.clickup_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,415,1.0,1,673,1.0,clickup.clickup_create_task clickup.clickup_get_task clickup.clickup_update_task asana.asana_create_task clickup.clickup_list_tasks asana.asana_get_task asana.asana_update_task puppeteer.puppeteer_click playwright.browser_click id_map.json +q062,Create a Confluence page,verb_product,confluence.confluence_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,401,1.0,1,654,1.0,confluence.confluence_create_page confluence.confluence_get_page confluence.confluence_update_page notion.notion_create_page confluence.confluence_search notion.notion_get_page notion.notion_update_page id_map.json playwright.browser_take_screenshot trello.trello_create_board +q063,Send a Discord message,verb_product,discord.discord_send_message,1.0,1.0,1.0,1.0000,1.0000,1,1,262,1.0,1,404,1.0,1,628,1.0,discord.discord_send_message discord.discord_add_reaction twilio.twilio_send_sms gmail.gmail_send_email discord.discord_get_channel discord.discord_list_channels discord.discord_list_members slack.slack_post_message id_map.json gmail.gmail_trash_email +q064,Search Elasticsearch,verb_product,elasticsearch.es_search,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,348,1.0,1,617,1.0,elasticsearch.es_search elasticsearch.es_get_doc elasticsearch.es_index_doc grafana.grafana_search_dashboards github.search_users hubspot.hubspot_search_contacts github.search_orgs confluence.confluence_search github.search_commits github.search_pull_requests +q065,Get a Redis key,verb_product,redis.redis_get,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,281,1.0,1,626,1.0,redis.redis_get redis.redis_set jira.jira_get_issue redis.redis_delete redis.redis_keys figma.figma_get_file figma.figma_get_comments aws_s3.s3_get_object figma.figma_get_file_nodes figma.figma_get_images +q066,Find MongoDB documents,verb_product,mongodb.mongo_find,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,381,1.0,1,631,1.0,mongodb.mongo_find mongodb.mongo_insert mongodb.mongo_delete mongodb.mongo_update mongodb.mongo_list_collections id_map.json +q067,List SQLite tables,verb_product,sqlite.sqlite_list_tables,1.0,1.0,1.0,1.0000,1.0000,1,1,142,1.0,1,240,1.0,1,417,1.0,sqlite.sqlite_list_tables mysql.mysql_list_tables supabase.supabase_list_tables sqlite.sqlite_describe_table postgres.list_tables sqlite.sqlite_query id_map.json clickup.clickup_list_tasks vercel.vercel_list_projects jira.jira_list_projects +q068,Describe a postgres table,verb_product,postgres.describe_table,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,274,1.0,1,543,1.0,postgres.describe_table sqlite.sqlite_describe_table postgres.explain_query postgres.list_schemas postgres.list_tables postgres.query supabase.supabase_insert supabase.supabase_select supabase.supabase_update id_map.json +q069,github_create_issue,identifier,,0.0,0.0,0.0,0.0000,0.0000,29,10,2762,0.0,6,2686,0.0,3,2629,0.0, +q070,createIssue,identifier,,0.0,0.0,0.0,0.0000,0.0000,29,10,2752,0.0,6,2680,0.0,3,2626,0.0, +q071,list-directory,identifier,,0.0,0.0,0.0,0.0000,0.0000,82,28,7599,0.0,17,7462,0.0,9,7580,0.0, +q072,sequential thinking next thought,paraphrase,sequential_thinking.sequentialthinking,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,291,1.0,1,713,1.0,sequential_thinking.sequentialthinking id_map.json +q073,What time is it in Tokyo,paraphrase,time.get_current_time,1.0,1.0,1.0,1.0000,1.0000,1,1,222,1.0,1,329,1.0,1,633,1.0,time.get_current_time time.convert_time slack.slack_create_channel filesystem.list_allowed_directories playwright.browser_wait_for grafana.grafana_create_annotation google_calendar.gcal_list_events jira.jira_add_worklog id_map.json supabase.supabase_list_tables +q074,Convert time between timezones,verb_product,time.convert_time,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,319,1.0,1,650,1.0,time.convert_time salesforce.salesforce_convert_lead time.get_current_time memory.create_relations git.git_diff playwright.browser_wait_for grafana.grafana_create_annotation google_calendar.gcal_list_events jira.jira_add_worklog id_map.json +q075,Star a GitHub repo,verb_product,github.star_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,226,1.0,1,376,1.0,1,772,1.0,github.star_repository github.unstar_repository github.delete_repository github.actions_list github.get_label github.get_discussion github.search_issues github.fork_repository github.list_releases github.list_labels +q076,Fork a repository,resource,github.fork_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,384,1.0,1,783,1.0,github.fork_repository github.star_repository github.unstar_repository github.delete_repository github.list_repository_collaborators github.create_repository github.get_repository_tree github.list_releases github.list_labels github.list_discussions +q077,Get the latest GitHub release,verb_product,github.get_latest_release,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,331,1.0,1,704,1.0,github.get_latest_release github.get_release_by_tag gitlab.get_release github.get_teams github.get_me github.get_repository_tree github.list_notifications github.projects_get github.get_gist github.get_label +q078,Create a gist,resource,github.create_gist,1.0,1.0,1.0,1.0000,1.0000,1,1,229,1.0,1,334,1.0,1,653,1.0,github.create_gist github.get_gist github.update_gist trello.trello_create_board stripe.stripe_create_invoice vercel.vercel_create_project okta.okta_create_user slack.slack_create_channel shopify.shopify_create_order stripe.stripe_create_product +q079,Search GitHub code,verb_product,github.search_code,1.0,1.0,1.0,1.0000,1.0000,1,1,283,1.0,1,385,1.0,1,725,1.0,github.search_code github.list_code_scanning_alerts github.get_code_scanning_alert github.search_users github.search_orgs github.search_commits github.search_pull_requests github.search_issues github.search_repositories hubspot.hubspot_search_contacts +q080,List GitHub notifications,verb_product,github.list_notifications,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,307,1.0,1,655,1.0,github.list_notifications github.mark_all_notifications_read github.projects_list github.list_gists github.actions_list github.list_starred_repositories github.list_releases github.list_labels github.list_discussions github.list_secret_scanning_alerts +q081,Assign Copilot to an issue,verb_product,github.assign_copilot_to_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,402,1.0,1,875,1.0,github.assign_copilot_to_issue linear.linear_assign_issue jira.jira_assign_issue jira.jira_update_issue jira.jira_transition_issue github.add_issue_comment github.create_issue github.request_copilot_review linear.linear_create_comment jira.jira_add_comment +q082,Get secret scanning alerts,resource,github.list_secret_scanning_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,295,1.0,1,470,1.0,1,681,1.0,github.list_secret_scanning_alerts github.get_secret_scanning_alert github.list_code_scanning_alerts github.get_code_scanning_alert github.list_dependabot_alerts id_map.json discord.discord_get_channel vercel.vercel_get_deployment confluence.confluence_get_page redis.redis_get +q083,Create or update a file on GitHub,verb_product,github.create_or_update_file,1.0,1.0,1.0,1.0000,0.7602,1,1,327,1.0,1,532,1.0,1,1080,1.0,github.create_or_update_file google_drive.gdrive_update_file slack.update_file slack.create_file google_drive.gdrive_create_file github.projects_write github.label_write github.get_file_contents github.issue_write github.update_issue_comment +q084,List collaborators on a repo,resource,github.list_repository_collaborators,1.0,1.0,1.0,1.0000,1.0000,1,1,264,1.0,1,416,1.0,1,776,1.0,github.list_repository_collaborators github.list_commits pagerduty.pagerduty_list_oncalls trello.trello_list_cards bitbucket.bitbucket_list_prs github.actions_list github.list_releases github.list_labels github.list_discussions github.list_secret_scanning_alerts +q085,Reply in a Slack thread,verb_product,slack.slack_reply_to_thread,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,365,1.0,1,642,1.0,slack.slack_reply_to_thread slack.slack_get_thread_replies slack.slack_list_channels slack.delete_file slack.get_canvase slack.delete_canvase slack.delete_usergroup slack.delete_bookmark slack.delete_pin slack.get_reminder +q086,Add an emoji reaction on Slack,verb_product,slack.slack_add_reaction,1.0,1.0,1.0,1.0000,1.0000,1,1,271,1.0,1,433,1.0,1,903,1.0,slack.slack_add_reaction discord.discord_add_reaction trello.trello_add_comment gmail.gmail_modify_email jira.jira_add_comment jira.jira_add_worklog memory.add_observations git.git_add github.add_issue_comment github.add_comment_to_pending_review +q087,Search Slack messages,verb_product,slack.slack_search_messages,1.0,1.0,1.0,1.0000,1.0000,1,1,232,1.0,1,343,1.0,1,609,1.0,slack.slack_search_messages gmail.gmail_search_emails slack.slack_get_channel_history twilio.twilio_list_messages grafana.grafana_search_dashboards github.search_users hubspot.hubspot_search_contacts github.search_orgs confluence.confluence_search github.search_commits +q088,Create a Slack channel,verb_product,slack.slack_create_channel,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,342,1.0,1,751,1.0,slack.slack_create_channel slack.slack_leave_channel slack.slack_join_channel slack.slack_set_channel_topic slack.slack_get_channel_history slack.slack_post_message slack.slack_get_thread_replies slack.slack_reply_to_thread slack.slack_upload_file slack.slack_add_reaction +q089,Transition a Jira issue,verb_product,jira.jira_transition_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,389,1.0,1,778,1.0,jira.jira_transition_issue jira.jira_assign_issue jira.jira_get_issue jira.jira_update_issue jira.jira_create_issue jira.jira_add_comment jira.jira_add_worklog gitlab.gitlab_get_issue sentry.sentry_get_issue sentry.sentry_update_issue +q090,Search Jira with JQL,identifier,jira.jira_search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,318,1.0,1,599,1.0,jira.jira_search_issues filesystem.list_directory_with_sizes grafana.grafana_search_dashboards github.search_users hubspot.hubspot_search_contacts github.search_orgs confluence.confluence_search github.search_commits github.search_pull_requests slack.slack_search_messages +q091,Add a worklog to Jira,verb_product,jira.jira_add_worklog,1.0,1.0,1.0,1.0000,1.0000,1,1,334,1.0,1,478,1.0,1,878,1.0,jira.jira_add_worklog jira.jira_add_comment github.add_comment_to_pending_review discord.discord_add_reaction memory.add_observations git.git_add github.add_issue_comment slack.slack_add_reaction jira.jira_transition_issue trello.trello_add_comment +q092,List Linear teams,verb_product,linear.linear_list_teams,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,295,1.0,1,646,1.0,linear.linear_list_teams linear.linear_list_projects linear.linear_list_issues github.get_teams linear.list_comments linear.list_attachments linear.list_labels linear.list_users linear.list_cycles linear.linear_search_issues +q093,Comment on a Linear issue,verb_product,github.update_issue_comment,0.0,1.0,1.0,0.5000,0.6309,2,1,298,1.0,1,428,1.0,1,886,1.0,github.update_issue_comment linear.linear_create_comment github.add_issue_comment linear.linear_assign_issue linear.linear_get_issue linear.linear_update_issue linear.linear_create_issue jira.jira_add_comment trello.trello_add_comment figma.figma_post_comment +q094,Move a local file,paraphrase,filesystem.move_file,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,358,1.0,1,698,1.0,filesystem.move_file filesystem.read_media_file filesystem.read_file filesystem.read_text_file brave_search.brave_local_search figma.figma_get_file slack.delete_file google_drive.gdrive_get_file google_drive.gdrive_update_file google_drive.gdrive_share_file +q095,Recursively search files by pattern,paraphrase,filesystem.search_files,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,366,1.0,1,694,1.0,filesystem.search_files google_drive.gdrive_search linear.linear_search_issues github.search_repositories slack.list_files github.push_files redis.redis_keys grafana.grafana_search_dashboards github.search_users hubspot.hubspot_search_contacts +q096,Edit a file in place with old and new string,paraphrase,filesystem.edit_file,1.0,1.0,1.0,1.0000,1.0000,1,1,247,1.0,1,358,1.0,1,736,1.0,filesystem.edit_file filesystem.list_directory_with_sizes filesystem.write_file filesystem.move_file playwright.browser_tab_new slack.delete_file slack.get_file slack.update_file slack.create_file google_drive.gdrive_create_file +q097,Read the whole knowledge graph,paraphrase,memory.read_graph,1.0,1.0,1.0,1.0000,1.0000,1,1,170,1.0,1,305,1.0,1,562,1.0,memory.read_graph filesystem.read_media_file filesystem.read_file filesystem.read_text_file memory.delete_relations memory.create_entities memory.create_relations memory.delete_entities memory.add_observations memory.open_nodes +q098,Search memory nodes,verb_product,memory.search_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,202,1.0,1,290,1.0,1,546,1.0,memory.search_nodes memory.open_nodes figma.figma_get_file_nodes memory.read_graph memory.delete_observations memory.delete_relations memory.create_entities memory.create_relations memory.delete_entities memory.add_observations +q099,Show unstaged git diff,verb_product,git.git_diff_unstaged,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,313,1.0,1,557,1.0,git.git_diff_unstaged git.git_diff_staged git.git_diff git.git_show git.git_log git.git_status id_map.json git.git_branch git.git_checkout git.git_create_branch +q100,Stage files with git add,identifier,git.git_add,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,519,1.0,1,923,1.0,git.git_add filesystem.list_directory_with_sizes github.add_issue_comment github.add_comment_to_pending_review github.push_files github.create_gist github.update_gist jira.jira_add_comment discord.discord_add_reaction jira.jira_add_worklog +q101,Create a Shopify product,verb_product,shopify.shopify_create_product,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,377,1.0,1,702,1.0,shopify.shopify_create_product shopify.shopify_get_product shopify.shopify_update_product stripe.stripe_create_product shopify.shopify_create_order stripe.stripe_create_price shopify.shopify_list_customers shopify.shopify_get_order shopify.shopify_list_products shopify.shopify_list_orders +q102,Cancel a Stripe subscription,verb_product,stripe.stripe_cancel_subscription,1.0,1.0,1.0,1.0000,1.0000,1,1,176,1.0,1,275,1.0,1,610,1.0,stripe.stripe_cancel_subscription stripe.stripe_get_payment_intent stripe.stripe_create_invoice stripe.stripe_get_customer stripe.stripe_list_invoices stripe.stripe_list_customers stripe.stripe_create_product stripe.stripe_create_refund stripe.stripe_list_subscriptions stripe.stripe_create_customer +q103,Create a payment intent,resource,stripe.stripe_create_payment_intent,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,291,1.0,1,590,1.0,stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent stripe.stripe_create_refund id_map.json trello.trello_create_board stripe.stripe_create_invoice vercel.vercel_create_project okta.okta_create_user slack.slack_create_channel shopify.shopify_create_order +q104,Acknowledge a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,405,1.0,1,626,1.0,pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls id_map.json +q105,Create a Datadog monitor,verb_product,datadog.datadog_create_monitor,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,459,1.0,1,674,1.0,datadog.datadog_create_monitor datadog.datadog_get_monitor datadog.datadog_list_monitors datadog.datadog_list_logs datadog.datadog_query_metrics id_map.json trello.trello_create_board stripe.stripe_create_invoice vercel.vercel_create_project okta.okta_create_user +q106,Resolve a Sentry issue,paraphrase,sentry.sentry_get_issue,0.0,1.0,1.0,0.5000,0.6309,2,1,251,1.0,1,368,1.0,1,721,1.0,sentry.sentry_get_issue sentry.sentry_update_issue sentry.sentry_list_events pagerduty.pagerduty_resolve_incident sentry.sentry_list_projects sentry.sentry_list_issues gitlab.gitlab_get_issue jira.jira_assign_issue jira.jira_get_issue jira.jira_update_issue +q107,Scale a Kubernetes deployment,verb_product,kubernetes.k8s_scale_deployment,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,338,1.0,1,726,1.0,kubernetes.k8s_scale_deployment vercel.vercel_get_deployment kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_apply kubernetes.k8s_get_pod kubernetes.k8s_list_pods kubernetes.k8s_delete kubernetes.k8s_pod_logs +q108,Apply a Kubernetes manifest,verb_product,kubernetes.k8s_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,308,1.0,1,727,1.0,kubernetes.k8s_apply terraform.tf_apply kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_get_pod kubernetes.k8s_list_pods kubernetes.k8s_scale_deployment kubernetes.k8s_delete kubernetes.k8s_pod_logs +q109,Stop a docker container,verb_product,docker.docker_stop,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,345,1.0,1,587,1.0,docker.docker_stop docker.docker_rm docker.docker_logs docker.docker_exec docker.docker_run docker.docker_images docker.docker_ps docker.docker_build kubernetes.k8s_pod_logs id_map.json +q110,Build a docker image,verb_product,docker.docker_build,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,303,1.0,1,547,1.0,docker.docker_build docker.docker_run docker.docker_images docker.docker_stop docker.docker_ps docker.docker_rm docker.docker_logs docker.docker_exec filesystem.read_media_file id_map.json +q111,Click an element in Playwright,verb_product,playwright.browser_click,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,329,1.0,1,571,1.0,playwright.browser_click puppeteer.puppeteer_click playwright.browser_type playwright.browser_take_screenshot playwright.browser_snapshot clickup.clickup_list_tasks clickup.clickup_get_task playwright.browser_tab_list playwright.browser_tab_close playwright.browser_tab_new +q112,Type text into a browser input,paraphrase,playwright.browser_type,1.0,1.0,1.0,1.0000,0.7602,1,1,252,1.0,1,337,1.0,1,589,1.0,playwright.browser_type playwright.browser_wait_for filesystem.read_text_file playwright.browser_tab_list playwright.browser_tab_close playwright.browser_tab_new playwright.browser_navigate playwright.browser_fill_form playwright.browser_evaluate playwright.browser_press_key +q113,List GitLab projects,verb_product,gitlab.gitlab_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,320,1.0,1,595,1.0,gitlab.gitlab_list_projects gitlab.gitlab_list_issues gitlab.gitlab_list_merge_requests github.projects_list vercel.vercel_list_projects jira.jira_list_projects sentry.sentry_list_projects linear.linear_list_projects gitlab.gitlab_get_issue gitlab.gitlab_create_pipeline +q114,Create a Bitbucket pull request,verb_product,bitbucket.bitbucket_create_pr,1.0,1.0,1.0,1.0000,1.0000,1,1,439,1.0,1,634,1.0,1,1155,1.0,bitbucket.bitbucket_create_pr github.create_pull_request github.pull_request_review_write github.merge_pull_request github.pull_request_read github.update_pull_request_branch github.update_pull_request bitbucket.bitbucket_create_issue gitlab.gitlab_create_merge_request bitbucket.bitbucket_list_prs +q115,Create a Gmail draft,verb_product,gmail.gmail_create_draft,1.0,1.0,1.0,1.0000,1.0000,1,1,284,1.0,1,411,1.0,1,709,1.0,gmail.gmail_create_draft github.create_pull_request gmail.gmail_list_labels gmail.gmail_search_emails gmail.gmail_trash_email gmail.gmail_get_email gmail.gmail_send_email gmail.gmail_modify_email id_map.json trello.trello_create_board +q116,Share a Google Drive file,verb_product,google_drive.gdrive_share_file,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,417,1.0,1,760,1.0,google_drive.gdrive_share_file google_drive.gdrive_get_file google_drive.gdrive_update_file google_drive.gdrive_create_file google_drive.gdrive_search google_drive.gdrive_list_folder google_calendar.gcal_list_calendars google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_list_events +q117,List Google Calendar events,verb_product,google_calendar.gcal_list_events,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,379,1.0,1,718,1.0,google_calendar.gcal_list_events sentry.sentry_list_events google_calendar.gcal_list_calendars google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event kubernetes.k8s_get_events google_drive.gdrive_list_folder google_drive.gdrive_search google_drive.gdrive_get_file +q118,Delete a calendar event,resource,google_calendar.gcal_delete_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,427,1.0,1,675,1.0,google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars redis.redis_delete linear.delete_attachment jira.delete_attachment linear.delete_comment linear.delete_label +q119,List Vercel projects,verb_product,vercel.vercel_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,151,1.0,1,253,1.0,1,550,1.0,vercel.vercel_list_projects vercel.vercel_list_deployments jira.jira_list_projects sentry.sentry_list_projects gitlab.gitlab_list_projects linear.linear_list_projects github.projects_list vercel.vercel_get_deployment vercel.vercel_create_project github.projects_get +q120,Select rows from Supabase,verb_product,supabase.supabase_select,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,354,1.0,1,697,1.0,supabase.supabase_select supabase.supabase_insert supabase.supabase_update supabase.supabase_list_tables supabase.supabase_invoke_fn memory.delete_observations elasticsearch.es_get_doc datadog.datadog_list_logs memory.delete_relations aws_s3.s3_get_object +q121,Post a Figma comment,verb_product,figma.figma_post_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,350,1.0,1,646,1.0,figma.figma_post_comment slack.slack_post_message figma.figma_get_file figma.figma_get_comments figma.figma_get_file_nodes figma.figma_get_images linear.delete_comment notion.get_comment notion.delete_comment linear.get_comment +q122,List Okta users,verb_product,okta.okta_list_users,1.0,1.0,1.0,1.0000,1.0000,1,1,174,1.0,1,327,1.0,1,618,1.0,okta.okta_list_users notion.notion_list_users slack.slack_list_users linear.list_users notion.list_users_tokens slack.slack_get_users okta.okta_create_user okta.okta_deactivate_user okta.okta_get_user pagerduty.pagerduty_list_oncalls +q123,Range query Prometheus,verb_product,prometheus.prom_query_range,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,375,1.0,1,680,1.0,prometheus.prom_query_range prometheus.prom_query prometheus.prom_list_metrics postgres.query datadog.datadog_query_metrics postgres.explain_query mysql.mysql_query sqlite.sqlite_query salesforce.salesforce_query notion.notion_query_database +q124,Create a Grafana annotation,verb_product,grafana.grafana_create_annotation,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,275,1.0,1,574,1.0,grafana.grafana_create_annotation grafana.grafana_search_dashboards grafana.grafana_get_dashboard id_map.json trello.trello_create_board stripe.stripe_create_invoice vercel.vercel_create_project okta.okta_create_user slack.slack_create_channel shopify.shopify_create_order +q125,Apply terraform,identifier,terraform.tf_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,238,1.0,1,595,1.0,terraform.tf_apply kubernetes.k8s_apply terraform.tf_plan terraform.tf_state_list id_map.json +q126,List Asana tasks,verb_product,asana.asana_list_tasks,1.0,1.0,1.0,1.0000,1.0000,1,1,192,1.0,1,352,1.0,1,566,1.0,asana.asana_list_tasks clickup.clickup_list_tasks asana.asana_get_task asana.asana_create_project asana.asana_update_task asana.asana_create_task id_map.json vercel.vercel_list_projects jira.jira_list_projects cloudflare.cf_list_zones +q127,Comment on a Trello card,verb_product,trello.trello_add_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,438,1.0,1,703,1.0,trello.trello_add_comment trello.trello_create_card trello.trello_list_cards figma.figma_post_comment github.update_issue_comment trello.trello_create_board linear.delete_comment notion.get_comment notion.delete_comment linear.get_comment +q128,Update a ClickUp task,verb_product,clickup.clickup_update_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,399,1.0,1,673,1.0,clickup.clickup_update_task clickup.clickup_get_task clickup.clickup_create_task asana.asana_update_task clickup.clickup_list_tasks asana.asana_get_task puppeteer.puppeteer_click playwright.browser_click asana.asana_create_task id_map.json +q129,Search Confluence,verb_product,confluence.confluence_search,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,371,1.0,1,636,1.0,confluence.confluence_search confluence.confluence_get_page confluence.confluence_create_page confluence.confluence_update_page grafana.grafana_search_dashboards github.search_users hubspot.hubspot_search_contacts github.search_orgs github.search_commits github.search_pull_requests +q130,Create a HubSpot deal,verb_product,hubspot.hubspot_create_deal,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,899,1.0,hubspot.hubspot_create_deal hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_search_contacts hubspot.hubspot_list_contacts github.create_gist github.create_repository github.create_branch github.create_issue github.create_or_update_file +q131,List Zendesk tickets,verb_product,zendesk.zendesk_list_tickets,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,367,1.0,1,529,1.0,zendesk.zendesk_list_tickets zendesk.zendesk_search zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_create_ticket id_map.json clickup.clickup_list_tasks vercel.vercel_list_projects jira.jira_list_projects cloudflare.cf_list_zones +q132,List Discord channels,verb_product,discord.discord_list_channels,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,300,1.0,1,513,1.0,discord.discord_list_channels slack.slack_list_channels discord.discord_list_members discord.discord_get_channel discord.discord_send_message discord.discord_add_reaction id_map.json clickup.clickup_list_tasks vercel.vercel_list_projects jira.jira_list_projects +q133,Set a Redis key,verb_product,redis.redis_set,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,302,1.0,1,526,1.0,redis.redis_set redis.redis_get slack.slack_set_channel_topic redis.redis_delete redis.redis_keys playwright.browser_press_key id_map.json jira.jira_get_issue figma.figma_get_file jira.jira_assign_issue +q134,Insert MongoDB documents,verb_product,mongodb.mongo_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,381,1.0,1,607,1.0,mongodb.mongo_insert mongodb.mongo_delete mongodb.mongo_find mongodb.mongo_update mongodb.mongo_list_collections supabase.supabase_insert id_map.json +q135,Run a MySQL query,verb_product,mysql.mysql_query,1.0,1.0,1.0,1.0000,1.0000,1,1,174,1.0,1,281,1.0,1,611,1.0,mysql.mysql_query postgres.query mysql.mysql_list_tables sqlite.sqlite_query postgres.explain_query salesforce.salesforce_query github.delete_check_run github.get_check_run github.update_check_run docker.docker_run +q136,Index a document in Elasticsearch,verb_product,elasticsearch.es_index_doc,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,291,1.0,1,538,1.0,elasticsearch.es_index_doc elasticsearch.es_get_doc elasticsearch.es_search playwright.browser_tab_close id_map.json supabase.supabase_list_tables jira.get_attachment linear.get_attachment linear.delete_attachment gitlab.get_label +q137,Get an S3 object,verb_product,aws_s3.s3_get_object,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,373,1.0,1,634,1.0,aws_s3.s3_get_object aws_s3.s3_delete_object aws_s3.s3_put_object aws_s3.s3_list_objects asana.asana_get_task okta.okta_get_user aws_s3.s3_list_buckets discord.discord_get_channel vercel.vercel_get_deployment confluence.confluence_get_page +q138,List Cloudflare zones,verb_product,cloudflare.cf_list_zones,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,285,1.0,1,482,1.0,cloudflare.cf_list_zones cloudflare.cf_list_dns cloudflare.cf_purge_cache cloudflare.cf_create_dns id_map.json clickup.clickup_list_tasks vercel.vercel_list_projects jira.jira_list_projects linear.linear_list_teams google_calendar.gcal_list_calendars +q139,Create a Stripe invoice,verb_product,stripe.stripe_create_invoice,1.0,1.0,1.0,1.0000,1.0000,1,1,205,1.0,1,353,1.0,1,647,1.0,stripe.stripe_create_invoice stripe.stripe_create_product stripe.stripe_create_refund stripe.stripe_create_customer stripe.stripe_create_price stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent stripe.stripe_get_customer stripe.stripe_cancel_subscription stripe.stripe_list_invoices +q140,List GitHub Actions workflows,verb_product,github.actions_list,1.0,1.0,1.0,1.0000,1.0000,1,1,268,1.0,1,453,1.0,1,772,1.0,github.actions_list github.actions_get_job_logs github.actions_get github.actions_run_trigger github.projects_list github.list_gists github.list_starred_repositories github.list_releases github.list_labels github.list_discussions +q141,Get GitHub Actions job logs,verb_product,github.actions_get_job_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,256,1.0,1,442,1.0,1,769,1.0,github.actions_get_job_logs github.actions_get gitlab.get_job github.actions_list github.actions_run_trigger github.projects_get github.get_gist github.get_teams github.get_label github.get_discussion +q142,Create a GitHub repository,verb_product,github.create_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,333,1.0,1,562,1.0,1,927,1.0,github.create_repository github.create_branch github.create_issue github.create_or_update_file git.git_create_branch github.star_repository github.unstar_repository github.delete_repository github.fork_repository github.list_repository_collaborators +q143,Checkout a git branch,verb_product,git.git_checkout,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,429,1.0,1,1001,1.0,git.git_checkout git.git_create_branch git.git_branch github.create_branch gitlab.gitlab_create_merge_request github.list_commits github.delete_file github.push_files github.create_or_update_file github.update_pull_request_branch +q144,Show git log,identifier,git.git_log,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,299,1.0,1,555,1.0,git.git_log git.git_show git.git_diff_unstaged git.git_diff_staged git.git_status git.git_diff git.git_branch git.git_checkout id_map.json git.git_create_branch +q145,List postgres schemas,verb_product,postgres.list_schemas,1.0,1.0,1.0,1.0000,1.0000,1,1,160,1.0,1,277,1.0,1,439,1.0,postgres.list_schemas postgres.list_tables postgres.explain_query postgres.query postgres.describe_table id_map.json clickup.clickup_list_tasks vercel.vercel_list_projects jira.jira_list_projects cloudflare.cf_list_zones +q146,Open nodes in the knowledge graph,verb_product,memory.open_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,163,1.0,1,262,1.0,1,531,1.0,memory.open_nodes memory.search_nodes memory.read_graph memory.create_entities memory.create_relations memory.add_observations memory.delete_relations memory.delete_entities figma.figma_get_file_nodes playwright.browser_tab_list +q147,Get file metadata from the filesystem,paraphrase,filesystem.get_file_info,1.0,1.0,1.0,1.0000,1.0000,1,1,209,1.0,1,407,1.0,1,715,1.0,filesystem.get_file_info google_drive.gdrive_get_file figma.figma_get_file_nodes gitlab.gitlab_get_file github.get_file_contents filesystem.read_media_file filesystem.read_file filesystem.read_text_file figma.figma_get_file slack.get_file +q148,Write a new local file,paraphrase,filesystem.write_file,1.0,1.0,1.0,1.0000,1.0000,1,1,206,1.0,1,384,1.0,1,846,1.0,filesystem.write_file filesystem.read_media_file filesystem.read_file filesystem.read_text_file filesystem.edit_file brave_search.brave_local_search github.issue_write playwright.browser_tab_new github.projects_write github.label_write +q149,Search GitHub issues,verb_product,github.search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,233,1.0,1,385,1.0,1,702,1.0,github.search_issues linear.linear_search_issues jira.jira_search_issues github.list_issues github.search_users github.search_orgs github.search_commits github.search_pull_requests github.search_code github.search_repositories +q150,Get the authenticated GitHub user,paraphrase,github.get_teams,0.0,1.0,1.0,0.5000,0.6309,2,1,179,1.0,1,288,1.0,1,622,1.0,github.get_teams github.get_me github.list_notifications slack.slack_get_user_profile okta.okta_get_user linear.get_user github.get_latest_release github.get_repository_tree github.list_gists github.list_starred_repositories diff --git a/experiments/mcp-tool-search/results/codedb_search.json b/experiments/mcp-tool-search/results/codedb_search.json new file mode 100644 index 00000000..fbe137f5 --- /dev/null +++ b/experiments/mcp-tool-search/results/codedb_search.json @@ -0,0 +1,6948 @@ +{ + "summary": { + "n": 150, + "index_s": 0.44171834199994464, + "rss_mb": null, + "p50_ms": 10.278972499975225, + "p95_ms": 10.414098049938048, + "recall@1": 0.9133333333333333, + "recall@1_lo": 0.8666666666666667, + "recall@1_hi": 0.96, + "recall@3": 0.96, + "recall@3_lo": 0.9266666666666666, + "recall@3_hi": 0.9866666666666667, + "recall@5": 0.9666666666666667, + "recall@5_lo": 0.9333333333333333, + "recall@5_hi": 0.9933333333333333, + "mrr@10": 0.938, + "mrr@10_lo": 0.9011111111111111, + "mrr@10_hi": 0.97, + "ndcg@10": 0.9427449090658193, + "ndcg@10_lo": 0.9091698969544683, + "ndcg@10_hi": 0.9714351410870373 + }, + "meta": { + "name": "codedb_search", + "needs_model": false, + "needs_api": false, + "notes": "codedb_search via MCP (multi-word \u2192 searchContentRanked BM25+; single token \u2192 trigram substring)" + }, + "rows": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "gitlab.gitlab_create_issue", + "bitbucket.bitbucket_create_issue", + "linear.linear_create_issue", + "jira.jira_create_issue", + "github.issue_write", + "git.git_create_branch", + "github.issue_read", + "github.add_issue_comment", + "github.update_issue_comment" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8955796649110442, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 335, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 582, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1111, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "kind": "verb_product", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_create_issue", + "zendesk.zendesk_create_ticket", + "jira.jira_create_sprint", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "jira.create_attachment", + "jira.create_webhook", + "jira.create_sprint", + "jira.create_filter", + "jira.create_version" + ], + "hit1": "jira.jira_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 481, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 966, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "kind": "verb_product", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "slack.slack_post_message", + "slack.slack_add_reaction", + "twilio.twilio_send_sms", + "gmail.gmail_send_email", + "slack.slack_leave_channel", + "slack.slack_join_channel", + "slack.slack_get_user_profile", + "slack.slack_search_messages", + "slack.slack_create_channel" + ], + "hit1": "discord.discord_send_message", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "kind": "resource", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "ranking": [ + "github.list_pull_requests", + "github.search_pull_requests", + "gitlab.gitlab_list_merge_requests", + "bitbucket.bitbucket_list_prs", + "github.create_pull_request", + "playwright.browser_tab_list", + "memory.open_nodes", + "github.merge_pull_request", + "github.pull_request_read", + "github.update_pull_request_branch" + ], + "hit1": "github.list_pull_requests", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 460, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 836, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "ranking": [ + "github.create_pull_request", + "github.merge_pull_request", + "github.pull_request_read", + "github.pull_request_review_write", + "github.update_pull_request", + "github.update_pull_request_branch", + "github.request_copilot_review", + "github.add_comment_to_pending_review", + "github.search_pull_requests", + "github.list_pull_requests" + ], + "hit1": "github.create_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 368, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 653, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1144, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "kind": "verb_product", + "gold": [ + "postgres.query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "mysql.mysql_query", + "sqlite.sqlite_query", + "postgres.explain_query", + "salesforce.salesforce_query", + "postgres.list_schemas", + "postgres.list_tables", + "postgres.describe_table", + "github.delete_check_run", + "github.get_check_run" + ], + "hit1": "postgres.query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 553, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "kind": "paraphrase", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "ranking": [ + "filesystem.read_media_file", + "filesystem.read_file", + "filesystem.read_text_file", + "figma.figma_get_file_nodes", + "gitlab.gitlab_get_file", + "github.delete_file", + "github.get_file_contents", + "brave_search.brave_local_search", + "memory.read_graph", + "github.mark_all_notifications_read" + ], + "hit1": "filesystem.read_media_file", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.66967181649423, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "kind": "paraphrase", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "ranking": [ + "google_drive.gdrive_list_folder", + "slack.list_files", + "github.push_files", + "filesystem.search_files", + "cloudflare.cf_purge_cache", + "filesystem.list_directory", + "filesystem.move_file", + "filesystem.directory_tree", + "filesystem.list_directory_with_sizes", + "google_drive.gdrive_search" + ], + "hit1": "google_drive.gdrive_list_folder", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.16666666666666666, + "ndcg@10": 0.424445564500406, + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 482, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 712, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 690, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "ranking": [ + "memory.read_graph", + "memory.create_entities", + "memory.create_relations", + "memory.add_observations", + "memory.open_nodes", + "memory.search_nodes", + "memory.delete_relations", + "memory.delete_entities", + "postgres.list_schemas", + "sqlite.sqlite_list_tables" + ], + "hit1": "memory.read_graph", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.5672074169568709, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 156, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 493, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "kind": "verb_product", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "ranking": [ + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "jira.jira_search_issues", + "memory.search_nodes", + "filesystem.list_directory_with_sizes", + "github.update_pull_request_branch", + "grafana.grafana_search_dashboards", + "github.search_users", + "hubspot.hubspot_search_contacts", + "github.search_orgs" + ], + "hit1": "brave_search.brave_web_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 374, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "kind": "verb_product", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "ranking": [ + "fetch.fetch", + "filesystem.read_file", + "github.get_file_contents", + "playwright.browser_navigate", + "git.git_add", + "playwright.browser_snapshot", + "slack.slack_set_channel_topic", + "git.git_status", + "playwright.browser_take_screenshot", + "github.get_me" + ], + "hit1": "fetch.fetch", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 264, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 672, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "kind": "identifier", + "gold": [ + "git.git_status" + ], + "alts": [], + "ranking": [ + "git.git_status", + "git.git_show", + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_log", + "git.git_diff", + "gitlab.list_releases", + "gitlab.list_milestones", + "github.list_milestones", + "gitlab.list_labels" + ], + "hit1": "git.git_status", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "kind": "paraphrase", + "gold": [ + "git.git_commit" + ], + "alts": [], + "ranking": [ + "git.git_commit", + "git.git_reset", + "git.git_diff_staged", + "github.get_commit", + "git.git_show", + "git.git_log", + "github.push_files", + "id_map.json" + ], + "hit1": "git.git_commit", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 596, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_issue", + "linear.linear_create_comment", + "linear.linear_assign_issue", + "linear.linear_get_issue", + "linear.linear_update_issue", + "bitbucket.bitbucket_create_issue", + "gitlab.gitlab_create_issue", + "jira.jira_create_issue", + "github.create_issue", + "linear.linear_create_project" + ], + "hit1": "linear.linear_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 974, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "kind": "paraphrase", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_project", + "twilio.twilio_make_call", + "clickup.clickup_get_task", + "asana.asana_get_task", + "clickup.clickup_update_task", + "asana.asana_update_task", + "clickup.clickup_create_task", + "asana.asana_create_task", + "linear.get_attachment", + "linear.delete_attachment" + ], + "hit1": "linear.linear_create_project", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 146, + "k3_calls": 49, + "k3_tokens": 12453, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 30, + "k5_tokens": 12193, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11833, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "ranking": [ + "github.update_issue_comment", + "figma.figma_post_comment", + "github.add_issue_comment", + "github.assign_copilot_to_issue", + "github.issue_read", + "github.add_comment_to_pending_review", + "github.list_repository_collaborators", + "github.create_issue", + "github.get_discussion_comments", + "github.issue_write" + ], + "hit1": "github.update_issue_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 307, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 510, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1103, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "kind": "ambiguous", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "ranking": [ + "github.merge_pull_request", + "github.update_pull_request_branch", + "gitlab.gitlab_get_merge_request", + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_create_merge_request", + "github.pull_request_read", + "github.pull_request_review_write", + "github.update_pull_request", + "github.create_pull_request", + "github.request_copilot_review" + ], + "hit1": "github.merge_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9238850086262381, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 299, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 495, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1121, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "kind": "verb_product", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "ranking": [ + "github.actions_run_trigger", + "github.actions_get", + "github.actions_list", + "github.actions_get_job_logs", + "circleci.circleci_trigger_pipeline", + "github.search_users", + "github.search_orgs", + "github.search_commits", + "github.search_pull_requests", + "github.star_repository" + ], + "hit1": "github.actions_run_trigger", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "kind": "resource", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "ranking": [ + "github.list_dependabot_alerts", + "github.get_dependabot_alert", + "github.list_secret_scanning_alerts", + "github.list_code_scanning_alerts", + "id_map.json", + "discord.discord_get_channel", + "vercel.vercel_get_deployment", + "confluence.confluence_get_page", + "redis.redis_get", + "pagerduty.pagerduty_get_incident" + ], + "hit1": "github.list_dependabot_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 642, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "kind": "verb_product", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_issues", + "sentry.sentry_list_projects", + "sentry.sentry_list_events", + "gitlab.gitlab_list_issues", + "linear.linear_list_issues", + "github.list_issues", + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "linear.linear_search_issues", + "github.search_issues" + ], + "hit1": "sentry.sentry_list_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 395, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_customer", + "stripe.stripe_create_invoice", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_customer", + "stripe.stripe_create_product", + "stripe.stripe_create_refund", + "stripe.stripe_create_price", + "stripe.stripe_list_invoices", + "stripe.stripe_list_subscriptions", + "stripe.stripe_get_payment_intent" + ], + "hit1": "stripe.stripe_create_customer", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "kind": "paraphrase", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_refund", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_invoice", + "stripe.stripe_get_customer", + "stripe.stripe_cancel_subscription", + "stripe.stripe_list_invoices", + "stripe.stripe_list_customers", + "stripe.stripe_create_product", + "stripe.stripe_list_subscriptions" + ], + "hit1": "stripe.stripe_create_refund", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 332, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 629, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "kind": "verb_product", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "ranking": [ + "gmail.gmail_send_email", + "gmail.gmail_trash_email", + "gmail.gmail_get_email", + "gmail.gmail_modify_email", + "twilio.twilio_send_sms", + "discord.discord_send_message", + "gmail.gmail_list_labels", + "gmail.gmail_search_emails", + "gmail.gmail_create_draft", + "filesystem.list_directory_with_sizes" + ], + "hit1": "gmail.gmail_send_email", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 716, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "kind": "verb_product", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "ranking": [ + "gmail.gmail_search_emails", + "gmail.gmail_list_labels", + "gmail.gmail_trash_email", + "gmail.gmail_get_email", + "gmail.gmail_create_draft", + "gmail.gmail_send_email", + "gmail.gmail_modify_email", + "grafana.grafana_search_dashboards", + "github.search_users", + "hubspot.hubspot_search_contacts" + ], + "hit1": "gmail.gmail_search_emails", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 668, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "kind": "paraphrase", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "ranking": [ + "slack.slack_upload_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_update_file", + "google_drive.gdrive_share_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "filesystem.edit_file", + "aws_s3.s3_put_object", + "google_calendar.gcal_list_calendars" + ], + "hit1": "slack.slack_upload_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.2, + "ndcg@10": 0.38685280723454163, + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 523, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 446, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_create_event", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "github.pull_request_review_write", + "id_map.json", + "trello.trello_create_board", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project" + ], + "hit1": "google_calendar.gcal_create_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_list_pods", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_apply", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_delete", + "kubernetes.k8s_pod_logs", + "id_map.json" + ], + "hit1": "kubernetes.k8s_list_pods", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_get_pod", + "github.actions_get_job_logs", + "datadog.datadog_list_logs", + "docker.docker_logs", + "elasticsearch.es_get_doc", + "aws_s3.s3_get_object", + "slack.slack_get_channel_history", + "figma.figma_get_file_nodes", + "gitlab.gitlab_get_file" + ], + "hit1": "kubernetes.k8s_pod_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 836, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "ranking": [ + "docker.docker_run", + "docker.docker_exec", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_images", + "docker.docker_ps", + "docker.docker_build", + "github.delete_check_run", + "github.get_check_run" + ], + "hit1": "docker.docker_run", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "kind": "verb_product", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "ranking": [ + "docker.docker_logs", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_exec", + "kubernetes.k8s_pod_logs", + "docker.docker_run", + "git.git_log", + "datadog.datadog_list_logs", + "github.actions_get_job_logs", + "git.git_show" + ], + "hit1": "docker.docker_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "kind": "ambiguous", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "ranking": [ + "playwright.browser_navigate", + "puppeteer.puppeteer_navigate", + "playwright.browser_tab_new", + "playwright.browser_evaluate", + "playwright.browser_press_key", + "playwright.browser_snapshot", + "playwright.browser_click", + "playwright.browser_wait_for", + "playwright.browser_take_screenshot", + "fetch.fetch" + ], + "hit1": "playwright.browser_navigate", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 282, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 603, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "kind": "ambiguous", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "ranking": [ + "playwright.browser_take_screenshot", + "puppeteer.puppeteer_screenshot", + "playwright.browser_snapshot", + "notion.notion_get_page", + "notion.notion_update_page", + "confluence.confluence_get_page", + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "notion.notion_create_page", + "playwright.browser_evaluate" + ], + "hit1": "playwright.browser_take_screenshot", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 711, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_create_merge_request", + "gitlab.gitlab_get_merge_request", + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_list_merge_requests", + "github.merge_pull_request", + "gitlab.gitlab_create_pipeline", + "gitlab.gitlab_create_issue", + "github.create_pull_request", + "git.git_create_branch", + "gitlab.gitlab_list_projects" + ], + "hit1": "gitlab.gitlab_create_merge_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 922, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "kind": "verb_product", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "ranking": [ + "notion.notion_search", + "notion.notion_list_users", + "notion.notion_get_block", + "notion.notion_get_page", + "grafana.grafana_search_dashboards", + "github.search_users", + "hubspot.hubspot_search_contacts", + "github.search_orgs", + "confluence.confluence_search", + "github.search_commits" + ], + "hit1": "notion.notion_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 558, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "kind": "verb_product", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "ranking": [ + "notion.notion_create_page", + "notion.notion_get_page", + "notion.notion_update_page", + "confluence.confluence_create_page", + "notion.notion_create_database", + "notion.notion_get_comments", + "notion.create_comment", + "notion.create_data_source", + "notion.create_users_token", + "confluence.confluence_get_page" + ], + "hit1": "notion.notion_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 822, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "kind": "verb_product", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "ranking": [ + "notion.notion_query_database", + "notion.notion_create_database", + "notion.notion_create_page", + "notion.notion_search", + "postgres.query", + "notion.notion_list_users", + "notion.notion_get_block", + "notion.notion_get_page", + "notion.notion_get_comments", + "notion.notion_update_page" + ], + "hit1": "notion.notion_query_database", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 279, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "jira.jira_create_issue", + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_search", + "id_map.json", + "trello.trello_create_board", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project" + ], + "hit1": "zendesk.zendesk_create_ticket", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 426, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 646, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_create_deal", + "hubspot.hubspot_search_contacts", + "hubspot.hubspot_list_contacts", + "github.create_gist", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.create_or_update_file" + ], + "hit1": "hubspot.hubspot_create_contact", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 899, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "kind": "verb_product", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_create_lead", + "salesforce.salesforce_convert_lead", + "salesforce.salesforce_create_record", + "salesforce.salesforce_list_opportunities", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_query", + "salesforce.salesforce_update_record", + "salesforce.salesforce_get_record", + "id_map.json", + "trello.trello_create_board" + ], + "hit1": "salesforce.salesforce_create_lead", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 661, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "kind": "identifier", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_query", + "postgres.query", + "mysql.mysql_query", + "sqlite.sqlite_query", + "github.delete_check_run", + "github.get_check_run", + "github.update_check_run", + "docker.docker_run", + "github.create_check_run", + "github.actions_run_trigger" + ], + "hit1": "salesforce.salesforce_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 187, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 293, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "kind": "verb_product", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "ranking": [ + "shopify.shopify_list_orders", + "shopify.shopify_list_customers", + "shopify.shopify_list_products", + "shopify.shopify_get_product", + "shopify.shopify_get_order", + "shopify.shopify_update_product", + "shopify.shopify_create_order", + "shopify.shopify_create_product", + "id_map.json", + "clickup.clickup_list_tasks" + ], + "hit1": "shopify.shopify_list_orders", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "id_map.json", + "trello.trello_create_board", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project" + ], + "hit1": "pagerduty.pagerduty_create_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 628, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "kind": "paraphrase", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_list_oncalls", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_create_incident", + "twilio.twilio_make_call", + "slack.slack_create_channel", + "filesystem.list_allowed_directories", + "memory.search_nodes" + ], + "hit1": "pagerduty.pagerduty_list_oncalls", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "kind": "verb_product", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "ranking": [ + "datadog.datadog_query_metrics", + "datadog.datadog_list_monitors", + "datadog.datadog_list_logs", + "datadog.datadog_create_monitor", + "prometheus.prom_list_metrics", + "datadog.datadog_get_monitor", + "postgres.query", + "prometheus.prom_query", + "postgres.explain_query", + "prometheus.prom_query_range" + ], + "hit1": "datadog.datadog_query_metrics", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 284, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 449, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 767, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "kind": "verb_product", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "ranking": [ + "datadog.datadog_list_logs", + "datadog.datadog_list_monitors", + "datadog.datadog_get_monitor", + "datadog.datadog_query_metrics", + "datadog.datadog_create_monitor", + "docker.docker_logs", + "kubernetes.k8s_pod_logs", + "github.actions_get_job_logs", + "grafana.grafana_search_dashboards", + "github.search_users" + ], + "hit1": "datadog.datadog_list_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 821, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_list_buckets", + "aws_s3.s3_list_objects", + "aws_s3.s3_delete_object", + "aws_s3.s3_get_object", + "aws_s3.s3_put_object", + "clickup.clickup_list_tasks", + "id_map.json", + "vercel.vercel_list_projects", + "jira.jira_list_projects", + "cloudflare.cf_list_zones" + ], + "hit1": "aws_s3.s3_list_buckets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 203, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 361, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 523, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "kind": "paraphrase", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_put_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_get_object", + "slack.slack_upload_file", + "aws_s3.s3_list_objects", + "github.assign_copilot_to_issue", + "twilio.twilio_send_sms", + "gmail.gmail_send_email", + "slack.slack_add_reaction", + "aws_s3.s3_list_buckets" + ], + "hit1": "aws_s3.s3_put_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 820, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "cloudflare.cf_list_dns", + "cloudflare.cf_create_dns", + "id_map.json" + ], + "hit1": "cloudflare.cf_purge_cache", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 286, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 643, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_create_dns", + "salesforce.salesforce_create_record", + "cloudflare.cf_list_dns", + "cloudflare.cf_list_zones", + "cloudflare.cf_purge_cache", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_update_record", + "salesforce.salesforce_get_record", + "id_map.json", + "git.git_commit" + ], + "hit1": "cloudflare.cf_create_dns", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_deployments", + "kubernetes.k8s_list_deployments", + "vercel.vercel_list_projects", + "vercel.vercel_get_deployment", + "vercel.vercel_create_project", + "id_map.json", + "clickup.clickup_list_tasks", + "jira.jira_list_projects", + "cloudflare.cf_list_zones", + "linear.linear_list_teams" + ], + "hit1": "vercel.vercel_list_deployments", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 173, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 449, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "ranking": [ + "supabase.supabase_insert", + "mongodb.mongo_insert", + "supabase.supabase_list_tables", + "supabase.supabase_invoke_fn", + "supabase.supabase_select", + "supabase.supabase_update", + "playwright.browser_type", + "id_map.json" + ], + "hit1": "supabase.supabase_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 203, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 347, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 598, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "kind": "paraphrase", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "ranking": [ + "figma.figma_get_images", + "figma.figma_get_file_nodes", + "docker.docker_images", + "figma.figma_get_file", + "figma.figma_get_comments", + "figma.figma_post_comment", + "memory.delete_observations", + "elasticsearch.es_get_doc", + "datadog.datadog_list_logs", + "memory.delete_relations" + ], + "hit1": "figma.figma_get_images", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "kind": "verb_product", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "ranking": [ + "twilio.twilio_send_sms", + "twilio.twilio_list_messages", + "gmail.gmail_send_email", + "twilio.twilio_make_call", + "discord.discord_send_message", + "filesystem.list_directory_with_sizes", + "puppeteer.puppeteer_click", + "puppeteer.puppeteer_fill", + "id_map.json", + "puppeteer.puppeteer_evaluate" + ], + "hit1": "twilio.twilio_send_sms", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "kind": "verb_product", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "ranking": [ + "okta.okta_create_user", + "okta.okta_deactivate_user", + "okta.okta_get_user", + "linear.create_user", + "okta.okta_list_users", + "asana.asana_create_project", + "asana.asana_create_task", + "github.create_issue", + "slack.slack_get_user_profile", + "linear.get_user" + ], + "hit1": "okta.okta_create_user", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 743, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query", + "prometheus.prom_query_range", + "prometheus.prom_list_metrics", + "postgres.query", + "datadog.datadog_query_metrics", + "postgres.explain_query", + "mysql.mysql_query", + "sqlite.sqlite_query", + "salesforce.salesforce_query", + "notion.notion_query_database" + ], + "hit1": "prometheus.prom_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "kind": "verb_product", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "ranking": [ + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "grafana.grafana_create_annotation", + "github.search_users", + "hubspot.hubspot_search_contacts", + "github.search_orgs", + "confluence.confluence_search", + "github.search_commits", + "github.search_pull_requests", + "slack.slack_search_messages" + ], + "hit1": "grafana.grafana_search_dashboards", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 605, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "kind": "verb_product", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "ranking": [ + "circleci.circleci_trigger_pipeline", + "circleci.circleci_get_pipeline", + "circleci.circleci_list_pipelines", + "gitlab.gitlab_create_pipeline", + "github.actions_run_trigger", + "gitlab.get_pipeline", + "gitlab.delete_pipeline", + "gitlab.update_pipeline", + "gitlab.create_pipeline", + "id_map.json" + ], + "hit1": "circleci.circleci_trigger_pipeline", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 664, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "kind": "identifier", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "ranking": [ + "terraform.tf_plan", + "terraform.tf_apply", + "terraform.tf_state_list", + "github.delete_check_run", + "github.get_check_run", + "github.update_check_run", + "docker.docker_run", + "github.create_check_run", + "github.actions_run_trigger", + "postgres.explain_query" + ], + "hit1": "terraform.tf_plan", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "kind": "verb_product", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "ranking": [ + "asana.asana_create_task", + "asana.asana_get_task", + "asana.asana_update_task", + "asana.asana_create_project", + "clickup.clickup_create_task", + "asana.asana_list_tasks", + "clickup.clickup_get_task", + "clickup.clickup_update_task", + "okta.okta_create_user", + "github.create_issue" + ], + "hit1": "asana.asana_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "ranking": [ + "trello.trello_create_card", + "trello.trello_create_board", + "trello.trello_add_comment", + "trello.trello_list_cards", + "id_map.json", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project", + "okta.okta_create_user", + "slack.slack_create_channel", + "shopify.shopify_create_order" + ], + "hit1": "trello.trello_create_card", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 615, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_create_task", + "clickup.clickup_get_task", + "clickup.clickup_update_task", + "asana.asana_create_task", + "clickup.clickup_list_tasks", + "asana.asana_get_task", + "asana.asana_update_task", + "puppeteer.puppeteer_click", + "playwright.browser_click", + "id_map.json" + ], + "hit1": "clickup.clickup_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 415, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "kind": "verb_product", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "ranking": [ + "confluence.confluence_create_page", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "notion.notion_create_page", + "confluence.confluence_search", + "notion.notion_get_page", + "notion.notion_update_page", + "id_map.json", + "playwright.browser_take_screenshot", + "trello.trello_create_board" + ], + "hit1": "confluence.confluence_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 654, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "kind": "verb_product", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "discord.discord_add_reaction", + "twilio.twilio_send_sms", + "gmail.gmail_send_email", + "discord.discord_get_channel", + "discord.discord_list_channels", + "discord.discord_list_members", + "slack.slack_post_message", + "id_map.json", + "gmail.gmail_trash_email" + ], + "hit1": "discord.discord_send_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 628, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_search", + "elasticsearch.es_get_doc", + "elasticsearch.es_index_doc", + "grafana.grafana_search_dashboards", + "github.search_users", + "hubspot.hubspot_search_contacts", + "github.search_orgs", + "confluence.confluence_search", + "github.search_commits", + "github.search_pull_requests" + ], + "hit1": "elasticsearch.es_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 617, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "redis.redis_set", + "jira.jira_get_issue", + "redis.redis_delete", + "redis.redis_keys", + "figma.figma_get_file", + "figma.figma_get_comments", + "aws_s3.s3_get_object", + "figma.figma_get_file_nodes", + "figma.figma_get_images" + ], + "hit1": "redis.redis_get", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 626, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_find", + "mongodb.mongo_insert", + "mongodb.mongo_delete", + "mongodb.mongo_update", + "mongodb.mongo_list_collections", + "id_map.json" + ], + "hit1": "mongodb.mongo_find", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 631, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "kind": "verb_product", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "ranking": [ + "sqlite.sqlite_list_tables", + "mysql.mysql_list_tables", + "supabase.supabase_list_tables", + "sqlite.sqlite_describe_table", + "postgres.list_tables", + "sqlite.sqlite_query", + "id_map.json", + "clickup.clickup_list_tasks", + "vercel.vercel_list_projects", + "jira.jira_list_projects" + ], + "hit1": "sqlite.sqlite_list_tables", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 142, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 240, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 417, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "kind": "verb_product", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "ranking": [ + "postgres.describe_table", + "sqlite.sqlite_describe_table", + "postgres.explain_query", + "postgres.list_schemas", + "postgres.list_tables", + "postgres.query", + "supabase.supabase_insert", + "supabase.supabase_select", + "supabase.supabase_update", + "id_map.json" + ], + "hit1": "postgres.describe_table", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 274, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 543, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 29, + "k3_calls": 10, + "k3_tokens": 2762, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 6, + "k5_tokens": 2686, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 3, + "k10_tokens": 2629, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 29, + "k3_calls": 10, + "k3_tokens": 2752, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 6, + "k5_tokens": 2680, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 3, + "k10_tokens": 2626, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "kind": "identifier", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 82, + "k3_calls": 28, + "k3_tokens": 7599, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 17, + "k5_tokens": 7462, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7580, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "kind": "paraphrase", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "ranking": [ + "sequential_thinking.sequentialthinking", + "id_map.json" + ], + "hit1": "sequential_thinking.sequentialthinking", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "kind": "paraphrase", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "ranking": [ + "time.get_current_time", + "time.convert_time", + "slack.slack_create_channel", + "filesystem.list_allowed_directories", + "playwright.browser_wait_for", + "grafana.grafana_create_annotation", + "google_calendar.gcal_list_events", + "jira.jira_add_worklog", + "id_map.json", + "supabase.supabase_list_tables" + ], + "hit1": "time.get_current_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 222, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 633, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "kind": "verb_product", + "gold": [ + "time.convert_time" + ], + "alts": [], + "ranking": [ + "time.convert_time", + "salesforce.salesforce_convert_lead", + "time.get_current_time", + "memory.create_relations", + "git.git_diff", + "playwright.browser_wait_for", + "grafana.grafana_create_annotation", + "google_calendar.gcal_list_events", + "jira.jira_add_worklog", + "id_map.json" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 650, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "kind": "verb_product", + "gold": [ + "github.star_repository" + ], + "alts": [], + "ranking": [ + "github.star_repository", + "github.unstar_repository", + "github.delete_repository", + "github.actions_list", + "github.get_label", + "github.get_discussion", + "github.search_issues", + "github.fork_repository", + "github.list_releases", + "github.list_labels" + ], + "hit1": "github.star_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 376, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 772, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "kind": "resource", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "ranking": [ + "github.fork_repository", + "github.star_repository", + "github.unstar_repository", + "github.delete_repository", + "github.list_repository_collaborators", + "github.create_repository", + "github.get_repository_tree", + "github.list_releases", + "github.list_labels", + "github.list_discussions" + ], + "hit1": "github.fork_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 783, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "kind": "verb_product", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "ranking": [ + "github.get_latest_release", + "github.get_release_by_tag", + "gitlab.get_release", + "github.get_teams", + "github.get_me", + "github.get_repository_tree", + "github.list_notifications", + "github.projects_get", + "github.get_gist", + "github.get_label" + ], + "hit1": "github.get_latest_release", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "kind": "resource", + "gold": [ + "github.create_gist" + ], + "alts": [], + "ranking": [ + "github.create_gist", + "github.get_gist", + "github.update_gist", + "trello.trello_create_board", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project", + "okta.okta_create_user", + "slack.slack_create_channel", + "shopify.shopify_create_order", + "stripe.stripe_create_product" + ], + "hit1": "github.create_gist", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 653, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "kind": "verb_product", + "gold": [ + "github.search_code" + ], + "alts": [], + "ranking": [ + "github.search_code", + "github.list_code_scanning_alerts", + "github.get_code_scanning_alert", + "github.search_users", + "github.search_orgs", + "github.search_commits", + "github.search_pull_requests", + "github.search_issues", + "github.search_repositories", + "hubspot.hubspot_search_contacts" + ], + "hit1": "github.search_code", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "kind": "verb_product", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "ranking": [ + "github.list_notifications", + "github.mark_all_notifications_read", + "github.projects_list", + "github.list_gists", + "github.actions_list", + "github.list_starred_repositories", + "github.list_releases", + "github.list_labels", + "github.list_discussions", + "github.list_secret_scanning_alerts" + ], + "hit1": "github.list_notifications", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 307, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 655, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "kind": "verb_product", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "ranking": [ + "github.assign_copilot_to_issue", + "linear.linear_assign_issue", + "jira.jira_assign_issue", + "jira.jira_update_issue", + "jira.jira_transition_issue", + "github.add_issue_comment", + "github.create_issue", + "github.request_copilot_review", + "linear.linear_create_comment", + "jira.jira_add_comment" + ], + "hit1": "github.assign_copilot_to_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 875, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "kind": "resource", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "ranking": [ + "github.list_secret_scanning_alerts", + "github.get_secret_scanning_alert", + "github.list_code_scanning_alerts", + "github.get_code_scanning_alert", + "github.list_dependabot_alerts", + "id_map.json", + "discord.discord_get_channel", + "vercel.vercel_get_deployment", + "confluence.confluence_get_page", + "redis.redis_get" + ], + "hit1": "github.list_secret_scanning_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 681, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "ranking": [ + "github.create_or_update_file", + "google_drive.gdrive_update_file", + "slack.update_file", + "slack.create_file", + "google_drive.gdrive_create_file", + "github.projects_write", + "github.label_write", + "github.get_file_contents", + "github.issue_write", + "github.update_issue_comment" + ], + "hit1": "github.create_or_update_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 327, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 532, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1080, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "kind": "resource", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "ranking": [ + "github.list_repository_collaborators", + "github.list_commits", + "pagerduty.pagerduty_list_oncalls", + "trello.trello_list_cards", + "bitbucket.bitbucket_list_prs", + "github.actions_list", + "github.list_releases", + "github.list_labels", + "github.list_discussions", + "github.list_secret_scanning_alerts" + ], + "hit1": "github.list_repository_collaborators", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 264, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 776, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "kind": "verb_product", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "ranking": [ + "slack.slack_reply_to_thread", + "slack.slack_get_thread_replies", + "slack.slack_list_channels", + "slack.delete_file", + "slack.get_canvase", + "slack.delete_canvase", + "slack.delete_usergroup", + "slack.delete_bookmark", + "slack.delete_pin", + "slack.get_reminder" + ], + "hit1": "slack.slack_reply_to_thread", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 365, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 642, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "kind": "verb_product", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "ranking": [ + "slack.slack_add_reaction", + "discord.discord_add_reaction", + "trello.trello_add_comment", + "gmail.gmail_modify_email", + "jira.jira_add_comment", + "jira.jira_add_worklog", + "memory.add_observations", + "git.git_add", + "github.add_issue_comment", + "github.add_comment_to_pending_review" + ], + "hit1": "slack.slack_add_reaction", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 271, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 433, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 903, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "kind": "verb_product", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "ranking": [ + "slack.slack_search_messages", + "gmail.gmail_search_emails", + "slack.slack_get_channel_history", + "twilio.twilio_list_messages", + "grafana.grafana_search_dashboards", + "github.search_users", + "hubspot.hubspot_search_contacts", + "github.search_orgs", + "confluence.confluence_search", + "github.search_commits" + ], + "hit1": "slack.slack_search_messages", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 609, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "kind": "verb_product", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "ranking": [ + "slack.slack_create_channel", + "slack.slack_leave_channel", + "slack.slack_join_channel", + "slack.slack_set_channel_topic", + "slack.slack_get_channel_history", + "slack.slack_post_message", + "slack.slack_get_thread_replies", + "slack.slack_reply_to_thread", + "slack.slack_upload_file", + "slack.slack_add_reaction" + ], + "hit1": "slack.slack_create_channel", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "kind": "verb_product", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_transition_issue", + "jira.jira_assign_issue", + "jira.jira_get_issue", + "jira.jira_update_issue", + "jira.jira_create_issue", + "jira.jira_add_comment", + "jira.jira_add_worklog", + "gitlab.gitlab_get_issue", + "sentry.sentry_get_issue", + "sentry.sentry_update_issue" + ], + "hit1": "jira.jira_transition_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 778, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "kind": "identifier", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "ranking": [ + "jira.jira_search_issues", + "filesystem.list_directory_with_sizes", + "grafana.grafana_search_dashboards", + "github.search_users", + "hubspot.hubspot_search_contacts", + "github.search_orgs", + "confluence.confluence_search", + "github.search_commits", + "github.search_pull_requests", + "slack.slack_search_messages" + ], + "hit1": "jira.jira_search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 318, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 599, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "kind": "verb_product", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "ranking": [ + "jira.jira_add_worklog", + "jira.jira_add_comment", + "github.add_comment_to_pending_review", + "discord.discord_add_reaction", + "memory.add_observations", + "git.git_add", + "github.add_issue_comment", + "slack.slack_add_reaction", + "jira.jira_transition_issue", + "trello.trello_add_comment" + ], + "hit1": "jira.jira_add_worklog", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 334, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 478, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 878, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "kind": "verb_product", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "ranking": [ + "linear.linear_list_teams", + "linear.linear_list_projects", + "linear.linear_list_issues", + "github.get_teams", + "linear.list_comments", + "linear.list_attachments", + "linear.list_labels", + "linear.list_users", + "linear.list_cycles", + "linear.linear_search_issues" + ], + "hit1": "linear.linear_list_teams", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 646, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "ranking": [ + "github.update_issue_comment", + "linear.linear_create_comment", + "github.add_issue_comment", + "linear.linear_assign_issue", + "linear.linear_get_issue", + "linear.linear_update_issue", + "linear.linear_create_issue", + "jira.jira_add_comment", + "trello.trello_add_comment", + "figma.figma_post_comment" + ], + "hit1": "github.update_issue_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 298, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 886, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "kind": "paraphrase", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "ranking": [ + "filesystem.move_file", + "filesystem.read_media_file", + "filesystem.read_file", + "filesystem.read_text_file", + "brave_search.brave_local_search", + "figma.figma_get_file", + "slack.delete_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_update_file", + "google_drive.gdrive_share_file" + ], + "hit1": "filesystem.move_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "kind": "paraphrase", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "ranking": [ + "filesystem.search_files", + "google_drive.gdrive_search", + "linear.linear_search_issues", + "github.search_repositories", + "slack.list_files", + "github.push_files", + "redis.redis_keys", + "grafana.grafana_search_dashboards", + "github.search_users", + "hubspot.hubspot_search_contacts" + ], + "hit1": "filesystem.search_files", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "kind": "paraphrase", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "ranking": [ + "filesystem.edit_file", + "filesystem.list_directory_with_sizes", + "filesystem.write_file", + "filesystem.move_file", + "playwright.browser_tab_new", + "slack.delete_file", + "slack.get_file", + "slack.update_file", + "slack.create_file", + "google_drive.gdrive_create_file" + ], + "hit1": "filesystem.edit_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 736, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "ranking": [ + "memory.read_graph", + "filesystem.read_media_file", + "filesystem.read_file", + "filesystem.read_text_file", + "memory.delete_relations", + "memory.create_entities", + "memory.create_relations", + "memory.delete_entities", + "memory.add_observations", + "memory.open_nodes" + ], + "hit1": "memory.read_graph", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 170, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 305, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 562, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "kind": "verb_product", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "memory.open_nodes", + "figma.figma_get_file_nodes", + "memory.read_graph", + "memory.delete_observations", + "memory.delete_relations", + "memory.create_entities", + "memory.create_relations", + "memory.delete_entities", + "memory.add_observations" + ], + "hit1": "memory.search_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 290, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 546, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "kind": "verb_product", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "ranking": [ + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_diff", + "git.git_show", + "git.git_log", + "git.git_status", + "id_map.json", + "git.git_branch", + "git.git_checkout", + "git.git_create_branch" + ], + "hit1": "git.git_diff_unstaged", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 313, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 557, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "kind": "identifier", + "gold": [ + "git.git_add" + ], + "alts": [], + "ranking": [ + "git.git_add", + "filesystem.list_directory_with_sizes", + "github.add_issue_comment", + "github.add_comment_to_pending_review", + "github.push_files", + "github.create_gist", + "github.update_gist", + "jira.jira_add_comment", + "discord.discord_add_reaction", + "jira.jira_add_worklog" + ], + "hit1": "git.git_add", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 519, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 923, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "kind": "verb_product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "ranking": [ + "shopify.shopify_create_product", + "shopify.shopify_get_product", + "shopify.shopify_update_product", + "stripe.stripe_create_product", + "shopify.shopify_create_order", + "stripe.stripe_create_price", + "shopify.shopify_list_customers", + "shopify.shopify_get_order", + "shopify.shopify_list_products", + "shopify.shopify_list_orders" + ], + "hit1": "shopify.shopify_create_product", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "kind": "verb_product", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "ranking": [ + "stripe.stripe_cancel_subscription", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_invoice", + "stripe.stripe_get_customer", + "stripe.stripe_list_invoices", + "stripe.stripe_list_customers", + "stripe.stripe_create_product", + "stripe.stripe_create_refund", + "stripe.stripe_list_subscriptions", + "stripe.stripe_create_customer" + ], + "hit1": "stripe.stripe_cancel_subscription", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 176, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 275, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 610, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "kind": "resource", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_refund", + "id_map.json", + "trello.trello_create_board", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project", + "okta.okta_create_user", + "slack.slack_create_channel", + "shopify.shopify_create_order" + ], + "hit1": "stripe.stripe_create_payment_intent", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 590, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "id_map.json" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 626, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "kind": "verb_product", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "ranking": [ + "datadog.datadog_create_monitor", + "datadog.datadog_get_monitor", + "datadog.datadog_list_monitors", + "datadog.datadog_list_logs", + "datadog.datadog_query_metrics", + "id_map.json", + "trello.trello_create_board", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project", + "okta.okta_create_user" + ], + "hit1": "datadog.datadog_create_monitor", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 674, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "kind": "paraphrase", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "ranking": [ + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "sentry.sentry_list_events", + "pagerduty.pagerduty_resolve_incident", + "sentry.sentry_list_projects", + "sentry.sentry_list_issues", + "gitlab.gitlab_get_issue", + "jira.jira_assign_issue", + "jira.jira_get_issue", + "jira.jira_update_issue" + ], + "hit1": "sentry.sentry_get_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_scale_deployment", + "vercel.vercel_get_deployment", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_apply", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_delete", + "kubernetes.k8s_pod_logs" + ], + "hit1": "kubernetes.k8s_scale_deployment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_apply", + "terraform.tf_apply", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_delete", + "kubernetes.k8s_pod_logs" + ], + "hit1": "kubernetes.k8s_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 308, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "ranking": [ + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_exec", + "docker.docker_run", + "docker.docker_images", + "docker.docker_ps", + "docker.docker_build", + "kubernetes.k8s_pod_logs", + "id_map.json" + ], + "hit1": "docker.docker_stop", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 587, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "kind": "verb_product", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "ranking": [ + "docker.docker_build", + "docker.docker_run", + "docker.docker_images", + "docker.docker_stop", + "docker.docker_ps", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_exec", + "filesystem.read_media_file", + "id_map.json" + ], + "hit1": "docker.docker_build", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 547, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "kind": "verb_product", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "ranking": [ + "playwright.browser_click", + "puppeteer.puppeteer_click", + "playwright.browser_type", + "playwright.browser_take_screenshot", + "playwright.browser_snapshot", + "clickup.clickup_list_tasks", + "clickup.clickup_get_task", + "playwright.browser_tab_list", + "playwright.browser_tab_close", + "playwright.browser_tab_new" + ], + "hit1": "playwright.browser_click", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 571, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "kind": "paraphrase", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "ranking": [ + "playwright.browser_type", + "playwright.browser_wait_for", + "filesystem.read_text_file", + "playwright.browser_tab_list", + "playwright.browser_tab_close", + "playwright.browser_tab_new", + "playwright.browser_navigate", + "playwright.browser_fill_form", + "playwright.browser_evaluate", + "playwright.browser_press_key" + ], + "hit1": "playwright.browser_type", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 589, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_list_projects", + "gitlab.gitlab_list_issues", + "gitlab.gitlab_list_merge_requests", + "github.projects_list", + "vercel.vercel_list_projects", + "jira.jira_list_projects", + "sentry.sentry_list_projects", + "linear.linear_list_projects", + "gitlab.gitlab_get_issue", + "gitlab.gitlab_create_pipeline" + ], + "hit1": "gitlab.gitlab_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 595, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "kind": "verb_product", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "ranking": [ + "bitbucket.bitbucket_create_pr", + "github.create_pull_request", + "github.pull_request_review_write", + "github.merge_pull_request", + "github.pull_request_read", + "github.update_pull_request_branch", + "github.update_pull_request", + "bitbucket.bitbucket_create_issue", + "gitlab.gitlab_create_merge_request", + "bitbucket.bitbucket_list_prs" + ], + "hit1": "bitbucket.bitbucket_create_pr", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 439, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 634, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1155, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "kind": "verb_product", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "ranking": [ + "gmail.gmail_create_draft", + "github.create_pull_request", + "gmail.gmail_list_labels", + "gmail.gmail_search_emails", + "gmail.gmail_trash_email", + "gmail.gmail_get_email", + "gmail.gmail_send_email", + "gmail.gmail_modify_email", + "id_map.json", + "trello.trello_create_board" + ], + "hit1": "gmail.gmail_create_draft", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 284, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 411, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "kind": "verb_product", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_share_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_update_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "google_calendar.gcal_list_calendars", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_list_events" + ], + "hit1": "google_drive.gdrive_share_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 760, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "kind": "verb_product", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_list_events", + "sentry.sentry_list_events", + "google_calendar.gcal_list_calendars", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "kubernetes.k8s_get_events", + "google_drive.gdrive_list_folder", + "google_drive.gdrive_search", + "google_drive.gdrive_get_file" + ], + "hit1": "google_calendar.gcal_list_events", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "redis.redis_delete", + "linear.delete_attachment", + "jira.delete_attachment", + "linear.delete_comment", + "linear.delete_label" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_projects", + "vercel.vercel_list_deployments", + "jira.jira_list_projects", + "sentry.sentry_list_projects", + "gitlab.gitlab_list_projects", + "linear.linear_list_projects", + "github.projects_list", + "vercel.vercel_get_deployment", + "vercel.vercel_create_project", + "github.projects_get" + ], + "hit1": "vercel.vercel_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 253, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "ranking": [ + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update", + "supabase.supabase_list_tables", + "supabase.supabase_invoke_fn", + "memory.delete_observations", + "elasticsearch.es_get_doc", + "datadog.datadog_list_logs", + "memory.delete_relations", + "aws_s3.s3_get_object" + ], + "hit1": "supabase.supabase_select", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 697, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "kind": "verb_product", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "slack.slack_post_message", + "figma.figma_get_file", + "figma.figma_get_comments", + "figma.figma_get_file_nodes", + "figma.figma_get_images", + "linear.delete_comment", + "notion.get_comment", + "notion.delete_comment", + "linear.get_comment" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 646, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "kind": "verb_product", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "ranking": [ + "okta.okta_list_users", + "notion.notion_list_users", + "slack.slack_list_users", + "linear.list_users", + "notion.list_users_tokens", + "slack.slack_get_users", + "okta.okta_create_user", + "okta.okta_deactivate_user", + "okta.okta_get_user", + "pagerduty.pagerduty_list_oncalls" + ], + "hit1": "okta.okta_list_users", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 174, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 618, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query_range", + "prometheus.prom_query", + "prometheus.prom_list_metrics", + "postgres.query", + "datadog.datadog_query_metrics", + "postgres.explain_query", + "mysql.mysql_query", + "sqlite.sqlite_query", + "salesforce.salesforce_query", + "notion.notion_query_database" + ], + "hit1": "prometheus.prom_query_range", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "kind": "verb_product", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "ranking": [ + "grafana.grafana_create_annotation", + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "id_map.json", + "trello.trello_create_board", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project", + "okta.okta_create_user", + "slack.slack_create_channel", + "shopify.shopify_create_order" + ], + "hit1": "grafana.grafana_create_annotation", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 275, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 574, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "kind": "identifier", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "ranking": [ + "terraform.tf_apply", + "kubernetes.k8s_apply", + "terraform.tf_plan", + "terraform.tf_state_list", + "id_map.json" + ], + "hit1": "terraform.tf_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 238, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 595, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "kind": "verb_product", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "ranking": [ + "asana.asana_list_tasks", + "clickup.clickup_list_tasks", + "asana.asana_get_task", + "asana.asana_create_project", + "asana.asana_update_task", + "asana.asana_create_task", + "id_map.json", + "vercel.vercel_list_projects", + "jira.jira_list_projects", + "cloudflare.cf_list_zones" + ], + "hit1": "asana.asana_list_tasks", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 566, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "ranking": [ + "trello.trello_add_comment", + "trello.trello_create_card", + "trello.trello_list_cards", + "figma.figma_post_comment", + "github.update_issue_comment", + "trello.trello_create_board", + "linear.delete_comment", + "notion.get_comment", + "notion.delete_comment", + "linear.get_comment" + ], + "hit1": "trello.trello_add_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 438, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_update_task", + "clickup.clickup_get_task", + "clickup.clickup_create_task", + "asana.asana_update_task", + "clickup.clickup_list_tasks", + "asana.asana_get_task", + "puppeteer.puppeteer_click", + "playwright.browser_click", + "asana.asana_create_task", + "id_map.json" + ], + "hit1": "clickup.clickup_update_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "kind": "verb_product", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "ranking": [ + "confluence.confluence_search", + "confluence.confluence_get_page", + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "grafana.grafana_search_dashboards", + "github.search_users", + "hubspot.hubspot_search_contacts", + "github.search_orgs", + "github.search_commits", + "github.search_pull_requests" + ], + "hit1": "confluence.confluence_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 636, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_deal", + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_search_contacts", + "hubspot.hubspot_list_contacts", + "github.create_gist", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.create_or_update_file" + ], + "hit1": "hubspot.hubspot_create_deal", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 899, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_search", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_create_ticket", + "id_map.json", + "clickup.clickup_list_tasks", + "vercel.vercel_list_projects", + "jira.jira_list_projects", + "cloudflare.cf_list_zones" + ], + "hit1": "zendesk.zendesk_list_tickets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 529, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "kind": "verb_product", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "ranking": [ + "discord.discord_list_channels", + "slack.slack_list_channels", + "discord.discord_list_members", + "discord.discord_get_channel", + "discord.discord_send_message", + "discord.discord_add_reaction", + "id_map.json", + "clickup.clickup_list_tasks", + "vercel.vercel_list_projects", + "jira.jira_list_projects" + ], + "hit1": "discord.discord_list_channels", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 513, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "ranking": [ + "redis.redis_set", + "redis.redis_get", + "slack.slack_set_channel_topic", + "redis.redis_delete", + "redis.redis_keys", + "playwright.browser_press_key", + "id_map.json", + "jira.jira_get_issue", + "figma.figma_get_file", + "jira.jira_assign_issue" + ], + "hit1": "redis.redis_set", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 526, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert", + "mongodb.mongo_delete", + "mongodb.mongo_find", + "mongodb.mongo_update", + "mongodb.mongo_list_collections", + "supabase.supabase_insert", + "id_map.json" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 607, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "kind": "verb_product", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "ranking": [ + "mysql.mysql_query", + "postgres.query", + "mysql.mysql_list_tables", + "sqlite.sqlite_query", + "postgres.explain_query", + "salesforce.salesforce_query", + "github.delete_check_run", + "github.get_check_run", + "github.update_check_run", + "docker.docker_run" + ], + "hit1": "mysql.mysql_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 174, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 611, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "elasticsearch.es_search", + "playwright.browser_tab_close", + "id_map.json", + "supabase.supabase_list_tables", + "jira.get_attachment", + "linear.get_attachment", + "linear.delete_attachment", + "gitlab.get_label" + ], + "hit1": "elasticsearch.es_index_doc", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 538, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_put_object", + "aws_s3.s3_list_objects", + "asana.asana_get_task", + "okta.okta_get_user", + "aws_s3.s3_list_buckets", + "discord.discord_get_channel", + "vercel.vercel_get_deployment", + "confluence.confluence_get_page" + ], + "hit1": "aws_s3.s3_get_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 634, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones", + "cloudflare.cf_list_dns", + "cloudflare.cf_purge_cache", + "cloudflare.cf_create_dns", + "id_map.json", + "clickup.clickup_list_tasks", + "vercel.vercel_list_projects", + "jira.jira_list_projects", + "linear.linear_list_teams", + "google_calendar.gcal_list_calendars" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 285, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 482, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_create_product", + "stripe.stripe_create_refund", + "stripe.stripe_create_customer", + "stripe.stripe_create_price", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent", + "stripe.stripe_get_customer", + "stripe.stripe_cancel_subscription", + "stripe.stripe_list_invoices" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "kind": "verb_product", + "gold": [ + "github.actions_list" + ], + "alts": [], + "ranking": [ + "github.actions_list", + "github.actions_get_job_logs", + "github.actions_get", + "github.actions_run_trigger", + "github.projects_list", + "github.list_gists", + "github.list_starred_repositories", + "github.list_releases", + "github.list_labels", + "github.list_discussions" + ], + "hit1": "github.actions_list", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 453, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 772, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "kind": "verb_product", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "ranking": [ + "github.actions_get_job_logs", + "github.actions_get", + "gitlab.get_job", + "github.actions_list", + "github.actions_run_trigger", + "github.projects_get", + "github.get_gist", + "github.get_teams", + "github.get_label", + "github.get_discussion" + ], + "hit1": "github.actions_get_job_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 442, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 769, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "kind": "verb_product", + "gold": [ + "github.create_repository" + ], + "alts": [], + "ranking": [ + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.create_or_update_file", + "git.git_create_branch", + "github.star_repository", + "github.unstar_repository", + "github.delete_repository", + "github.fork_repository", + "github.list_repository_collaborators" + ], + "hit1": "github.create_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 562, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 927, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "kind": "verb_product", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "ranking": [ + "git.git_checkout", + "git.git_create_branch", + "git.git_branch", + "github.create_branch", + "gitlab.gitlab_create_merge_request", + "github.list_commits", + "github.delete_file", + "github.push_files", + "github.create_or_update_file", + "github.update_pull_request_branch" + ], + "hit1": "git.git_checkout", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1001, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "kind": "identifier", + "gold": [ + "git.git_log" + ], + "alts": [], + "ranking": [ + "git.git_log", + "git.git_show", + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_status", + "git.git_diff", + "git.git_branch", + "git.git_checkout", + "id_map.json", + "git.git_create_branch" + ], + "hit1": "git.git_log", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 555, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "kind": "verb_product", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "ranking": [ + "postgres.list_schemas", + "postgres.list_tables", + "postgres.explain_query", + "postgres.query", + "postgres.describe_table", + "id_map.json", + "clickup.clickup_list_tasks", + "vercel.vercel_list_projects", + "jira.jira_list_projects", + "cloudflare.cf_list_zones" + ], + "hit1": "postgres.list_schemas", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 160, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 439, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "kind": "verb_product", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "ranking": [ + "memory.open_nodes", + "memory.search_nodes", + "memory.read_graph", + "memory.create_entities", + "memory.create_relations", + "memory.add_observations", + "memory.delete_relations", + "memory.delete_entities", + "figma.figma_get_file_nodes", + "playwright.browser_tab_list" + ], + "hit1": "memory.open_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 531, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "kind": "paraphrase", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "ranking": [ + "filesystem.get_file_info", + "google_drive.gdrive_get_file", + "figma.figma_get_file_nodes", + "gitlab.gitlab_get_file", + "github.get_file_contents", + "filesystem.read_media_file", + "filesystem.read_file", + "filesystem.read_text_file", + "figma.figma_get_file", + "slack.get_file" + ], + "hit1": "filesystem.get_file_info", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 209, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 715, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "kind": "paraphrase", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "ranking": [ + "filesystem.write_file", + "filesystem.read_media_file", + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.edit_file", + "brave_search.brave_local_search", + "github.issue_write", + "playwright.browser_tab_new", + "github.projects_write", + "github.label_write" + ], + "hit1": "filesystem.write_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 846, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "kind": "verb_product", + "gold": [ + "github.search_issues" + ], + "alts": [], + "ranking": [ + "github.search_issues", + "linear.linear_search_issues", + "jira.jira_search_issues", + "github.list_issues", + "github.search_users", + "github.search_orgs", + "github.search_commits", + "github.search_pull_requests", + "github.search_code", + "github.search_repositories" + ], + "hit1": "github.search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 233, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "kind": "paraphrase", + "gold": [ + "github.get_me" + ], + "alts": [], + "ranking": [ + "github.get_teams", + "github.get_me", + "github.list_notifications", + "slack.slack_get_user_profile", + "okta.okta_get_user", + "linear.get_user", + "github.get_latest_release", + "github.get_repository_tree", + "github.list_gists", + "github.list_starred_repositories" + ], + "hit1": "github.get_teams", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 288, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 622, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "cost": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.6333333333333333, + "p95_calls": 1.0, + "success_within_1": 0.96, + "success_within_2": 0.9733333333333334, + "success_within_3": 0.9733333333333334, + "mean_tokens": 399.6333333333333, + "p95_tokens": 353.1499999999994, + "total_tokens": 59945, + "tokens_per_successful_query": 399.6333333333333, + "total_tokens_successful": 59945 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.3733333333333333, + "p95_calls": 1.0, + "success_within_1": 0.9666666666666667, + "success_within_2": 0.9733333333333334, + "success_within_3": 0.9733333333333334, + "mean_tokens": 529.34, + "p95_tokens": 572.9999999999997, + "total_tokens": 79401, + "tokens_per_successful_query": 529.34, + "total_tokens_successful": 79401 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.1733333333333333, + "p95_calls": 1.0, + "success_within_1": 0.9733333333333334, + "success_within_2": 0.9733333333333334, + "success_within_3": 0.9866666666666667, + "mean_tokens": 844.0266666666666, + "p95_tokens": 1107.3999999999999, + "total_tokens": 126604, + "tokens_per_successful_query": 844.0266666666666, + "total_tokens_successful": 126604 + } + } +} diff --git a/experiments/mcp-tool-search/results/codedb_symbol.csv b/experiments/mcp-tool-search/results/codedb_symbol.csv new file mode 100644 index 00000000..2a515b6c --- /dev/null +++ b/experiments/mcp-tool-search/results/codedb_symbol.csv @@ -0,0 +1,151 @@ +id,query,kind,hit1,recall@1,recall@3,recall@5,mrr@10,ndcg@10,hit_rank,k3_calls,k3_tokens,k3_success1,k5_calls,k5_tokens,k5_success1,k10_calls,k10_tokens,k10_success1,ranking +q001,Create a GitHub issue,verb_product,github.list_issues,0.0,1.0,1.0,0.3333,0.3801,3,1,341,1.0,1,466,1.0,1,802,1.0,github.list_issues github.search_issues github.create_issue gitlab.gitlab_get_issue jira.jira_get_issue jira.jira_update_issue jira.jira_search_issues linear.linear_get_issue gitlab.gitlab_list_issues sentry.sentry_get_issue +q002,Create a Jira ticket,verb_product,zendesk.zendesk_create_ticket,0.0,0.0,0.0,0.1250,0.3155,8,3,719,0.0,2,809,0.0,1,789,1.0,zendesk.zendesk_create_ticket zendesk.zendesk_list_tickets zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket stripe.stripe_create_price jira.jira_get_issue jira.jira_add_comment jira.jira_create_issue clickup.clickup_get_task jira.jira_create_sprint +q003,Send a Slack message,verb_product,slack.slack_post_message,1.0,1.0,1.0,1.0000,1.0000,1,1,226,1.0,1,345,1.0,1,694,1.0,slack.slack_post_message slack.slack_search_messages discord.discord_send_message twilio.twilio_list_messages slack.slack_get_users slack.slack_list_users github.get_secret_scanning_alert mysql.mysql_list_tables figma.figma_get_images slack.slack_leave_channel +q004,List open pull requests,resource,github.list_pull_requests,1.0,1.0,1.0,1.0000,1.0000,1,1,251,1.0,1,506,1.0,1,1038,1.0,github.list_pull_requests github.search_pull_requests github.pull_request_read github.merge_pull_request github.create_pull_request github.update_pull_request gitlab.gitlab_list_merge_requests github.pull_request_review_write gitlab.gitlab_get_merge_request github.update_pull_request_branch +q005,Open a pull request on GitHub,verb_product,git.git_checkout,0.0,0.0,0.0,0.0000,0.0000,46,16,4346,0.0,10,4440,0.0,5,4325,0.0,git.git_checkout github.get_check_run hubspot.hubspot_create_deal git.git_show hubspot.hubspot_get_contact git.git_branch hubspot.hubspot_list_contacts hubspot.hubspot_create_contact git.git_status hubspot.hubspot_search_contacts +q006,Run a SQL query against postgres,verb_product,github.list_starred_repositories,0.0,0.0,0.0,0.0000,0.0000,104,35,9005,0.0,21,8697,0.0,11,8763,0.0,github.list_starred_repositories github.search_repositories gitlab.gitlab_list_projects github.projects_get okta.okta_get_user datadog.datadog_list_monitors slack.slack_post_message gitlab.gitlab_list_merge_requests github.star_repository salesforce.salesforce_list_opportunities +q007,Read a local file from disk,paraphrase,redis.redis_keys,0.0,0.0,0.0,0.0000,0.0000,85,29,7857,0.0,17,7449,0.0,9,7634,0.0,redis.redis_keys prometheus.prom_list_metrics redis.redis_set github.get_notification_details redis.redis_get gitlab.delete_wiki docker.docker_images gitlab.list_wikis github.list_hooks git.git_diff_staged +q008,What files are in this folder,paraphrase,google_drive.gdrive_list_folder,0.0,0.0,0.0,0.1429,0.2129,7,3,648,0.0,2,691,0.0,1,669,1.0,google_drive.gdrive_list_folder jira.list_filters shopify.shopify_get_order shopify.shopify_list_orders shopify.shopify_create_order jira.get_filter filesystem.list_directory slack.list_reminders github.list_dependabot_alerts stripe.stripe_list_customers +q009,Remember this fact in the knowledge graph,paraphrase,memory.read_graph,0.0,0.0,0.0,0.0000,0.0000,99,33,8732,0.0,20,8484,0.0,10,8254,0.0,memory.read_graph pagerduty.pagerduty_acknowledge_incident git.git_branch google_drive.gdrive_search gitlab.get_release gmail.gmail_trash_email playwright.browser_snapshot git.git_create_branch slack.create_pin gitlab.gitlab_create_pipeline +q010,Search the web with Brave,verb_product,github.list_branches,0.0,0.0,1.0,0.2500,0.4307,4,2,500,0.0,1,428,1.0,1,810,1.0,github.list_branches git.git_branch bitbucket.bitbucket_create_pr brave_search.brave_web_search brave_search.brave_local_search playwright.browser_navigate github.list_releases gitlab.list_releases jira.create_version slack.list_canvases +q011,Fetch the contents of a URL,verb_product,slack.slack_get_user_profile,0.0,0.0,0.0,0.0000,0.0000,111,37,9389,0.0,23,9302,0.0,12,9383,0.0,slack.slack_get_user_profile gitlab.update_release okta.okta_get_user postgres.query prometheus.prom_query filesystem.read_file okta.okta_list_users linear.get_user cloudflare.cf_purge_cache gitlab.update_label +q012,Show git status,identifier,git.git_status,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,315,1.0,1,602,1.0,git.git_status terraform.tf_state_list github.list_tags github.list_artifacts postgres.list_tables okta.okta_get_user github.get_teams slack.slack_get_users git.git_reset notion.list_data_sources +q013,Commit staged changes,paraphrase,slack.slack_list_channels,0.0,0.0,0.0,0.0000,0.0000,117,39,9686,0.0,24,9576,0.0,12,9348,0.0,slack.slack_list_channels slack.slack_get_channel_history discord.discord_list_channels slack.slack_search_messages slack.slack_leave_channel discord.discord_get_channel slack.slack_create_channel slack.slack_set_channel_topic github.list_code_scanning_alerts stripe.stripe_cancel_subscription +q014,Create a Linear issue,verb_product,linear.linear_get_issue,0.0,0.0,0.0,0.0000,0.0000,143,48,12006,0.0,29,11681,0.0,15,11704,0.0,linear.linear_get_issue github.list_issues github.search_issues github.create_issue linear.linear_list_issues jira.jira_get_issue sentry.sentry_get_issue jira.jira_search_issues jira.jira_create_issue gitlab.gitlab_get_issue +q015,Make a task in my project tracker Linear,paraphrase,linear.linear_get_issue,0.0,0.0,0.0,0.1111,0.3010,9,3,741,0.0,2,831,0.0,1,807,1.0,linear.linear_get_issue gitlab.gitlab_create_pipeline linear.linear_list_teams circleci.circleci_trigger_pipeline linear.linear_list_issues slack.slack_get_thread_replies linear.linear_assign_issue pagerduty.pagerduty_acknowledge_incident linear.linear_create_issue linear.linear_update_issue +q016,Post a comment on a GitHub issue,verb_product,github.list_issues,0.0,0.0,0.0,0.0000,0.0000,41,14,3866,0.0,9,4037,0.0,5,4516,0.0,github.list_issues github.search_issues github.create_issue gitlab.gitlab_get_issue jira.jira_get_issue jira.jira_update_issue jira.jira_search_issues linear.linear_get_issue gitlab.gitlab_list_issues sentry.sentry_get_issue +q017,Merge the pull request,ambiguous,github.pull_request_read,0.0,1.0,1.0,0.3333,0.3801,3,1,303,1.0,1,595,1.0,1,1038,1.0,github.pull_request_read github.list_pull_requests github.merge_pull_request github.create_pull_request github.update_pull_request github.search_pull_requests github.pull_request_review_write gitlab.gitlab_get_merge_request gitlab.gitlab_list_merge_requests github.update_pull_request_branch +q018,Trigger a GitHub Actions workflow,verb_product,jira.jira_add_worklog,0.0,0.0,0.0,0.0000,0.0000,65,22,6335,0.0,13,6086,0.0,7,6281,0.0,jira.jira_add_worklog github.actions_get_job_logs playwright.browser_fill_form +q019,Get Dependabot alerts,resource,github.list_dependabot_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,262,1.0,1,437,1.0,1,747,1.0,github.list_dependabot_alerts github.get_dependabot_alert linear.list_users github.list_code_scanning_alerts github.list_environments notion.list_users_tokens okta.okta_list_users google_calendar.gcal_list_events linear.linear_get_issue linear.linear_list_teams +q020,List Sentry issues,verb_product,github.search_issues,0.0,1.0,1.0,0.3333,0.5000,3,1,303,1.0,1,507,1.0,1,841,1.0,github.search_issues github.list_issues sentry.sentry_list_issues github.create_issue jira.jira_search_issues gitlab.gitlab_list_issues sentry.sentry_get_issue linear.linear_search_issues linear.linear_list_issues jira.jira_get_issue +q021,Create a Stripe customer,verb_product,stripe.stripe_get_customer,0.0,1.0,1.0,0.3333,0.5000,3,1,203,1.0,1,324,1.0,1,720,1.0,stripe.stripe_get_customer stripe.stripe_list_customers stripe.stripe_create_customer shopify.shopify_list_customers stripe.stripe_create_refund linear.create_user stripe.stripe_create_price notion.create_users_token stripe.stripe_create_product github.search_users +q022,Refund a Stripe payment,paraphrase,stripe.stripe_get_payment_intent,0.0,0.0,0.0,0.1111,0.3010,9,3,678,0.0,2,706,0.0,1,686,1.0,stripe.stripe_get_payment_intent stripe.stripe_create_payment_intent kubernetes.k8s_scale_deployment kubernetes.k8s_list_deployments vercel.vercel_get_deployment vercel.vercel_list_deployments linear.update_comment notion.update_comment stripe.stripe_create_refund stripe.stripe_create_invoice +q023,Send an email with Gmail,verb_product,gmail.gmail_get_email,0.0,1.0,1.0,0.3333,0.5000,3,1,224,1.0,1,349,1.0,1,713,1.0,gmail.gmail_get_email gmail.gmail_trash_email gmail.gmail_send_email figma.figma_get_file gmail.gmail_search_emails gmail.gmail_modify_email gitlab.gitlab_get_file figma.figma_get_file_nodes filesystem.write_file gmail.gmail_list_labels +q024,Search my Gmail inbox,verb_product,filesystem.get_file_info,0.0,0.0,0.0,0.0000,0.0000,251,84,19977,0.0,51,19522,0.0,26,19419,0.0,filesystem.get_file_info figma.figma_get_file_nodes notion.notion_get_block jira.list_boards gmail.gmail_list_labels elasticsearch.es_index_doc gmail.gmail_get_email notion.notion_append_block gmail.gmail_send_email slack.list_bookmarks +q025,Upload a file to Google Drive,paraphrase,google_drive.gdrive_search,0.0,0.0,1.0,0.2500,0.4307,4,2,496,0.0,1,383,1.0,1,671,1.0,google_drive.gdrive_search google_drive.gdrive_get_file google_drive.gdrive_share_file google_drive.gdrive_create_file google_drive.gdrive_list_folder google_drive.gdrive_update_file docker.docker_images postgres.describe_table slack.delete_reminder redis.redis_get +q026,Create a calendar event,resource,google_calendar.gcal_create_event,1.0,1.0,1.0,1.0000,1.0000,1,1,317,1.0,1,461,1.0,1,834,1.0,google_calendar.gcal_create_event google_calendar.gcal_list_events google_calendar.gcal_update_event google_calendar.gcal_delete_event sentry.sentry_list_events github.create_environment kubernetes.k8s_get_events github.get_environment kubernetes.k8s_scale_deployment github.list_environments +q027,List Kubernetes pods,verb_product,kubernetes.k8s_list_pods,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,370,1.0,1,717,1.0,kubernetes.k8s_list_pods stripe.stripe_create_product kubernetes.k8s_pod_logs memory.open_nodes kubernetes.k8s_get_pod github.projects_get shopify.shopify_list_products jira.list_boards github.projects_list shopify.shopify_list_orders +q028,Get logs from a k8s pod,verb_product,shopify.shopify_get_product,0.0,0.0,0.0,0.1111,0.3010,9,3,758,0.0,2,807,0.0,1,783,1.0,shopify.shopify_get_product shopify.shopify_create_product stripe.stripe_create_product shopify.shopify_list_products shopify.shopify_update_product shopify.shopify_create_order kubernetes.k8s_get_pod github.fork_repository kubernetes.k8s_pod_logs figma.figma_post_comment +q029,Run a docker container,verb_product,time.convert_time,0.0,0.0,0.0,0.0000,0.0000,189,63,15143,0.0,38,14689,0.0,19,14309,0.0,time.convert_time mongodb.mongo_insert notion.notion_search slack.create_reminder okta.okta_list_users github.actions_run_trigger notion.notion_list_users confluence.confluence_search github.get_code_scanning_alert github.list_code_scanning_alerts +q030,Show docker container logs,verb_product,github.actions_get_job_logs,0.0,1.0,1.0,0.5000,0.6309,2,1,274,1.0,1,387,1.0,1,757,1.0,github.actions_get_job_logs docker.docker_logs kubernetes.k8s_pod_logs git.git_log github.list_gists datadog.datadog_list_logs github.list_tags gitlab.list_jobs github.search_orgs github.list_hooks +q031,Navigate the browser to a URL,ambiguous,linear.get_user,0.0,0.0,0.0,0.0000,0.0000,174,58,14269,0.0,35,13802,0.0,18,13742,0.0,linear.get_user gitlab.get_release linear.list_users gitlab.update_release linear.create_user linear.delete_user linear.update_user postgres.query filesystem.read_file okta.okta_get_user +q032,Take a screenshot of the page,ambiguous,github.get_package,0.0,0.0,0.0,0.0000,0.0000,178,60,14674,0.0,36,14146,0.0,18,13750,0.0,github.get_package github.create_package github.delete_package notion.notion_get_page github.update_package github.list_packages notion.notion_create_page notion.notion_update_page confluence.confluence_get_page confluence.confluence_create_page +q033,Create a GitLab merge request,verb_product,github.merge_pull_request,0.0,0.0,0.0,0.1000,0.2891,10,4,1156,0.0,2,1020,0.0,1,998,1.0,github.merge_pull_request github.pull_request_read gitlab.gitlab_get_merge_request github.create_pull_request github.list_pull_requests gitlab.gitlab_list_merge_requests github.update_pull_request github.search_pull_requests gitlab.gitlab_merge_merge_request gitlab.gitlab_create_merge_request +q034,Search Notion pages,verb_product,notion.notion_get_page,0.0,0.0,0.0,0.1000,0.2891,10,4,782,0.0,2,650,0.0,1,630,1.0,notion.notion_get_page github.list_packages github.get_package notion.notion_create_page linear.get_user notion.notion_update_page asana.asana_get_task notion.notion_get_block github.get_gist notion.notion_search +q035,Create a Notion page,verb_product,notion.notion_get_page,0.0,1.0,1.0,0.5000,0.6309,2,1,230,1.0,1,373,1.0,1,774,1.0,notion.notion_get_page notion.notion_create_page notion.notion_update_page github.actions_get confluence.confluence_get_page confluence.confluence_create_page confluence.confluence_update_page github.get_package github.list_packages github.create_package +q036,Query a Notion database,verb_product,notion.notion_query_database,1.0,1.0,1.0,1.0000,1.0000,1,1,266,1.0,1,403,1.0,1,733,1.0,notion.notion_query_database notion.notion_create_database slack.update_canvase notion.notion_update_page gitlab.update_label notion.notion_search linear.update_label notion.notion_create_page notion.get_data_source notion.notion_get_page +q037,Create a Zendesk ticket,verb_product,zendesk.zendesk_get_ticket,0.0,1.0,1.0,0.3333,0.5000,3,1,221,1.0,1,365,1.0,1,645,1.0,zendesk.zendesk_get_ticket zendesk.zendesk_list_tickets zendesk.zendesk_create_ticket zendesk.zendesk_update_ticket clickup.clickup_get_task zendesk.zendesk_search aws_s3.s3_list_buckets pagerduty.pagerduty_resolve_incident docker.docker_stop pagerduty.pagerduty_acknowledge_incident +q038,Create a HubSpot contact,verb_product,hubspot.hubspot_get_contact,0.0,1.0,1.0,0.3333,0.5000,3,1,216,1.0,1,328,1.0,1,722,1.0,hubspot.hubspot_get_contact hubspot.hubspot_list_contacts hubspot.hubspot_create_contact hubspot.hubspot_search_contacts salesforce.salesforce_convert_lead time.convert_time figma.figma_post_comment okta.okta_create_user hubspot.hubspot_create_deal github.create_artifact +q039,Create a Salesforce lead,verb_product,salesforce.salesforce_create_lead,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,405,1.0,1,741,1.0,salesforce.salesforce_create_lead salesforce.salesforce_convert_lead jira.delete_board filesystem.read_file linear.linear_update_issue salesforce.salesforce_query github.issue_read salesforce.salesforce_create_record memory.read_graph salesforce.salesforce_delete_record +q040,Run a SOQL query,identifier,mysql.mysql_query,0.0,0.0,1.0,0.2000,0.3869,5,2,413,0.0,1,293,1.0,1,721,1.0,mysql.mysql_query prometheus.prom_query sqlite.sqlite_query postgres.explain_query salesforce.salesforce_query prometheus.prom_query_range datadog.datadog_query_metrics notion.notion_query_database github.pull_request_read postgres.query +q041,List Shopify orders,verb_product,shopify.shopify_list_orders,1.0,1.0,1.0,1.0000,1.0000,1,1,210,1.0,1,335,1.0,1,644,1.0,shopify.shopify_list_orders shopify.shopify_get_order shopify.shopify_create_order docker.docker_ps slack.list_reminders docker.docker_stop shopify.shopify_list_customers shopify.shopify_list_products memory.open_nodes shopify.shopify_create_product +q042,Create a PagerDuty incident,verb_product,pagerduty.pagerduty_get_incident,0.0,1.0,1.0,0.3333,0.5000,3,1,267,1.0,1,403,1.0,1,802,1.0,pagerduty.pagerduty_get_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_create_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_acknowledge_incident github.get_notification_details github.get_discussion_comments linear.list_comments github.update_issue_comment notion.list_comments +q043,Who is on call in PagerDuty,paraphrase,pagerduty.pagerduty_get_incident,0.0,1.0,1.0,0.5000,0.6309,2,1,227,1.0,1,397,1.0,1,714,1.0,pagerduty.pagerduty_get_incident pagerduty.pagerduty_list_oncalls pagerduty.pagerduty_list_incidents pagerduty.pagerduty_create_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_acknowledge_incident filesystem.create_directory notion.update_users_token shopify.shopify_get_product linear.update_cycle +q044,Query Datadog metrics,verb_product,datadog.datadog_query_metrics,1.0,1.0,1.0,1.0000,1.0000,1,1,235,1.0,1,347,1.0,1,801,1.0,datadog.datadog_query_metrics prometheus.prom_list_metrics datadog.datadog_list_monitors github.get_artifact datadog.datadog_get_monitor trello.trello_list_cards datadog.datadog_list_logs github.get_discussion datadog.datadog_create_monitor github.list_artifacts +q045,Search Datadog logs,verb_product,datadog.datadog_list_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,276,1.0,1,437,1.0,1,780,1.0,datadog.datadog_list_logs github.actions_get_job_logs docker.docker_logs kubernetes.k8s_pod_logs datadog.datadog_list_monitors git.git_log datadog.datadog_get_monitor github.list_gists datadog.datadog_query_metrics github.list_tags +q046,List S3 buckets,verb_product,aws_s3.s3_list_buckets,1.0,1.0,1.0,1.0000,1.0000,1,1,191,1.0,1,317,1.0,1,752,1.0,aws_s3.s3_list_buckets zendesk.zendesk_list_tickets bitbucket.bitbucket_list_prs linear.list_users bitbucket.bitbucket_list_repos linear.list_comments bitbucket.bitbucket_create_issue notion.list_comments bitbucket.bitbucket_create_pr github.list_branches +q047,Upload an object to S3,paraphrase,aws_s3.s3_get_object,0.0,1.0,1.0,0.5000,0.6309,2,1,234,1.0,1,363,1.0,1,676,1.0,aws_s3.s3_get_object aws_s3.s3_put_object gitlab.get_job github.projects_get git.git_log github.projects_list github.get_hook github.projects_write git.git_show jira.get_board +q048,Purge Cloudflare cache,verb_product,cloudflare.cf_purge_cache,1.0,1.0,1.0,1.0000,1.0000,1,1,287,1.0,1,477,1.0,1,917,1.0,cloudflare.cf_purge_cache github.create_check_run jira.create_attachment linear.create_attachment linear.create_cycle slack.slack_create_channel linear.create_comment notion.create_comment github.search_code slack.create_canvase +q049,Create a Cloudflare DNS record,verb_product,salesforce.salesforce_get_record,0.0,0.0,0.0,0.0000,0.0000,301,101,24119,0.0,61,23357,0.0,31,23011,0.0,salesforce.salesforce_get_record salesforce.salesforce_delete_record salesforce.salesforce_create_record filesystem.list_directory salesforce.salesforce_update_record filesystem.directory_tree filesystem.create_directory shopify.shopify_create_order salesforce.salesforce_convert_lead jira.create_board +q050,List Vercel deployments,verb_product,vercel.vercel_list_deployments,1.0,1.0,1.0,1.0000,1.0000,1,1,195,1.0,1,344,1.0,1,618,1.0,vercel.vercel_list_deployments kubernetes.k8s_list_deployments vercel.vercel_get_deployment kubernetes.k8s_scale_deployment linear.delete_comment vercel.vercel_list_projects notion.delete_comment vercel.vercel_create_project linear.list_comments salesforce.salesforce_list_opportunities +q051,Insert a row into Supabase,verb_product,supabase.supabase_select,0.0,1.0,1.0,0.5000,0.6309,2,1,238,1.0,1,355,1.0,1,739,1.0,supabase.supabase_select supabase.supabase_insert supabase.supabase_update supabase.supabase_invoke_fn supabase.supabase_list_tables sentry.sentry_update_issue notion.notion_query_database notion.notion_update_page slack.update_canvase gitlab.update_release +q052,Export images from Figma,paraphrase,figma.figma_get_images,1.0,1.0,1.0,1.0000,1.0000,1,1,214,1.0,1,320,1.0,1,627,1.0,figma.figma_get_images playwright.browser_navigate figma.figma_get_file prometheus.prom_list_metrics figma.figma_get_comments playwright.browser_fill_form figma.figma_post_comment figma.figma_get_file_nodes gmail.gmail_get_email slack.get_bookmark +q053,Send an SMS with Twilio,verb_product,twilio.twilio_send_sms,1.0,1.0,1.0,1.0000,1.0000,1,1,262,1.0,1,385,1.0,1,739,1.0,twilio.twilio_send_sms github.list_notifications twilio.twilio_make_call gitlab.gitlab_create_pipeline twilio.twilio_list_messages filesystem.list_directory_with_sizes gitlab.get_wiki filesystem.get_file_info gitlab.list_wikis gitlab.create_wiki +q054,Create an Okta user,verb_product,linear.get_user,0.0,0.0,0.0,0.1250,0.3155,8,3,605,0.0,2,640,0.0,1,619,1.0,linear.get_user linear.list_users okta.okta_get_user linear.create_user linear.delete_user linear.update_user okta.okta_list_users okta.okta_create_user github.search_users slack.get_usergroup +q055,Instant query Prometheus,verb_product,sentry.sentry_list_projects,0.0,0.0,0.0,0.0000,0.0000,327,109,25677,0.0,66,25014,0.0,33,24354,0.0,sentry.sentry_list_projects linear.create_user datadog.datadog_query_metrics github.projects_get prometheus.prom_list_metrics github.projects_list github.projects_write okta.okta_create_user slack.create_usergroup prometheus.prom_query_range +q056,Search Grafana dashboards,verb_product,grafana.grafana_get_dashboard,0.0,1.0,1.0,0.5000,0.6309,2,1,209,1.0,1,359,1.0,1,732,1.0,grafana.grafana_get_dashboard grafana.grafana_search_dashboards jira.list_boards grafana.grafana_create_annotation jira.update_board jira.delete_board jira.create_board slack.list_bookmarks slack.update_bookmark github.list_dependabot_alerts +q057,Trigger a CircleCI pipeline,verb_product,gitlab.list_pipelines,0.0,0.0,0.0,0.1111,0.3010,9,3,667,0.0,2,695,0.0,1,674,1.0,gitlab.list_pipelines gitlab.get_pipeline gitlab.create_pipeline gitlab.delete_pipeline circleci.circleci_get_pipeline gitlab.update_pipeline circleci.circleci_list_pipelines gitlab.gitlab_create_pipeline circleci.circleci_trigger_pipeline stripe.stripe_list_invoices +q058,Run terraform plan,identifier,terraform.tf_plan,1.0,1.0,1.0,1.0000,1.0000,1,1,202,1.0,1,342,1.0,1,682,1.0,terraform.tf_plan postgres.explain_query slack.list_canvases github.list_branches gitlab.update_label linear.update_label gitlab.get_pipeline jira.list_attachments linear.list_attachments slack.update_pin +q059,Create an Asana task,verb_product,asana.asana_get_task,0.0,1.0,1.0,0.3333,0.5000,3,1,245,1.0,1,387,1.0,1,747,1.0,asana.asana_get_task asana.asana_list_tasks asana.asana_create_task asana.asana_update_task clickup.clickup_list_tasks clickup.clickup_create_task clickup.clickup_update_task clickup.clickup_get_task grafana.grafana_get_dashboard asana.asana_create_project +q060,Create a Trello card,verb_product,trello.trello_list_cards,0.0,1.0,1.0,0.5000,0.6309,2,1,245,1.0,1,382,1.0,1,796,1.0,trello.trello_list_cards trello.trello_create_card trello.trello_create_board stripe.stripe_create_refund stripe.stripe_create_product jira.create_board slack.create_reminder trello.trello_add_comment jira.get_board github.create_runner +q061,Create a ClickUp task,verb_product,clickup.clickup_get_task,0.0,0.0,0.0,0.1429,0.3333,7,3,736,0.0,2,785,0.0,1,764,1.0,clickup.clickup_get_task asana.asana_update_task asana.asana_get_task asana.asana_list_tasks clickup.clickup_list_tasks clickup.clickup_update_task clickup.clickup_create_task asana.asana_create_task notion.update_data_source github.list_tags +q062,Create a Confluence page,verb_product,confluence.confluence_get_page,0.0,0.0,1.0,0.2500,0.4307,4,2,488,0.0,1,416,1.0,1,742,1.0,confluence.confluence_get_page notion.notion_create_page notion.notion_update_page confluence.confluence_create_page confluence.confluence_update_page github.get_package github.list_packages notion.notion_get_page github.create_package confluence.confluence_search +q063,Send a Discord message,verb_product,slack.slack_post_message,0.0,1.0,1.0,0.5000,0.6309,2,1,226,1.0,1,376,1.0,1,698,1.0,slack.slack_post_message discord.discord_send_message slack.slack_search_messages twilio.twilio_list_messages github.list_code_scanning_alerts github.list_secret_scanning_alerts discord.discord_list_members discord.discord_get_channel github.get_secret_scanning_alert mysql.mysql_list_tables +q064,Search Elasticsearch,verb_product,elasticsearch.es_search,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,377,1.0,1,706,1.0,elasticsearch.es_search github.search_repositories notion.notion_search brave_search.brave_local_search zendesk.zendesk_search slack.slack_search_messages confluence.confluence_search linear.linear_search_issues gmail.gmail_search_emails stripe.stripe_create_price +q065,Get a Redis key,verb_product,redis.redis_keys,0.0,0.0,1.0,0.2500,0.4307,4,2,397,0.0,1,289,1.0,1,602,1.0,redis.redis_keys playwright.browser_press_key postgres.query redis.redis_get redis.redis_set kubernetes.k8s_delete redis.redis_delete kubernetes.k8s_apply kubernetes.k8s_list_deployments filesystem.read_media_file +q066,Find MongoDB documents,verb_product,github.get_discussion_comments,0.0,0.0,0.0,0.0000,0.0000,286,96,22835,0.0,58,22198,0.0,29,21618,0.0,github.get_discussion_comments kubernetes.k8s_list_deployments datadog.datadog_query_metrics vercel.vercel_list_deployments linear.list_comments notion.list_comments docker.docker_images linear.delete_comment notion.delete_comment notion.notion_get_comments +q067,List SQLite tables,verb_product,postgres.list_tables,0.0,1.0,1.0,0.3333,0.5000,3,1,151,1.0,1,260,1.0,1,626,1.0,postgres.list_tables mysql.mysql_list_tables sqlite.sqlite_list_tables github.list_labels supabase.supabase_list_tables gitlab.list_labels linear.list_labels postgres.describe_table slack.list_files github.list_dependabot_alerts +q068,Describe a postgres table,verb_product,postgres.describe_table,1.0,1.0,1.0,1.0000,1.0000,1,1,160,1.0,1,258,1.0,1,577,1.0,postgres.describe_table sqlite.sqlite_list_tables supabase.supabase_list_tables playwright.browser_tab_close sqlite.sqlite_describe_table postgres.list_tables mysql.mysql_list_tables github.get_label gitlab.get_label notion.notion_create_database +q069,github_create_issue,identifier,github.create_issue,1.0,1.0,1.0,1.0000,0.8956,1,1,357,1.0,1,522,1.0,1,1073,1.0,github.create_issue jira.jira_create_issue gitlab.gitlab_create_issue jira.jira_update_issue github.issue_read github.issue_write github.list_issues linear.linear_create_issue github.search_issues linear.create_user +q070,createIssue,identifier,github.create_issue,1.0,1.0,1.0,1.0000,0.9411,1,1,356,1.0,1,521,1.0,1,1072,1.0,github.create_issue jira.jira_create_issue gitlab.gitlab_create_issue jira.jira_update_issue github.issue_read github.issue_write github.list_issues linear.linear_create_issue github.search_issues linear.create_user +q071,list-directory,identifier,filesystem.list_directory,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,281,1.0,1,629,1.0,filesystem.list_directory filesystem.directory_tree filesystem.list_directory_with_sizes filesystem.create_directory filesystem.list_allowed_directories github.star_repository github.delete_repository github.unstar_repository github.list_repository_collaborators stripe.stripe_list_customers +q072,sequential thinking next thought,paraphrase,git.git_checkout,0.0,0.0,0.0,0.0000,0.0000,116,39,9799,0.0,24,9727,0.0,12,9475,0.0,git.git_checkout +q073,What time is it in Tokyo,paraphrase,notion.list_users_tokens,0.0,0.0,0.0,0.0000,0.0000,125,42,10496,0.0,25,10044,0.0,13,10182,0.0,notion.list_users_tokens filesystem.list_directory github.get_hook notion.notion_get_block github.list_hooks github.actions_get_job_logs notion.get_users_token github.create_hook github.delete_hook notion.notion_get_comments +q074,Convert time between timezones,verb_product,github.list_milestones,0.0,0.0,0.0,0.0000,0.0000,127,43,10789,0.0,26,10502,0.0,13,10229,0.0,github.list_milestones gitlab.list_milestones cloudflare.cf_list_zones github.get_milestone gitlab.get_milestone github.get_file_contents gitlab.list_pipelines github.create_milestone gitlab.create_milestone github.delete_milestone +q075,Star a GitHub repo,verb_product,github.unstar_repository,0.0,1.0,1.0,0.3333,0.5000,3,1,241,1.0,1,412,1.0,1,766,1.0,github.unstar_repository github.fork_repository github.star_repository github.search_repositories github.create_repository bitbucket.bitbucket_list_repos github.delete_repository github.get_repository_tree slack.slack_reply_to_thread playwright.browser_snapshot +q076,Fork a repository,resource,github.fork_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,248,1.0,1,386,1.0,1,831,1.0,github.fork_repository github.star_repository github.create_repository github.unstar_repository github.delete_repository github.get_repository_tree github.search_repositories github.list_repository_collaborators github.list_starred_repositories datadog.datadog_create_monitor +q077,Get the latest GitHub release,verb_product,gitlab.get_release,0.0,0.0,0.0,0.1429,0.3333,7,3,731,0.0,2,767,0.0,1,745,1.0,gitlab.get_release github.list_releases gitlab.update_release gitlab.list_releases gitlab.create_release gitlab.delete_release github.get_latest_release github.get_release_by_tag brave_search.brave_local_search slack.delete_canvase +q078,Create a gist,resource,github.create_gist,1.0,1.0,1.0,1.0000,1.0000,1,1,229,1.0,1,413,1.0,1,814,1.0,github.create_gist github.get_gist github.update_gist git.git_status github.create_issue git.git_reset jira.create_sprint github.list_gists slack.create_pin github.create_milestone +q079,Search GitHub code,verb_product,github.search_code,1.0,1.0,1.0,1.0000,1.0000,1,1,214,1.0,1,409,1.0,1,751,1.0,github.search_code memory.search_nodes shopify.shopify_create_order github.get_code_scanning_alert github.list_code_scanning_alerts hubspot.hubspot_create_deal memory.open_nodes hubspot.hubspot_get_contact linear.update_comment time.convert_time +q080,List GitHub notifications,verb_product,github.list_notifications,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,283,1.0,1,620,1.0,github.list_notifications github.dismiss_notification github.get_notification_details github.mark_all_notifications_read notion.notion_get_comments mongodb.mongo_list_collections notion.notion_create_database stripe.stripe_list_subscriptions grafana.grafana_create_annotation shopify.shopify_list_customers +q081,Assign Copilot to an issue,verb_product,github.create_issue,0.0,0.0,0.0,0.0000,0.0000,49,17,5013,0.0,10,4762,0.0,5,4652,0.0,github.create_issue github.list_issues github.search_issues jira.jira_get_issue github.issue_read jira.jira_assign_issue github.issue_write github.add_issue_comment linear.linear_get_issue gitlab.gitlab_get_issue +q082,Get secret scanning alerts,resource,github.list_code_scanning_alerts,0.0,1.0,1.0,0.3333,0.5068,3,1,276,1.0,1,446,1.0,1,789,1.0,github.list_code_scanning_alerts github.list_dependabot_alerts github.list_secret_scanning_alerts linear.list_users github.get_code_scanning_alert github.list_environments github.get_secret_scanning_alert notion.list_users_tokens google_calendar.gcal_list_calendars okta.okta_list_users +q083,Create or update a file on GitHub,verb_product,git.git_checkout,0.0,0.0,0.0,0.0000,0.0000,27,9,2203,0.0,6,2360,0.0,3,2288,0.0,git.git_checkout github.get_check_run hubspot.hubspot_create_deal git.git_show hubspot.hubspot_get_contact git.git_branch hubspot.hubspot_list_contacts hubspot.hubspot_create_contact git.git_status hubspot.hubspot_search_contacts +q084,List collaborators on a repo,resource,github.fork_repository,0.0,0.0,0.0,0.0000,0.0000,31,11,3009,0.0,7,3110,0.0,4,3597,0.0,github.fork_repository github.star_repository github.unstar_repository github.create_repository github.delete_repository github.get_repository_tree asana.asana_create_project github.search_repositories bitbucket.bitbucket_list_repos linear.linear_create_project +q085,Reply in a Slack thread,verb_product,slack.slack_reply_to_thread,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,368,1.0,1,747,1.0,slack.slack_reply_to_thread slack.slack_get_thread_replies filesystem.read_file memory.read_graph slack.slack_add_reaction github.issue_read slack.slack_post_message trello.trello_add_comment slack.slack_list_channels slack.slack_create_channel +q086,Add an emoji reaction on Slack,verb_product,slack.slack_get_users,0.0,0.0,1.0,0.2500,0.3274,4,2,471,0.0,1,379,1.0,1,729,1.0,slack.slack_get_users slack.slack_list_users slack.slack_upload_file slack.slack_add_reaction slack.slack_join_channel slack.slack_post_message slack.slack_leave_channel slack.slack_list_channels slack.slack_create_channel slack.slack_reply_to_thread +q087,Search Slack messages,verb_product,slack.slack_post_message,0.0,1.0,1.0,0.5000,0.6309,2,1,210,1.0,1,344,1.0,1,666,1.0,slack.slack_post_message slack.slack_search_messages twilio.twilio_list_messages slack.slack_get_users discord.discord_send_message slack.slack_get_user_profile slack.slack_list_users figma.figma_get_images slack.slack_list_channels mysql.mysql_list_tables +q088,Create a Slack channel,verb_product,slack.slack_join_channel,0.0,0.0,1.0,0.2000,0.3869,5,2,405,0.0,1,312,1.0,1,681,1.0,slack.slack_join_channel slack.slack_list_channels slack.slack_leave_channel discord.discord_get_channel slack.slack_create_channel slack.slack_set_channel_topic discord.discord_list_channels slack.slack_get_channel_history slack.slack_get_thread_replies github.get_secret_scanning_alert +q089,Transition a Jira issue,verb_product,jira.jira_get_issue,0.0,0.0,0.0,0.0000,0.0000,153,51,12624,0.0,31,12368,0.0,16,12436,0.0,jira.jira_get_issue github.create_issue github.list_issues jira.jira_assign_issue jira.jira_create_issue jira.jira_update_issue github.search_issues jira.jira_search_issues gitlab.gitlab_get_issue linear.linear_get_issue +q090,Search Jira with JQL,identifier,github.actions_get_job_logs,0.0,0.0,0.0,0.0000,0.0000,155,52,12765,0.0,31,12256,0.0,16,12345,0.0,github.actions_get_job_logs mysql.mysql_query filesystem.write_file jira.jira_list_projects gmail.gmail_trash_email sqlite.sqlite_query github.projects_list filesystem.list_directory_with_sizes mysql.mysql_list_tables sqlite.sqlite_list_tables +q091,Add a worklog to Jira,verb_product,jira.jira_get_issue,0.0,0.0,1.0,0.2000,0.3869,5,2,478,0.0,1,399,1.0,1,814,1.0,jira.jira_get_issue trello.trello_create_card jira.jira_add_comment trello.trello_list_cards jira.jira_add_worklog trello.trello_create_board jira.jira_assign_issue slack.slack_reply_to_thread jira.jira_create_issue notion.notion_create_page +q092,List Linear teams,verb_product,linear.linear_list_teams,1.0,1.0,1.0,1.0000,1.0000,1,1,158,1.0,1,375,1.0,1,708,1.0,linear.linear_list_teams github.get_teams github.get_team_members linear.linear_list_issues linear.linear_create_issue gitlab.get_release linear.linear_update_issue postgres.list_schemas linear.linear_get_issue github.list_releases +q093,Comment on a Linear issue,verb_product,linear.linear_get_issue,0.0,0.0,0.0,0.0000,0.0000,146,49,12231,0.0,30,12034,0.0,15,11719,0.0,linear.linear_get_issue github.list_issues github.search_issues github.create_issue linear.linear_list_issues jira.jira_get_issue sentry.sentry_get_issue jira.jira_search_issues jira.jira_create_issue gitlab.gitlab_get_issue +q094,Move a local file,paraphrase,slack.list_files,0.0,1.0,1.0,0.5000,0.6309,2,1,213,1.0,1,405,1.0,1,780,1.0,slack.list_files filesystem.move_file filesystem.read_file slack.create_file jira.create_filter github.delete_file slack.delete_file jira.list_filters slack.update_file filesystem.search_files +q095,Recursively search files by pattern,paraphrase,github.update_runner,0.0,0.0,0.0,0.0000,0.0000,96,32,8562,0.0,20,8492,0.0,10,8272,0.0,github.update_runner jira.update_filter jira.update_sprint slack.update_reminder jira.update_version jira.get_attachment linear.get_attachment jira.update_attachment github.update_check_run linear.update_attachment +q096,Edit a file in place with old and new string,paraphrase,jira.get_sprint,0.0,0.0,0.0,0.0000,0.0000,89,30,8310,0.0,18,7998,0.0,9,7764,0.0,jira.get_sprint github.actions_run_trigger stripe.stripe_list_invoices jira.create_sprint jira.delete_sprint jira.update_sprint jira.list_sprints slack.list_reminders stripe.stripe_get_customer jira.list_versions +q097,Read the whole knowledge graph,paraphrase,memory.read_graph,1.0,1.0,1.0,1.0000,1.0000,1,1,171,1.0,1,280,1.0,1,627,1.0,memory.read_graph pagerduty.pagerduty_acknowledge_incident git.git_branch google_drive.gdrive_search gitlab.get_release gmail.gmail_trash_email playwright.browser_snapshot git.git_create_branch slack.create_pin gitlab.gitlab_create_pipeline +q098,Search memory nodes,verb_product,memory.search_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,202,1.0,1,354,1.0,1,652,1.0,memory.search_nodes figma.figma_get_file_nodes memory.open_nodes pagerduty.pagerduty_list_incidents mongodb.mongo_delete pagerduty.pagerduty_acknowledge_incident zendesk.zendesk_search docker.docker_stop docker.docker_ps slack.list_reminders +q099,Show unstaged git diff,verb_product,git.git_diff,0.0,1.0,1.0,0.3333,0.5000,3,1,190,1.0,1,340,1.0,1,719,1.0,git.git_diff git.git_diff_staged git.git_diff_unstaged gmail.gmail_modify_email google_drive.gdrive_get_file google_drive.gdrive_list_folder google_drive.gdrive_create_file filesystem.edit_file git.git_add filesystem.read_media_file +q100,Stage files with git add,identifier,git.git_add,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,355,1.0,1,678,1.0,git.git_add jira.jira_add_comment jira.jira_add_worklog trello.trello_add_comment filesystem.list_allowed_directories memory.add_observations git.git_diff github.add_issue_comment git.git_branch git.git_status +q101,Create a Shopify product,verb_product,shopify.shopify_get_product,0.0,1.0,1.0,0.3333,0.5000,3,1,237,1.0,1,379,1.0,1,703,1.0,shopify.shopify_get_product shopify.shopify_list_products shopify.shopify_create_product shopify.shopify_update_product stripe.stripe_create_product shopify.shopify_create_order github.projects_get shopify.shopify_get_order github.projects_list shopify.shopify_list_orders +q102,Cancel a Stripe subscription,verb_product,stripe.stripe_list_subscriptions,0.0,1.0,1.0,0.5000,0.6309,2,1,187,1.0,1,320,1.0,1,631,1.0,stripe.stripe_list_subscriptions stripe.stripe_cancel_subscription hubspot.hubspot_list_contacts stripe.stripe_create_invoice hubspot.hubspot_create_contact stripe.stripe_list_customers stripe.stripe_create_payment_intent +q103,Create a payment intent,resource,stripe.stripe_get_payment_intent,0.0,1.0,1.0,0.3333,0.5000,3,1,272,1.0,1,409,1.0,1,780,1.0,stripe.stripe_get_payment_intent github.get_file_contents stripe.stripe_create_payment_intent sentry.sentry_list_events github.update_environment notion.notion_get_comments pagerduty.pagerduty_get_incident linear.list_comments pagerduty.pagerduty_create_incident notion.list_comments +q104,Acknowledge a PagerDuty incident,verb_product,pagerduty.pagerduty_get_incident,0.0,0.0,1.0,0.2000,0.3869,5,2,481,0.0,1,405,1.0,1,804,1.0,pagerduty.pagerduty_get_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_create_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_acknowledge_incident github.get_notification_details github.get_discussion_comments linear.list_comments github.update_issue_comment notion.list_comments +q105,Create a Datadog monitor,verb_product,datadog.datadog_get_monitor,0.0,1.0,1.0,0.3333,0.5000,3,1,259,1.0,1,388,1.0,1,786,1.0,datadog.datadog_get_monitor datadog.datadog_list_monitors datadog.datadog_create_monitor github.list_starred_repositories mongodb.mongo_insert datadog.datadog_list_logs mongodb.mongo_list_collections datadog.datadog_query_metrics github.fork_repository github.star_repository +q106,Resolve a Sentry issue,paraphrase,github.list_issues,0.0,0.0,0.0,0.0000,0.0000,232,78,18579,0.0,47,17991,0.0,24,17923,0.0,github.list_issues github.search_issues sentry.sentry_get_issue github.create_issue jira.jira_get_issue sentry.sentry_list_issues gitlab.gitlab_get_issue linear.linear_get_issue jira.jira_search_issues jira.jira_assign_issue +q107,Scale a Kubernetes deployment,verb_product,vercel.vercel_get_deployment,0.0,1.0,1.0,0.3333,0.5000,3,1,234,1.0,1,343,1.0,1,652,1.0,vercel.vercel_get_deployment kubernetes.k8s_list_deployments kubernetes.k8s_scale_deployment vercel.vercel_list_deployments linear.delete_comment notion.delete_comment github.delete_environment linear.update_comment notion.update_comment linear.list_comments +q108,Apply a Kubernetes manifest,verb_product,mongodb.mongo_insert,0.0,0.0,0.0,0.0000,0.0000,203,68,16305,0.0,41,15821,0.0,21,15741,0.0,mongodb.mongo_insert github.actions_list github.get_milestone gitlab.get_milestone asana.asana_list_tasks github.list_milestones gitlab.list_milestones github.create_milestone gitlab.create_milestone github.delete_milestone +q109,Stop a docker container,verb_product,time.convert_time,0.0,0.0,0.0,0.0000,0.0000,190,64,15302,0.0,38,14689,0.0,19,14309,0.0,time.convert_time mongodb.mongo_insert notion.notion_search slack.create_reminder okta.okta_list_users github.actions_run_trigger notion.notion_list_users confluence.confluence_search github.get_code_scanning_alert github.list_code_scanning_alerts +q110,Build a docker image,verb_product,docker.docker_images,0.0,0.0,0.0,0.1000,0.2891,10,4,797,0.0,2,665,0.0,1,645,1.0,docker.docker_images discord.discord_send_message figma.figma_get_file docker.docker_rm figma.figma_get_images docker.docker_exec figma.figma_get_comments docker.docker_logs figma.figma_get_file_nodes docker.docker_build +q111,Click an element in Playwright,verb_product,github.label_write,0.0,0.0,0.0,0.0000,0.0000,171,57,14078,0.0,35,13845,0.0,18,13724,0.0,github.label_write github.pull_request_review_write pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_acknowledge_incident +q112,Type text into a browser input,paraphrase,gitlab.list_snippets,0.0,1.0,1.0,0.5000,0.4796,2,1,266,1.0,1,396,1.0,1,699,1.0,gitlab.list_snippets playwright.browser_type aws_s3.s3_put_object playwright.browser_snapshot clickup.clickup_update_task playwright.browser_navigate notion.notion_update_page playwright.browser_wait_for jira.get_sprint playwright.browser_click +q113,List GitLab projects,verb_product,github.projects_get,0.0,1.0,1.0,0.5000,0.6309,2,1,204,1.0,1,284,1.0,1,610,1.0,github.projects_get gitlab.gitlab_list_projects github.projects_list linear.linear_list_projects jira.jira_list_projects github.projects_write sentry.sentry_list_projects vercel.vercel_list_projects linear.linear_create_project asana.asana_create_project +q114,Create a Bitbucket pull request,verb_product,github.pull_request_read,0.0,0.0,0.0,0.0000,0.0000,272,91,21988,0.0,55,21340,0.0,28,20983,0.0,github.pull_request_read github.list_pull_requests github.merge_pull_request github.create_pull_request github.update_pull_request github.search_pull_requests github.pull_request_review_write gitlab.gitlab_get_merge_request gitlab.gitlab_list_merge_requests github.update_pull_request_branch +q115,Create a Gmail draft,verb_product,gmail.gmail_create_draft,1.0,1.0,1.0,1.0000,1.0000,1,1,214,1.0,1,402,1.0,1,717,1.0,gmail.gmail_create_draft filesystem.read_file gmail.gmail_trash_email jira.create_filter gmail.gmail_send_email jira.update_filter gmail.gmail_modify_email memory.delete_relations gmail.gmail_get_email github.delete_artifact +q116,Share a Google Drive file,verb_product,filesystem.read_file,0.0,0.0,0.0,0.0000,0.0000,261,87,20738,0.0,53,20393,0.0,27,20238,0.0,filesystem.read_file filesystem.move_file filesystem.write_file filesystem.edit_file slack.get_file github.delete_file slack.create_file slack.delete_file slack.list_files jira.get_filter +q117,List Google Calendar events,verb_product,google_calendar.gcal_list_events,1.0,1.0,1.0,1.0000,1.0000,1,1,301,1.0,1,461,1.0,1,771,1.0,google_calendar.gcal_list_events google_calendar.gcal_create_event sentry.sentry_list_events google_calendar.gcal_update_event google_calendar.gcal_delete_event github.create_environment kubernetes.k8s_get_events github.list_environments google_calendar.gcal_list_calendars memory.create_entities +q118,Delete a calendar event,resource,google_calendar.gcal_create_event,0.0,0.0,1.0,0.2500,0.4307,4,2,576,0.0,1,461,1.0,1,834,1.0,google_calendar.gcal_create_event google_calendar.gcal_list_events google_calendar.gcal_update_event google_calendar.gcal_delete_event sentry.sentry_list_events github.create_environment kubernetes.k8s_get_events github.get_environment kubernetes.k8s_scale_deployment github.list_environments +q119,List Vercel projects,verb_product,vercel.vercel_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,139,1.0,1,241,1.0,1,585,1.0,vercel.vercel_list_projects linear.linear_list_projects jira.jira_list_projects sentry.sentry_list_projects gitlab.gitlab_list_projects vercel.vercel_create_project linear.linear_create_project asana.asana_create_project github.projects_get github.projects_list +q120,Select rows from Supabase,verb_product,supabase.supabase_select,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,354,1.0,1,711,1.0,supabase.supabase_select supabase.supabase_insert supabase.supabase_update supabase.supabase_invoke_fn supabase.supabase_list_tables slack.update_canvase notion.notion_query_database gitlab.update_release linear.update_user jira.update_board +q121,Post a Figma comment,verb_product,linear.get_comment,0.0,0.0,0.0,0.1250,0.3155,8,3,704,0.0,2,748,0.0,1,727,1.0,linear.get_comment notion.get_comment linear.list_comments notion.list_comments figma.figma_get_comments linear.create_comment notion.create_comment figma.figma_post_comment jira.jira_add_comment linear.update_comment +q122,List Okta users,verb_product,linear.list_users,0.0,1.0,1.0,0.5000,0.6309,2,1,195,1.0,1,312,1.0,1,646,1.0,linear.list_users okta.okta_list_users linear.get_user github.search_users slack.slack_get_users notion.get_users_token slack.slack_list_users notion.notion_list_users slack.list_usergroups linear.create_user +q123,Range query Prometheus,verb_product,sentry.sentry_list_projects,0.0,0.0,0.0,0.1000,0.2891,10,4,940,0.0,2,808,0.0,1,788,1.0,sentry.sentry_list_projects linear.create_user datadog.datadog_query_metrics github.projects_get prometheus.prom_list_metrics github.projects_list github.projects_write okta.okta_create_user slack.create_usergroup prometheus.prom_query_range +q124,Create a Grafana annotation,verb_product,grafana.grafana_create_annotation,1.0,1.0,1.0,1.0000,1.0000,1,1,209,1.0,1,314,1.0,1,602,1.0,grafana.grafana_create_annotation github.mark_all_notifications_read github.list_notifications grafana.grafana_get_dashboard github.dismiss_notification memory.add_observations grafana.grafana_search_dashboards notion.notion_search github.get_notification_details memory.create_relations +q125,Apply terraform,identifier,playwright.browser_wait_for,0.0,0.0,0.0,0.0000,0.0000,334,112,26006,0.0,67,25091,0.0,34,24881,0.0,playwright.browser_wait_for playwright.browser_fill_form trello.trello_create_board linear.linear_create_comment github.list_repository_collaborators stripe.stripe_create_customer salesforce.salesforce_create_record +q126,List Asana tasks,verb_product,asana.asana_list_tasks,1.0,1.0,1.0,1.0000,1.0000,1,1,192,1.0,1,386,1.0,1,718,1.0,asana.asana_list_tasks asana.asana_get_task clickup.clickup_list_tasks asana.asana_create_task asana.asana_update_task clickup.clickup_get_task grafana.grafana_get_dashboard github.list_tags asana.asana_create_project slack.list_canvases +q127,Comment on a Trello card,verb_product,trello.trello_list_cards,0.0,0.0,0.0,0.1250,0.3155,8,3,746,0.0,2,819,0.0,1,797,1.0,trello.trello_list_cards trello.trello_create_card trello.trello_create_board stripe.stripe_create_refund stripe.stripe_create_product jira.create_board slack.create_reminder trello.trello_add_comment jira.get_board github.create_runner +q128,Update a ClickUp task,verb_product,clickup.clickup_get_task,0.0,0.0,0.0,0.1667,0.3562,6,2,448,0.0,2,785,0.0,1,764,1.0,clickup.clickup_get_task asana.asana_update_task asana.asana_get_task asana.asana_list_tasks clickup.clickup_list_tasks clickup.clickup_update_task clickup.clickup_create_task asana.asana_create_task notion.update_data_source github.list_tags +q129,Search Confluence,verb_product,confluence.confluence_search,1.0,1.0,1.0,1.0000,1.0000,1,1,187,1.0,1,369,1.0,1,848,1.0,confluence.confluence_search hubspot.hubspot_search_contacts confluence.confluence_get_page confluence.confluence_create_page confluence.confluence_update_page cloudflare.cf_purge_cache github.actions_run_trigger pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident github.add_comment_to_pending_review +q130,Create a HubSpot deal,verb_product,hubspot.hubspot_create_deal,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,378,1.0,1,676,1.0,hubspot.hubspot_create_deal github.get_dependabot_alert gitlab.delete_label linear.delete_label slack.slack_post_message postgres.describe_table sqlite.sqlite_describe_table gitlab.delete_job github.list_dependabot_alerts slack.delete_pin +q131,List Zendesk tickets,verb_product,zendesk.zendesk_list_tickets,1.0,1.0,1.0,1.0000,1.0000,1,1,220,1.0,1,388,1.0,1,713,1.0,zendesk.zendesk_list_tickets zendesk.zendesk_get_ticket zendesk.zendesk_create_ticket zendesk.zendesk_update_ticket pagerduty.pagerduty_list_incidents aws_s3.s3_list_buckets clickup.clickup_get_task zendesk.zendesk_search github.get_file_contents pagerduty.pagerduty_resolve_incident +q132,List Discord channels,verb_product,discord.discord_list_channels,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,308,1.0,1,622,1.0,discord.discord_list_channels slack.slack_list_channels discord.discord_get_channel slack.slack_join_channel slack.slack_create_channel slack.slack_leave_channel slack.slack_get_channel_history slack.slack_set_channel_topic discord.discord_send_message discord.discord_list_members +q133,Set a Redis key,verb_product,redis.redis_keys,0.0,0.0,1.0,0.2000,0.3869,5,2,397,0.0,1,289,1.0,1,602,1.0,redis.redis_keys playwright.browser_press_key postgres.query redis.redis_get redis.redis_set kubernetes.k8s_delete redis.redis_delete kubernetes.k8s_apply kubernetes.k8s_list_deployments filesystem.read_media_file +q134,Insert MongoDB documents,verb_product,github.get_discussion_comments,0.0,0.0,0.0,0.0000,0.0000,287,96,22835,0.0,58,22198,0.0,29,21618,0.0,github.get_discussion_comments kubernetes.k8s_list_deployments datadog.datadog_query_metrics vercel.vercel_list_deployments linear.list_comments notion.list_comments docker.docker_images linear.delete_comment notion.delete_comment notion.notion_get_comments +q135,Run a MySQL query,verb_product,mysql.mysql_query,1.0,1.0,1.0,1.0000,1.0000,1,1,178,1.0,1,293,1.0,1,721,1.0,mysql.mysql_query prometheus.prom_query sqlite.sqlite_query salesforce.salesforce_query postgres.explain_query prometheus.prom_query_range datadog.datadog_query_metrics notion.notion_query_database github.pull_request_read postgres.query +q136,Index a document in Elasticsearch,verb_product,linear.linear_search_issues,0.0,0.0,0.0,0.0000,0.0000,293,98,23327,0.0,59,22549,0.0,30,22277,0.0,linear.linear_search_issues elasticsearch.es_search notion.notion_search brave_search.brave_local_search zendesk.zendesk_search slack.slack_search_messages confluence.confluence_search gmail.gmail_search_emails stripe.stripe_create_price stripe.stripe_create_product +q137,Get an S3 object,verb_product,aws_s3.s3_get_object,1.0,1.0,1.0,1.0000,1.0000,1,1,251,1.0,1,392,1.0,1,630,1.0,aws_s3.s3_get_object aws_s3.s3_put_object aws_s3.s3_list_objects aws_s3.s3_delete_object asana.asana_create_project jira.jira_list_projects linear.linear_list_projects gitlab.gitlab_list_projects sentry.sentry_list_projects memory.add_observations +q138,List Cloudflare zones,verb_product,cloudflare.cf_list_zones,1.0,1.0,1.0,1.0000,1.0000,1,1,164,1.0,1,329,1.0,1,673,1.0,cloudflare.cf_list_zones memory.open_nodes zendesk.zendesk_search github.list_milestones gitlab.list_milestones zendesk.zendesk_get_ticket confluence.confluence_search zendesk.zendesk_list_tickets zendesk.zendesk_create_ticket zendesk.zendesk_update_ticket +q139,Create a Stripe invoice,verb_product,stripe.stripe_list_invoices,0.0,1.0,1.0,0.5000,0.6309,2,1,194,1.0,1,361,1.0,1,701,1.0,stripe.stripe_list_invoices stripe.stripe_create_invoice supabase.supabase_invoke_fn notion.notion_create_page stripe.stripe_create_price stripe.stripe_list_customers notion.notion_get_comments notion.notion_create_database stripe.stripe_create_refund github.list_notifications +q140,List GitHub Actions workflows,verb_product,jira.jira_add_worklog,0.0,0.0,0.0,0.0000,0.0000,62,21,6073,0.0,13,6075,0.0,7,6292,0.0,jira.jira_add_worklog github.actions_get_job_logs +q141,Get GitHub Actions job logs,verb_product,docker.docker_logs,0.0,1.0,1.0,0.3333,0.5000,3,1,276,1.0,1,437,1.0,1,759,1.0,docker.docker_logs kubernetes.k8s_pod_logs github.actions_get_job_logs datadog.datadog_list_logs git.git_log github.list_gists playwright.browser_tab_close github.list_tags brave_search.brave_local_search gitlab.list_jobs +q142,Create a GitHub repository,verb_product,github.fork_repository,0.0,0.0,1.0,0.2500,0.4307,4,2,515,0.0,1,388,1.0,1,773,1.0,github.fork_repository github.star_repository github.unstar_repository github.create_repository github.delete_repository github.get_repository_tree github.search_repositories github.list_repository_collaborators github.list_starred_repositories filesystem.create_directory +q143,Checkout a git branch,verb_product,git.git_branch,0.0,0.0,0.0,0.1429,0.3333,7,3,714,0.0,2,770,0.0,1,750,1.0,git.git_branch github.list_branches github.create_branch git.git_create_branch github.update_pull_request_branch google_drive.gdrive_search git.git_checkout brave_search.brave_web_search jira.get_attachment brave_search.brave_local_search +q144,Show git log,identifier,git.git_log,1.0,1.0,1.0,1.0000,1.0000,1,1,274,1.0,1,397,1.0,1,762,1.0,git.git_log datadog.datadog_list_logs github.actions_get_job_logs docker.docker_logs git.git_show kubernetes.k8s_pod_logs git.git_commit github.list_gists git.git_checkout jira.jira_add_worklog +q145,List postgres schemas,verb_product,postgres.list_schemas,1.0,1.0,1.0,1.0000,1.0000,1,1,195,1.0,1,302,1.0,1,615,1.0,postgres.list_schemas gmail.gmail_search_emails filesystem.search_files memory.search_nodes github.search_users github.search_issues slack.slack_search_messages github.search_commits github.list_branches docker.docker_images +q146,Open nodes in the knowledge graph,verb_product,memory.read_graph,0.0,0.0,0.0,0.0000,0.0000,106,36,9236,0.0,22,9029,0.0,11,8787,0.0,memory.read_graph pagerduty.pagerduty_acknowledge_incident git.git_branch google_drive.gdrive_search gitlab.get_release gmail.gmail_trash_email playwright.browser_snapshot git.git_create_branch slack.create_pin gitlab.gitlab_create_pipeline +q147,Get file metadata from the filesystem,paraphrase,github.create_milestone,0.0,0.0,0.0,0.0000,0.0000,97,33,8765,0.0,20,8530,0.0,10,8310,0.0,github.create_milestone gitlab.create_milestone github.delete_milestone gitlab.delete_milestone github.update_milestone gitlab.update_milestone github.get_milestone gitlab.get_milestone gmail.gmail_get_email linear.linear_list_teams +q148,Write a new local file,paraphrase,slack.list_files,0.0,0.0,0.0,0.0000,0.0000,87,29,7978,0.0,18,7948,0.0,9,7759,0.0,slack.list_files filesystem.move_file filesystem.read_file slack.create_file jira.create_filter github.delete_file slack.delete_file jira.list_filters slack.update_file filesystem.search_files +q149,Search GitHub issues,verb_product,github.list_issues,0.0,1.0,1.0,0.5000,0.6309,2,1,280,1.0,1,484,1.0,1,844,1.0,github.list_issues github.search_issues jira.jira_search_issues gitlab.gitlab_list_issues github.create_issue linear.linear_search_issues linear.linear_list_issues sentry.sentry_list_issues gitlab.gitlab_get_issue jira.jira_get_issue +q150,Get the authenticated GitHub user,paraphrase,linear.get_user,0.0,0.0,0.0,0.0000,0.0000,11,4,802,0.0,3,1045,0.0,2,1428,0.0,linear.get_user linear.update_user linear.list_users linear.create_user github.search_users linear.delete_user slack.get_usergroup okta.okta_get_user slack.update_usergroup notion.get_users_token diff --git a/experiments/mcp-tool-search/results/codedb_symbol.json b/experiments/mcp-tool-search/results/codedb_symbol.json new file mode 100644 index 00000000..93b87dfa --- /dev/null +++ b/experiments/mcp-tool-search/results/codedb_symbol.json @@ -0,0 +1,6978 @@ +{ + "summary": { + "n": 150, + "index_s": 0.44171834199994464, + "rss_mb": null, + "p50_ms": 20.615465000048516, + "p95_ms": 20.850576600042814, + "recall@1": 0.30666666666666664, + "recall@1_lo": 0.24, + "recall@1_hi": 0.38, + "recall@3": 0.5333333333333333, + "recall@3_lo": 0.4533333333333333, + "recall@3_hi": 0.6133333333333333, + "recall@5": 0.6133333333333333, + "recall@5_lo": 0.5333333333333333, + "recall@5_hi": 0.6866666666666666, + "mrr@10": 0.43221693121693117, + "mrr@10_lo": 0.3697010582010582, + "mrr@10_hi": 0.4974232804232804, + "ndcg@10": 0.4972285622400626, + "ndcg@10_lo": 0.43776722351442166, + "ndcg@10_hi": 0.5581462353915159 + }, + "meta": { + "name": "codedb_symbol", + "needs_model": false, + "needs_api": false, + "notes": "codedb_symbol fuzzy on guessed identifier tokens" + }, + "rows": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.list_issues", + "github.search_issues", + "github.create_issue", + "gitlab.gitlab_get_issue", + "jira.jira_get_issue", + "jira.jira_update_issue", + "jira.jira_search_issues", + "linear.linear_get_issue", + "gitlab.gitlab_list_issues", + "sentry.sentry_get_issue" + ], + "hit1": "github.list_issues", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.38009376671593426, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 341, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 466, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 802, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "kind": "verb_product", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "stripe.stripe_create_price", + "jira.jira_get_issue", + "jira.jira_add_comment", + "jira.jira_create_issue", + "clickup.clickup_get_task", + "jira.jira_create_sprint" + ], + "hit1": "zendesk.zendesk_create_ticket", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.125, + "ndcg@10": 0.31546487678572877, + "hit_rank": 8, + "k3_calls": 3, + "k3_tokens": 719, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 809, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 789, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "kind": "verb_product", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "ranking": [ + "slack.slack_post_message", + "slack.slack_search_messages", + "discord.discord_send_message", + "twilio.twilio_list_messages", + "slack.slack_get_users", + "slack.slack_list_users", + "github.get_secret_scanning_alert", + "mysql.mysql_list_tables", + "figma.figma_get_images", + "slack.slack_leave_channel" + ], + "hit1": "slack.slack_post_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "kind": "resource", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "ranking": [ + "github.list_pull_requests", + "github.search_pull_requests", + "github.pull_request_read", + "github.merge_pull_request", + "github.create_pull_request", + "github.update_pull_request", + "gitlab.gitlab_list_merge_requests", + "github.pull_request_review_write", + "gitlab.gitlab_get_merge_request", + "github.update_pull_request_branch" + ], + "hit1": "github.list_pull_requests", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 506, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1038, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "ranking": [ + "git.git_checkout", + "github.get_check_run", + "hubspot.hubspot_create_deal", + "git.git_show", + "hubspot.hubspot_get_contact", + "git.git_branch", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_create_contact", + "git.git_status", + "hubspot.hubspot_search_contacts" + ], + "hit1": "git.git_checkout", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 46, + "k3_calls": 16, + "k3_tokens": 4346, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 10, + "k5_tokens": 4440, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 5, + "k10_tokens": 4325, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "kind": "verb_product", + "gold": [ + "postgres.query" + ], + "alts": [], + "ranking": [ + "github.list_starred_repositories", + "github.search_repositories", + "gitlab.gitlab_list_projects", + "github.projects_get", + "okta.okta_get_user", + "datadog.datadog_list_monitors", + "slack.slack_post_message", + "gitlab.gitlab_list_merge_requests", + "github.star_repository", + "salesforce.salesforce_list_opportunities" + ], + "hit1": "github.list_starred_repositories", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 104, + "k3_calls": 35, + "k3_tokens": 9005, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 21, + "k5_tokens": 8697, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 11, + "k10_tokens": 8763, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "kind": "paraphrase", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "ranking": [ + "redis.redis_keys", + "prometheus.prom_list_metrics", + "redis.redis_set", + "github.get_notification_details", + "redis.redis_get", + "gitlab.delete_wiki", + "docker.docker_images", + "gitlab.list_wikis", + "github.list_hooks", + "git.git_diff_staged" + ], + "hit1": "redis.redis_keys", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 85, + "k3_calls": 29, + "k3_tokens": 7857, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 17, + "k5_tokens": 7449, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7634, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "kind": "paraphrase", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "ranking": [ + "google_drive.gdrive_list_folder", + "jira.list_filters", + "shopify.shopify_get_order", + "shopify.shopify_list_orders", + "shopify.shopify_create_order", + "jira.get_filter", + "filesystem.list_directory", + "slack.list_reminders", + "github.list_dependabot_alerts", + "stripe.stripe_list_customers" + ], + "hit1": "google_drive.gdrive_list_folder", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.14285714285714285, + "ndcg@10": 0.2129292954931993, + "hit_rank": 7, + "k3_calls": 3, + "k3_tokens": 648, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 691, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "ranking": [ + "memory.read_graph", + "pagerduty.pagerduty_acknowledge_incident", + "git.git_branch", + "google_drive.gdrive_search", + "gitlab.get_release", + "gmail.gmail_trash_email", + "playwright.browser_snapshot", + "git.git_create_branch", + "slack.create_pin", + "gitlab.gitlab_create_pipeline" + ], + "hit1": "memory.read_graph", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 99, + "k3_calls": 33, + "k3_tokens": 8732, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 20, + "k5_tokens": 8484, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 10, + "k10_tokens": 8254, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "kind": "verb_product", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "ranking": [ + "github.list_branches", + "git.git_branch", + "bitbucket.bitbucket_create_pr", + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "playwright.browser_navigate", + "github.list_releases", + "gitlab.list_releases", + "jira.create_version", + "slack.list_canvases" + ], + "hit1": "github.list_branches", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 500, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 810, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "kind": "verb_product", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "ranking": [ + "slack.slack_get_user_profile", + "gitlab.update_release", + "okta.okta_get_user", + "postgres.query", + "prometheus.prom_query", + "filesystem.read_file", + "okta.okta_list_users", + "linear.get_user", + "cloudflare.cf_purge_cache", + "gitlab.update_label" + ], + "hit1": "slack.slack_get_user_profile", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 111, + "k3_calls": 37, + "k3_tokens": 9389, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 23, + "k5_tokens": 9302, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9383, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q012", + "query": "Show git status", + "kind": "identifier", + "gold": [ + "git.git_status" + ], + "alts": [], + "ranking": [ + "git.git_status", + "terraform.tf_state_list", + "github.list_tags", + "github.list_artifacts", + "postgres.list_tables", + "okta.okta_get_user", + "github.get_teams", + "slack.slack_get_users", + "git.git_reset", + "notion.list_data_sources" + ], + "hit1": "git.git_status", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 602, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "kind": "paraphrase", + "gold": [ + "git.git_commit" + ], + "alts": [], + "ranking": [ + "slack.slack_list_channels", + "slack.slack_get_channel_history", + "discord.discord_list_channels", + "slack.slack_search_messages", + "slack.slack_leave_channel", + "discord.discord_get_channel", + "slack.slack_create_channel", + "slack.slack_set_channel_topic", + "github.list_code_scanning_alerts", + "stripe.stripe_cancel_subscription" + ], + "hit1": "slack.slack_list_channels", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 117, + "k3_calls": 39, + "k3_tokens": 9686, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 24, + "k5_tokens": 9576, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9348, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_get_issue", + "github.list_issues", + "github.search_issues", + "github.create_issue", + "linear.linear_list_issues", + "jira.jira_get_issue", + "sentry.sentry_get_issue", + "jira.jira_search_issues", + "jira.jira_create_issue", + "gitlab.gitlab_get_issue" + ], + "hit1": "linear.linear_get_issue", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 143, + "k3_calls": 48, + "k3_tokens": 12006, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 29, + "k5_tokens": 11681, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11704, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "kind": "paraphrase", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_get_issue", + "gitlab.gitlab_create_pipeline", + "linear.linear_list_teams", + "circleci.circleci_trigger_pipeline", + "linear.linear_list_issues", + "slack.slack_get_thread_replies", + "linear.linear_assign_issue", + "pagerduty.pagerduty_acknowledge_incident", + "linear.linear_create_issue", + "linear.linear_update_issue" + ], + "hit1": "linear.linear_get_issue", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.1111111111111111, + "ndcg@10": 0.3010299956639812, + "hit_rank": 9, + "k3_calls": 3, + "k3_tokens": 741, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 831, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 807, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "ranking": [ + "github.list_issues", + "github.search_issues", + "github.create_issue", + "gitlab.gitlab_get_issue", + "jira.jira_get_issue", + "jira.jira_update_issue", + "jira.jira_search_issues", + "linear.linear_get_issue", + "gitlab.gitlab_list_issues", + "sentry.sentry_get_issue" + ], + "hit1": "github.list_issues", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 41, + "k3_calls": 14, + "k3_tokens": 3866, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 9, + "k5_tokens": 4037, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 5, + "k10_tokens": 4516, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "kind": "ambiguous", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "ranking": [ + "github.pull_request_read", + "github.list_pull_requests", + "github.merge_pull_request", + "github.create_pull_request", + "github.update_pull_request", + "github.search_pull_requests", + "github.pull_request_review_write", + "gitlab.gitlab_get_merge_request", + "gitlab.gitlab_list_merge_requests", + "github.update_pull_request_branch" + ], + "hit1": "github.pull_request_read", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.38009376671593426, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 303, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 595, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1038, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "kind": "verb_product", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "ranking": [ + "jira.jira_add_worklog", + "github.actions_get_job_logs", + "playwright.browser_fill_form" + ], + "hit1": "jira.jira_add_worklog", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 65, + "k3_calls": 22, + "k3_tokens": 6335, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 13, + "k5_tokens": 6086, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 7, + "k10_tokens": 6281, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "kind": "resource", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "ranking": [ + "github.list_dependabot_alerts", + "github.get_dependabot_alert", + "linear.list_users", + "github.list_code_scanning_alerts", + "github.list_environments", + "notion.list_users_tokens", + "okta.okta_list_users", + "google_calendar.gcal_list_events", + "linear.linear_get_issue", + "linear.linear_list_teams" + ], + "hit1": "github.list_dependabot_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 747, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "kind": "verb_product", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "ranking": [ + "github.search_issues", + "github.list_issues", + "sentry.sentry_list_issues", + "github.create_issue", + "jira.jira_search_issues", + "gitlab.gitlab_list_issues", + "sentry.sentry_get_issue", + "linear.linear_search_issues", + "linear.linear_list_issues", + "jira.jira_get_issue" + ], + "hit1": "github.search_issues", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 303, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 507, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 841, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "ranking": [ + "stripe.stripe_get_customer", + "stripe.stripe_list_customers", + "stripe.stripe_create_customer", + "shopify.shopify_list_customers", + "stripe.stripe_create_refund", + "linear.create_user", + "stripe.stripe_create_price", + "notion.create_users_token", + "stripe.stripe_create_product", + "github.search_users" + ], + "hit1": "stripe.stripe_get_customer", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 203, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 720, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "kind": "paraphrase", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "ranking": [ + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_payment_intent", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_list_deployments", + "vercel.vercel_get_deployment", + "vercel.vercel_list_deployments", + "linear.update_comment", + "notion.update_comment", + "stripe.stripe_create_refund", + "stripe.stripe_create_invoice" + ], + "hit1": "stripe.stripe_get_payment_intent", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.1111111111111111, + "ndcg@10": 0.3010299956639812, + "hit_rank": 9, + "k3_calls": 3, + "k3_tokens": 678, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 706, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "kind": "verb_product", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "ranking": [ + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "gmail.gmail_send_email", + "figma.figma_get_file", + "gmail.gmail_search_emails", + "gmail.gmail_modify_email", + "gitlab.gitlab_get_file", + "figma.figma_get_file_nodes", + "filesystem.write_file", + "gmail.gmail_list_labels" + ], + "hit1": "gmail.gmail_get_email", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "kind": "verb_product", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "ranking": [ + "filesystem.get_file_info", + "figma.figma_get_file_nodes", + "notion.notion_get_block", + "jira.list_boards", + "gmail.gmail_list_labels", + "elasticsearch.es_index_doc", + "gmail.gmail_get_email", + "notion.notion_append_block", + "gmail.gmail_send_email", + "slack.list_bookmarks" + ], + "hit1": "filesystem.get_file_info", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 251, + "k3_calls": 84, + "k3_tokens": 19977, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 51, + "k5_tokens": 19522, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 26, + "k10_tokens": 19419, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "kind": "paraphrase", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_search", + "google_drive.gdrive_get_file", + "google_drive.gdrive_share_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_list_folder", + "google_drive.gdrive_update_file", + "docker.docker_images", + "postgres.describe_table", + "slack.delete_reminder", + "redis.redis_get" + ], + "hit1": "google_drive.gdrive_search", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 496, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 383, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 671, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_create_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_update_event", + "google_calendar.gcal_delete_event", + "sentry.sentry_list_events", + "github.create_environment", + "kubernetes.k8s_get_events", + "github.get_environment", + "kubernetes.k8s_scale_deployment", + "github.list_environments" + ], + "hit1": "google_calendar.gcal_create_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 317, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 461, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 834, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_list_pods", + "stripe.stripe_create_product", + "kubernetes.k8s_pod_logs", + "memory.open_nodes", + "kubernetes.k8s_get_pod", + "github.projects_get", + "shopify.shopify_list_products", + "jira.list_boards", + "github.projects_list", + "shopify.shopify_list_orders" + ], + "hit1": "kubernetes.k8s_list_pods", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "ranking": [ + "shopify.shopify_get_product", + "shopify.shopify_create_product", + "stripe.stripe_create_product", + "shopify.shopify_list_products", + "shopify.shopify_update_product", + "shopify.shopify_create_order", + "kubernetes.k8s_get_pod", + "github.fork_repository", + "kubernetes.k8s_pod_logs", + "figma.figma_post_comment" + ], + "hit1": "shopify.shopify_get_product", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.1111111111111111, + "ndcg@10": 0.3010299956639812, + "hit_rank": 9, + "k3_calls": 3, + "k3_tokens": 758, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 807, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 783, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "ranking": [ + "time.convert_time", + "mongodb.mongo_insert", + "notion.notion_search", + "slack.create_reminder", + "okta.okta_list_users", + "github.actions_run_trigger", + "notion.notion_list_users", + "confluence.confluence_search", + "github.get_code_scanning_alert", + "github.list_code_scanning_alerts" + ], + "hit1": "time.convert_time", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 189, + "k3_calls": 63, + "k3_tokens": 15143, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 38, + "k5_tokens": 14689, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 19, + "k10_tokens": 14309, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "kind": "verb_product", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "ranking": [ + "github.actions_get_job_logs", + "docker.docker_logs", + "kubernetes.k8s_pod_logs", + "git.git_log", + "github.list_gists", + "datadog.datadog_list_logs", + "github.list_tags", + "gitlab.list_jobs", + "github.search_orgs", + "github.list_hooks" + ], + "hit1": "github.actions_get_job_logs", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 274, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 387, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "kind": "ambiguous", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "ranking": [ + "linear.get_user", + "gitlab.get_release", + "linear.list_users", + "gitlab.update_release", + "linear.create_user", + "linear.delete_user", + "linear.update_user", + "postgres.query", + "filesystem.read_file", + "okta.okta_get_user" + ], + "hit1": "linear.get_user", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 174, + "k3_calls": 58, + "k3_tokens": 14269, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 35, + "k5_tokens": 13802, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 18, + "k10_tokens": 13742, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "kind": "ambiguous", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "ranking": [ + "github.get_package", + "github.create_package", + "github.delete_package", + "notion.notion_get_page", + "github.update_package", + "github.list_packages", + "notion.notion_create_page", + "notion.notion_update_page", + "confluence.confluence_get_page", + "confluence.confluence_create_page" + ], + "hit1": "github.get_package", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 178, + "k3_calls": 60, + "k3_tokens": 14674, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 36, + "k5_tokens": 14146, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 18, + "k10_tokens": 13750, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "ranking": [ + "github.merge_pull_request", + "github.pull_request_read", + "gitlab.gitlab_get_merge_request", + "github.create_pull_request", + "github.list_pull_requests", + "gitlab.gitlab_list_merge_requests", + "github.update_pull_request", + "github.search_pull_requests", + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_create_merge_request" + ], + "hit1": "github.merge_pull_request", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.1, + "ndcg@10": 0.2890648263178879, + "hit_rank": 10, + "k3_calls": 4, + "k3_tokens": 1156, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 2, + "k5_tokens": 1020, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 998, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "kind": "verb_product", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "ranking": [ + "notion.notion_get_page", + "github.list_packages", + "github.get_package", + "notion.notion_create_page", + "linear.get_user", + "notion.notion_update_page", + "asana.asana_get_task", + "notion.notion_get_block", + "github.get_gist", + "notion.notion_search" + ], + "hit1": "notion.notion_get_page", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.1, + "ndcg@10": 0.2890648263178879, + "hit_rank": 10, + "k3_calls": 4, + "k3_tokens": 782, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 2, + "k5_tokens": 650, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 630, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "kind": "verb_product", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "ranking": [ + "notion.notion_get_page", + "notion.notion_create_page", + "notion.notion_update_page", + "github.actions_get", + "confluence.confluence_get_page", + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "github.get_package", + "github.list_packages", + "github.create_package" + ], + "hit1": "notion.notion_get_page", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "kind": "verb_product", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "ranking": [ + "notion.notion_query_database", + "notion.notion_create_database", + "slack.update_canvase", + "notion.notion_update_page", + "gitlab.update_label", + "notion.notion_search", + "linear.update_label", + "notion.notion_create_page", + "notion.get_data_source", + "notion.notion_get_page" + ], + "hit1": "notion.notion_query_database", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 266, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_update_ticket", + "clickup.clickup_get_task", + "zendesk.zendesk_search", + "aws_s3.s3_list_buckets", + "pagerduty.pagerduty_resolve_incident", + "docker.docker_stop", + "pagerduty.pagerduty_acknowledge_incident" + ], + "hit1": "zendesk.zendesk_get_ticket", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 365, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_get_contact", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_create_contact", + "hubspot.hubspot_search_contacts", + "salesforce.salesforce_convert_lead", + "time.convert_time", + "figma.figma_post_comment", + "okta.okta_create_user", + "hubspot.hubspot_create_deal", + "github.create_artifact" + ], + "hit1": "hubspot.hubspot_get_contact", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 722, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "kind": "verb_product", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_create_lead", + "salesforce.salesforce_convert_lead", + "jira.delete_board", + "filesystem.read_file", + "linear.linear_update_issue", + "salesforce.salesforce_query", + "github.issue_read", + "salesforce.salesforce_create_record", + "memory.read_graph", + "salesforce.salesforce_delete_record" + ], + "hit1": "salesforce.salesforce_create_lead", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 741, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "kind": "identifier", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "ranking": [ + "mysql.mysql_query", + "prometheus.prom_query", + "sqlite.sqlite_query", + "postgres.explain_query", + "salesforce.salesforce_query", + "prometheus.prom_query_range", + "datadog.datadog_query_metrics", + "notion.notion_query_database", + "github.pull_request_read", + "postgres.query" + ], + "hit1": "mysql.mysql_query", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.2, + "ndcg@10": 0.38685280723454163, + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 413, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 293, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "kind": "verb_product", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "ranking": [ + "shopify.shopify_list_orders", + "shopify.shopify_get_order", + "shopify.shopify_create_order", + "docker.docker_ps", + "slack.list_reminders", + "docker.docker_stop", + "shopify.shopify_list_customers", + "shopify.shopify_list_products", + "memory.open_nodes", + "shopify.shopify_create_product" + ], + "hit1": "shopify.shopify_list_orders", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 210, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 644, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_acknowledge_incident", + "github.get_notification_details", + "github.get_discussion_comments", + "linear.list_comments", + "github.update_issue_comment", + "notion.list_comments" + ], + "hit1": "pagerduty.pagerduty_get_incident", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 802, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "kind": "paraphrase", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_list_oncalls", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_acknowledge_incident", + "filesystem.create_directory", + "notion.update_users_token", + "shopify.shopify_get_product", + "linear.update_cycle" + ], + "hit1": "pagerduty.pagerduty_get_incident", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "kind": "verb_product", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "ranking": [ + "datadog.datadog_query_metrics", + "prometheus.prom_list_metrics", + "datadog.datadog_list_monitors", + "github.get_artifact", + "datadog.datadog_get_monitor", + "trello.trello_list_cards", + "datadog.datadog_list_logs", + "github.get_discussion", + "datadog.datadog_create_monitor", + "github.list_artifacts" + ], + "hit1": "datadog.datadog_query_metrics", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 347, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 801, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "kind": "verb_product", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "ranking": [ + "datadog.datadog_list_logs", + "github.actions_get_job_logs", + "docker.docker_logs", + "kubernetes.k8s_pod_logs", + "datadog.datadog_list_monitors", + "git.git_log", + "datadog.datadog_get_monitor", + "github.list_gists", + "datadog.datadog_query_metrics", + "github.list_tags" + ], + "hit1": "datadog.datadog_list_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 276, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 780, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_list_buckets", + "zendesk.zendesk_list_tickets", + "bitbucket.bitbucket_list_prs", + "linear.list_users", + "bitbucket.bitbucket_list_repos", + "linear.list_comments", + "bitbucket.bitbucket_create_issue", + "notion.list_comments", + "bitbucket.bitbucket_create_pr", + "github.list_branches" + ], + "hit1": "aws_s3.s3_list_buckets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 191, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 752, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "kind": "paraphrase", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_get_object", + "aws_s3.s3_put_object", + "gitlab.get_job", + "github.projects_get", + "git.git_log", + "github.projects_list", + "github.get_hook", + "github.projects_write", + "git.git_show", + "jira.get_board" + ], + "hit1": "aws_s3.s3_get_object", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 676, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_purge_cache", + "github.create_check_run", + "jira.create_attachment", + "linear.create_attachment", + "linear.create_cycle", + "slack.slack_create_channel", + "linear.create_comment", + "notion.create_comment", + "github.search_code", + "slack.create_canvase" + ], + "hit1": "cloudflare.cf_purge_cache", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 287, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 477, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 917, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_get_record", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_create_record", + "filesystem.list_directory", + "salesforce.salesforce_update_record", + "filesystem.directory_tree", + "filesystem.create_directory", + "shopify.shopify_create_order", + "salesforce.salesforce_convert_lead", + "jira.create_board" + ], + "hit1": "salesforce.salesforce_get_record", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 301, + "k3_calls": 101, + "k3_tokens": 24119, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 61, + "k5_tokens": 23357, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 31, + "k10_tokens": 23011, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_deployments", + "kubernetes.k8s_list_deployments", + "vercel.vercel_get_deployment", + "kubernetes.k8s_scale_deployment", + "linear.delete_comment", + "vercel.vercel_list_projects", + "notion.delete_comment", + "vercel.vercel_create_project", + "linear.list_comments", + "salesforce.salesforce_list_opportunities" + ], + "hit1": "vercel.vercel_list_deployments", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 618, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "ranking": [ + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update", + "supabase.supabase_invoke_fn", + "supabase.supabase_list_tables", + "sentry.sentry_update_issue", + "notion.notion_query_database", + "notion.notion_update_page", + "slack.update_canvase", + "gitlab.update_release" + ], + "hit1": "supabase.supabase_select", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 739, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "kind": "paraphrase", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "ranking": [ + "figma.figma_get_images", + "playwright.browser_navigate", + "figma.figma_get_file", + "prometheus.prom_list_metrics", + "figma.figma_get_comments", + "playwright.browser_fill_form", + "figma.figma_post_comment", + "figma.figma_get_file_nodes", + "gmail.gmail_get_email", + "slack.get_bookmark" + ], + "hit1": "figma.figma_get_images", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 627, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "kind": "verb_product", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "ranking": [ + "twilio.twilio_send_sms", + "github.list_notifications", + "twilio.twilio_make_call", + "gitlab.gitlab_create_pipeline", + "twilio.twilio_list_messages", + "filesystem.list_directory_with_sizes", + "gitlab.get_wiki", + "filesystem.get_file_info", + "gitlab.list_wikis", + "gitlab.create_wiki" + ], + "hit1": "twilio.twilio_send_sms", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 739, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "kind": "verb_product", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "ranking": [ + "linear.get_user", + "linear.list_users", + "okta.okta_get_user", + "linear.create_user", + "linear.delete_user", + "linear.update_user", + "okta.okta_list_users", + "okta.okta_create_user", + "github.search_users", + "slack.get_usergroup" + ], + "hit1": "linear.get_user", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.125, + "ndcg@10": 0.31546487678572877, + "hit_rank": 8, + "k3_calls": 3, + "k3_tokens": 605, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 640, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 619, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_projects", + "linear.create_user", + "datadog.datadog_query_metrics", + "github.projects_get", + "prometheus.prom_list_metrics", + "github.projects_list", + "github.projects_write", + "okta.okta_create_user", + "slack.create_usergroup", + "prometheus.prom_query_range" + ], + "hit1": "sentry.sentry_list_projects", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 327, + "k3_calls": 109, + "k3_tokens": 25677, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 66, + "k5_tokens": 25014, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 33, + "k10_tokens": 24354, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "kind": "verb_product", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "ranking": [ + "grafana.grafana_get_dashboard", + "grafana.grafana_search_dashboards", + "jira.list_boards", + "grafana.grafana_create_annotation", + "jira.update_board", + "jira.delete_board", + "jira.create_board", + "slack.list_bookmarks", + "slack.update_bookmark", + "github.list_dependabot_alerts" + ], + "hit1": "grafana.grafana_get_dashboard", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 209, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 359, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 732, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "kind": "verb_product", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "ranking": [ + "gitlab.list_pipelines", + "gitlab.get_pipeline", + "gitlab.create_pipeline", + "gitlab.delete_pipeline", + "circleci.circleci_get_pipeline", + "gitlab.update_pipeline", + "circleci.circleci_list_pipelines", + "gitlab.gitlab_create_pipeline", + "circleci.circleci_trigger_pipeline", + "stripe.stripe_list_invoices" + ], + "hit1": "gitlab.list_pipelines", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.1111111111111111, + "ndcg@10": 0.3010299956639812, + "hit_rank": 9, + "k3_calls": 3, + "k3_tokens": 667, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 695, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 674, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "kind": "identifier", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "ranking": [ + "terraform.tf_plan", + "postgres.explain_query", + "slack.list_canvases", + "github.list_branches", + "gitlab.update_label", + "linear.update_label", + "gitlab.get_pipeline", + "jira.list_attachments", + "linear.list_attachments", + "slack.update_pin" + ], + "hit1": "terraform.tf_plan", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "kind": "verb_product", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "ranking": [ + "asana.asana_get_task", + "asana.asana_list_tasks", + "asana.asana_create_task", + "asana.asana_update_task", + "clickup.clickup_list_tasks", + "clickup.clickup_create_task", + "clickup.clickup_update_task", + "clickup.clickup_get_task", + "grafana.grafana_get_dashboard", + "asana.asana_create_project" + ], + "hit1": "asana.asana_get_task", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 387, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 747, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "ranking": [ + "trello.trello_list_cards", + "trello.trello_create_card", + "trello.trello_create_board", + "stripe.stripe_create_refund", + "stripe.stripe_create_product", + "jira.create_board", + "slack.create_reminder", + "trello.trello_add_comment", + "jira.get_board", + "github.create_runner" + ], + "hit1": "trello.trello_list_cards", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_get_task", + "asana.asana_update_task", + "asana.asana_get_task", + "asana.asana_list_tasks", + "clickup.clickup_list_tasks", + "clickup.clickup_update_task", + "clickup.clickup_create_task", + "asana.asana_create_task", + "notion.update_data_source", + "github.list_tags" + ], + "hit1": "clickup.clickup_get_task", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.14285714285714285, + "ndcg@10": 0.3333333333333333, + "hit_rank": 7, + "k3_calls": 3, + "k3_tokens": 736, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 785, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "kind": "verb_product", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "ranking": [ + "confluence.confluence_get_page", + "notion.notion_create_page", + "notion.notion_update_page", + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "github.get_package", + "github.list_packages", + "notion.notion_get_page", + "github.create_package", + "confluence.confluence_search" + ], + "hit1": "confluence.confluence_get_page", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 488, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "kind": "verb_product", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "ranking": [ + "slack.slack_post_message", + "discord.discord_send_message", + "slack.slack_search_messages", + "twilio.twilio_list_messages", + "github.list_code_scanning_alerts", + "github.list_secret_scanning_alerts", + "discord.discord_list_members", + "discord.discord_get_channel", + "github.get_secret_scanning_alert", + "mysql.mysql_list_tables" + ], + "hit1": "slack.slack_post_message", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 376, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_search", + "github.search_repositories", + "notion.notion_search", + "brave_search.brave_local_search", + "zendesk.zendesk_search", + "slack.slack_search_messages", + "confluence.confluence_search", + "linear.linear_search_issues", + "gmail.gmail_search_emails", + "stripe.stripe_create_price" + ], + "hit1": "elasticsearch.es_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "ranking": [ + "redis.redis_keys", + "playwright.browser_press_key", + "postgres.query", + "redis.redis_get", + "redis.redis_set", + "kubernetes.k8s_delete", + "redis.redis_delete", + "kubernetes.k8s_apply", + "kubernetes.k8s_list_deployments", + "filesystem.read_media_file" + ], + "hit1": "redis.redis_keys", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 397, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 289, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 602, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "ranking": [ + "github.get_discussion_comments", + "kubernetes.k8s_list_deployments", + "datadog.datadog_query_metrics", + "vercel.vercel_list_deployments", + "linear.list_comments", + "notion.list_comments", + "docker.docker_images", + "linear.delete_comment", + "notion.delete_comment", + "notion.notion_get_comments" + ], + "hit1": "github.get_discussion_comments", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 286, + "k3_calls": 96, + "k3_tokens": 22835, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 58, + "k5_tokens": 22198, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 29, + "k10_tokens": 21618, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "kind": "verb_product", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "ranking": [ + "postgres.list_tables", + "mysql.mysql_list_tables", + "sqlite.sqlite_list_tables", + "github.list_labels", + "supabase.supabase_list_tables", + "gitlab.list_labels", + "linear.list_labels", + "postgres.describe_table", + "slack.list_files", + "github.list_dependabot_alerts" + ], + "hit1": "postgres.list_tables", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 260, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 626, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "kind": "verb_product", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "ranking": [ + "postgres.describe_table", + "sqlite.sqlite_list_tables", + "supabase.supabase_list_tables", + "playwright.browser_tab_close", + "sqlite.sqlite_describe_table", + "postgres.list_tables", + "mysql.mysql_list_tables", + "github.get_label", + "gitlab.get_label", + "notion.notion_create_database" + ], + "hit1": "postgres.describe_table", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 160, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 258, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 577, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "jira.jira_create_issue", + "gitlab.gitlab_create_issue", + "jira.jira_update_issue", + "github.issue_read", + "github.issue_write", + "github.list_issues", + "linear.linear_create_issue", + "github.search_issues", + "linear.create_user" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8955796649110442, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 357, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 522, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1073, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "ranking": [ + "github.create_issue", + "jira.jira_create_issue", + "gitlab.gitlab_create_issue", + "jira.jira_update_issue", + "github.issue_read", + "github.issue_write", + "github.list_issues", + "linear.linear_create_issue", + "github.search_issues", + "linear.create_user" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9410605993303517, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 356, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 521, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1072, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "kind": "identifier", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "ranking": [ + "filesystem.list_directory", + "filesystem.directory_tree", + "filesystem.list_directory_with_sizes", + "filesystem.create_directory", + "filesystem.list_allowed_directories", + "github.star_repository", + "github.delete_repository", + "github.unstar_repository", + "github.list_repository_collaborators", + "stripe.stripe_list_customers" + ], + "hit1": "filesystem.list_directory", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 629, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "kind": "paraphrase", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "ranking": [ + "git.git_checkout" + ], + "hit1": "git.git_checkout", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 116, + "k3_calls": 39, + "k3_tokens": 9799, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 24, + "k5_tokens": 9727, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9475, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "kind": "paraphrase", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "ranking": [ + "notion.list_users_tokens", + "filesystem.list_directory", + "github.get_hook", + "notion.notion_get_block", + "github.list_hooks", + "github.actions_get_job_logs", + "notion.get_users_token", + "github.create_hook", + "github.delete_hook", + "notion.notion_get_comments" + ], + "hit1": "notion.list_users_tokens", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 125, + "k3_calls": 42, + "k3_tokens": 10496, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 25, + "k5_tokens": 10044, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 13, + "k10_tokens": 10182, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "kind": "verb_product", + "gold": [ + "time.convert_time" + ], + "alts": [], + "ranking": [ + "github.list_milestones", + "gitlab.list_milestones", + "cloudflare.cf_list_zones", + "github.get_milestone", + "gitlab.get_milestone", + "github.get_file_contents", + "gitlab.list_pipelines", + "github.create_milestone", + "gitlab.create_milestone", + "github.delete_milestone" + ], + "hit1": "github.list_milestones", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 127, + "k3_calls": 43, + "k3_tokens": 10789, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 26, + "k5_tokens": 10502, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 13, + "k10_tokens": 10229, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "kind": "verb_product", + "gold": [ + "github.star_repository" + ], + "alts": [], + "ranking": [ + "github.unstar_repository", + "github.fork_repository", + "github.star_repository", + "github.search_repositories", + "github.create_repository", + "bitbucket.bitbucket_list_repos", + "github.delete_repository", + "github.get_repository_tree", + "slack.slack_reply_to_thread", + "playwright.browser_snapshot" + ], + "hit1": "github.unstar_repository", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 241, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 766, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "kind": "resource", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "ranking": [ + "github.fork_repository", + "github.star_repository", + "github.create_repository", + "github.unstar_repository", + "github.delete_repository", + "github.get_repository_tree", + "github.search_repositories", + "github.list_repository_collaborators", + "github.list_starred_repositories", + "datadog.datadog_create_monitor" + ], + "hit1": "github.fork_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 248, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 831, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "kind": "verb_product", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "ranking": [ + "gitlab.get_release", + "github.list_releases", + "gitlab.update_release", + "gitlab.list_releases", + "gitlab.create_release", + "gitlab.delete_release", + "github.get_latest_release", + "github.get_release_by_tag", + "brave_search.brave_local_search", + "slack.delete_canvase" + ], + "hit1": "gitlab.get_release", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.14285714285714285, + "ndcg@10": 0.3333333333333333, + "hit_rank": 7, + "k3_calls": 3, + "k3_tokens": 731, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 767, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 745, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "kind": "resource", + "gold": [ + "github.create_gist" + ], + "alts": [], + "ranking": [ + "github.create_gist", + "github.get_gist", + "github.update_gist", + "git.git_status", + "github.create_issue", + "git.git_reset", + "jira.create_sprint", + "github.list_gists", + "slack.create_pin", + "github.create_milestone" + ], + "hit1": "github.create_gist", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 413, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 814, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "kind": "verb_product", + "gold": [ + "github.search_code" + ], + "alts": [], + "ranking": [ + "github.search_code", + "memory.search_nodes", + "shopify.shopify_create_order", + "github.get_code_scanning_alert", + "github.list_code_scanning_alerts", + "hubspot.hubspot_create_deal", + "memory.open_nodes", + "hubspot.hubspot_get_contact", + "linear.update_comment", + "time.convert_time" + ], + "hit1": "github.search_code", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "kind": "verb_product", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "ranking": [ + "github.list_notifications", + "github.dismiss_notification", + "github.get_notification_details", + "github.mark_all_notifications_read", + "notion.notion_get_comments", + "mongodb.mongo_list_collections", + "notion.notion_create_database", + "stripe.stripe_list_subscriptions", + "grafana.grafana_create_annotation", + "shopify.shopify_list_customers" + ], + "hit1": "github.list_notifications", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 283, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 620, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "kind": "verb_product", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "ranking": [ + "github.create_issue", + "github.list_issues", + "github.search_issues", + "jira.jira_get_issue", + "github.issue_read", + "jira.jira_assign_issue", + "github.issue_write", + "github.add_issue_comment", + "linear.linear_get_issue", + "gitlab.gitlab_get_issue" + ], + "hit1": "github.create_issue", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 49, + "k3_calls": 17, + "k3_tokens": 5013, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 10, + "k5_tokens": 4762, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 5, + "k10_tokens": 4652, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "kind": "resource", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "ranking": [ + "github.list_code_scanning_alerts", + "github.list_dependabot_alerts", + "github.list_secret_scanning_alerts", + "linear.list_users", + "github.get_code_scanning_alert", + "github.list_environments", + "github.get_secret_scanning_alert", + "notion.list_users_tokens", + "google_calendar.gcal_list_calendars", + "okta.okta_list_users" + ], + "hit1": "github.list_code_scanning_alerts", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5067916889545789, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 276, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 446, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 789, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "ranking": [ + "git.git_checkout", + "github.get_check_run", + "hubspot.hubspot_create_deal", + "git.git_show", + "hubspot.hubspot_get_contact", + "git.git_branch", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_create_contact", + "git.git_status", + "hubspot.hubspot_search_contacts" + ], + "hit1": "git.git_checkout", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 27, + "k3_calls": 9, + "k3_tokens": 2203, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 6, + "k5_tokens": 2360, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 3, + "k10_tokens": 2288, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "kind": "resource", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "ranking": [ + "github.fork_repository", + "github.star_repository", + "github.unstar_repository", + "github.create_repository", + "github.delete_repository", + "github.get_repository_tree", + "asana.asana_create_project", + "github.search_repositories", + "bitbucket.bitbucket_list_repos", + "linear.linear_create_project" + ], + "hit1": "github.fork_repository", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 31, + "k3_calls": 11, + "k3_tokens": 3009, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 7, + "k5_tokens": 3110, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3597, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "kind": "verb_product", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "ranking": [ + "slack.slack_reply_to_thread", + "slack.slack_get_thread_replies", + "filesystem.read_file", + "memory.read_graph", + "slack.slack_add_reaction", + "github.issue_read", + "slack.slack_post_message", + "trello.trello_add_comment", + "slack.slack_list_channels", + "slack.slack_create_channel" + ], + "hit1": "slack.slack_reply_to_thread", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 747, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "kind": "verb_product", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "ranking": [ + "slack.slack_get_users", + "slack.slack_list_users", + "slack.slack_upload_file", + "slack.slack_add_reaction", + "slack.slack_join_channel", + "slack.slack_post_message", + "slack.slack_leave_channel", + "slack.slack_list_channels", + "slack.slack_create_channel", + "slack.slack_reply_to_thread" + ], + "hit1": "slack.slack_get_users", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.3273949503887395, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 471, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "kind": "verb_product", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "ranking": [ + "slack.slack_post_message", + "slack.slack_search_messages", + "twilio.twilio_list_messages", + "slack.slack_get_users", + "discord.discord_send_message", + "slack.slack_get_user_profile", + "slack.slack_list_users", + "figma.figma_get_images", + "slack.slack_list_channels", + "mysql.mysql_list_tables" + ], + "hit1": "slack.slack_post_message", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 210, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 666, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "kind": "verb_product", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "ranking": [ + "slack.slack_join_channel", + "slack.slack_list_channels", + "slack.slack_leave_channel", + "discord.discord_get_channel", + "slack.slack_create_channel", + "slack.slack_set_channel_topic", + "discord.discord_list_channels", + "slack.slack_get_channel_history", + "slack.slack_get_thread_replies", + "github.get_secret_scanning_alert" + ], + "hit1": "slack.slack_join_channel", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.2, + "ndcg@10": 0.38685280723454163, + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 405, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 681, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "kind": "verb_product", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_get_issue", + "github.create_issue", + "github.list_issues", + "jira.jira_assign_issue", + "jira.jira_create_issue", + "jira.jira_update_issue", + "github.search_issues", + "jira.jira_search_issues", + "gitlab.gitlab_get_issue", + "linear.linear_get_issue" + ], + "hit1": "jira.jira_get_issue", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 153, + "k3_calls": 51, + "k3_tokens": 12624, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 31, + "k5_tokens": 12368, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 16, + "k10_tokens": 12436, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "kind": "identifier", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "ranking": [ + "github.actions_get_job_logs", + "mysql.mysql_query", + "filesystem.write_file", + "jira.jira_list_projects", + "gmail.gmail_trash_email", + "sqlite.sqlite_query", + "github.projects_list", + "filesystem.list_directory_with_sizes", + "mysql.mysql_list_tables", + "sqlite.sqlite_list_tables" + ], + "hit1": "github.actions_get_job_logs", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 155, + "k3_calls": 52, + "k3_tokens": 12765, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 31, + "k5_tokens": 12256, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 16, + "k10_tokens": 12345, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "kind": "verb_product", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "ranking": [ + "jira.jira_get_issue", + "trello.trello_create_card", + "jira.jira_add_comment", + "trello.trello_list_cards", + "jira.jira_add_worklog", + "trello.trello_create_board", + "jira.jira_assign_issue", + "slack.slack_reply_to_thread", + "jira.jira_create_issue", + "notion.notion_create_page" + ], + "hit1": "jira.jira_get_issue", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.2, + "ndcg@10": 0.38685280723454163, + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 478, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 814, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "kind": "verb_product", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "ranking": [ + "linear.linear_list_teams", + "github.get_teams", + "github.get_team_members", + "linear.linear_list_issues", + "linear.linear_create_issue", + "gitlab.get_release", + "linear.linear_update_issue", + "postgres.list_schemas", + "linear.linear_get_issue", + "github.list_releases" + ], + "hit1": "linear.linear_list_teams", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 158, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 708, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "ranking": [ + "linear.linear_get_issue", + "github.list_issues", + "github.search_issues", + "github.create_issue", + "linear.linear_list_issues", + "jira.jira_get_issue", + "sentry.sentry_get_issue", + "jira.jira_search_issues", + "jira.jira_create_issue", + "gitlab.gitlab_get_issue" + ], + "hit1": "linear.linear_get_issue", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 146, + "k3_calls": 49, + "k3_tokens": 12231, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 30, + "k5_tokens": 12034, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11719, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q094", + "query": "Move a local file", + "kind": "paraphrase", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "ranking": [ + "slack.list_files", + "filesystem.move_file", + "filesystem.read_file", + "slack.create_file", + "jira.create_filter", + "github.delete_file", + "slack.delete_file", + "jira.list_filters", + "slack.update_file", + "filesystem.search_files" + ], + "hit1": "slack.list_files", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 780, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "kind": "paraphrase", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "ranking": [ + "github.update_runner", + "jira.update_filter", + "jira.update_sprint", + "slack.update_reminder", + "jira.update_version", + "jira.get_attachment", + "linear.get_attachment", + "jira.update_attachment", + "github.update_check_run", + "linear.update_attachment" + ], + "hit1": "github.update_runner", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 96, + "k3_calls": 32, + "k3_tokens": 8562, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 20, + "k5_tokens": 8492, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 10, + "k10_tokens": 8272, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "kind": "paraphrase", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "ranking": [ + "jira.get_sprint", + "github.actions_run_trigger", + "stripe.stripe_list_invoices", + "jira.create_sprint", + "jira.delete_sprint", + "jira.update_sprint", + "jira.list_sprints", + "slack.list_reminders", + "stripe.stripe_get_customer", + "jira.list_versions" + ], + "hit1": "jira.get_sprint", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 89, + "k3_calls": 30, + "k3_tokens": 8310, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7998, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7764, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "ranking": [ + "memory.read_graph", + "pagerduty.pagerduty_acknowledge_incident", + "git.git_branch", + "google_drive.gdrive_search", + "gitlab.get_release", + "gmail.gmail_trash_email", + "playwright.browser_snapshot", + "git.git_create_branch", + "slack.create_pin", + "gitlab.gitlab_create_pipeline" + ], + "hit1": "memory.read_graph", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 627, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "kind": "verb_product", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "figma.figma_get_file_nodes", + "memory.open_nodes", + "pagerduty.pagerduty_list_incidents", + "mongodb.mongo_delete", + "pagerduty.pagerduty_acknowledge_incident", + "zendesk.zendesk_search", + "docker.docker_stop", + "docker.docker_ps", + "slack.list_reminders" + ], + "hit1": "memory.search_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 652, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "kind": "verb_product", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "ranking": [ + "git.git_diff", + "git.git_diff_staged", + "git.git_diff_unstaged", + "gmail.gmail_modify_email", + "google_drive.gdrive_get_file", + "google_drive.gdrive_list_folder", + "google_drive.gdrive_create_file", + "filesystem.edit_file", + "git.git_add", + "filesystem.read_media_file" + ], + "hit1": "git.git_diff", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 719, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "kind": "identifier", + "gold": [ + "git.git_add" + ], + "alts": [], + "ranking": [ + "git.git_add", + "jira.jira_add_comment", + "jira.jira_add_worklog", + "trello.trello_add_comment", + "filesystem.list_allowed_directories", + "memory.add_observations", + "git.git_diff", + "github.add_issue_comment", + "git.git_branch", + "git.git_status" + ], + "hit1": "git.git_add", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 678, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "kind": "verb_product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "ranking": [ + "shopify.shopify_get_product", + "shopify.shopify_list_products", + "shopify.shopify_create_product", + "shopify.shopify_update_product", + "stripe.stripe_create_product", + "shopify.shopify_create_order", + "github.projects_get", + "shopify.shopify_get_order", + "github.projects_list", + "shopify.shopify_list_orders" + ], + "hit1": "shopify.shopify_get_product", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "kind": "verb_product", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "ranking": [ + "stripe.stripe_list_subscriptions", + "stripe.stripe_cancel_subscription", + "hubspot.hubspot_list_contacts", + "stripe.stripe_create_invoice", + "hubspot.hubspot_create_contact", + "stripe.stripe_list_customers", + "stripe.stripe_create_payment_intent" + ], + "hit1": "stripe.stripe_list_subscriptions", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 187, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 631, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "kind": "resource", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "ranking": [ + "stripe.stripe_get_payment_intent", + "github.get_file_contents", + "stripe.stripe_create_payment_intent", + "sentry.sentry_list_events", + "github.update_environment", + "notion.notion_get_comments", + "pagerduty.pagerduty_get_incident", + "linear.list_comments", + "pagerduty.pagerduty_create_incident", + "notion.list_comments" + ], + "hit1": "stripe.stripe_get_payment_intent", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 272, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 780, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_acknowledge_incident", + "github.get_notification_details", + "github.get_discussion_comments", + "linear.list_comments", + "github.update_issue_comment", + "notion.list_comments" + ], + "hit1": "pagerduty.pagerduty_get_incident", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.2, + "ndcg@10": 0.38685280723454163, + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 481, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 804, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "kind": "verb_product", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "ranking": [ + "datadog.datadog_get_monitor", + "datadog.datadog_list_monitors", + "datadog.datadog_create_monitor", + "github.list_starred_repositories", + "mongodb.mongo_insert", + "datadog.datadog_list_logs", + "mongodb.mongo_list_collections", + "datadog.datadog_query_metrics", + "github.fork_repository", + "github.star_repository" + ], + "hit1": "datadog.datadog_get_monitor", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 786, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "kind": "paraphrase", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "ranking": [ + "github.list_issues", + "github.search_issues", + "sentry.sentry_get_issue", + "github.create_issue", + "jira.jira_get_issue", + "sentry.sentry_list_issues", + "gitlab.gitlab_get_issue", + "linear.linear_get_issue", + "jira.jira_search_issues", + "jira.jira_assign_issue" + ], + "hit1": "github.list_issues", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 232, + "k3_calls": 78, + "k3_tokens": 18579, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 47, + "k5_tokens": 17991, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 24, + "k10_tokens": 17923, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "ranking": [ + "vercel.vercel_get_deployment", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_scale_deployment", + "vercel.vercel_list_deployments", + "linear.delete_comment", + "notion.delete_comment", + "github.delete_environment", + "linear.update_comment", + "notion.update_comment", + "linear.list_comments" + ], + "hit1": "vercel.vercel_get_deployment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 652, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert", + "github.actions_list", + "github.get_milestone", + "gitlab.get_milestone", + "asana.asana_list_tasks", + "github.list_milestones", + "gitlab.list_milestones", + "github.create_milestone", + "gitlab.create_milestone", + "github.delete_milestone" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 203, + "k3_calls": 68, + "k3_tokens": 16305, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 41, + "k5_tokens": 15821, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 21, + "k10_tokens": 15741, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "ranking": [ + "time.convert_time", + "mongodb.mongo_insert", + "notion.notion_search", + "slack.create_reminder", + "okta.okta_list_users", + "github.actions_run_trigger", + "notion.notion_list_users", + "confluence.confluence_search", + "github.get_code_scanning_alert", + "github.list_code_scanning_alerts" + ], + "hit1": "time.convert_time", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 190, + "k3_calls": 64, + "k3_tokens": 15302, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 38, + "k5_tokens": 14689, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 19, + "k10_tokens": 14309, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "kind": "verb_product", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "ranking": [ + "docker.docker_images", + "discord.discord_send_message", + "figma.figma_get_file", + "docker.docker_rm", + "figma.figma_get_images", + "docker.docker_exec", + "figma.figma_get_comments", + "docker.docker_logs", + "figma.figma_get_file_nodes", + "docker.docker_build" + ], + "hit1": "docker.docker_images", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.1, + "ndcg@10": 0.2890648263178879, + "hit_rank": 10, + "k3_calls": 4, + "k3_tokens": 797, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 2, + "k5_tokens": 665, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "kind": "verb_product", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "ranking": [ + "github.label_write", + "github.pull_request_review_write", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_acknowledge_incident" + ], + "hit1": "github.label_write", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 171, + "k3_calls": 57, + "k3_tokens": 14078, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 35, + "k5_tokens": 13845, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 18, + "k10_tokens": 13724, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "kind": "paraphrase", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "ranking": [ + "gitlab.list_snippets", + "playwright.browser_type", + "aws_s3.s3_put_object", + "playwright.browser_snapshot", + "clickup.clickup_update_task", + "playwright.browser_navigate", + "notion.notion_update_page", + "playwright.browser_wait_for", + "jira.get_sprint", + "playwright.browser_click" + ], + "hit1": "gitlab.list_snippets", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.4796249331362629, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 266, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "ranking": [ + "github.projects_get", + "gitlab.gitlab_list_projects", + "github.projects_list", + "linear.linear_list_projects", + "jira.jira_list_projects", + "github.projects_write", + "sentry.sentry_list_projects", + "vercel.vercel_list_projects", + "linear.linear_create_project", + "asana.asana_create_project" + ], + "hit1": "github.projects_get", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 284, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 610, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "kind": "verb_product", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "ranking": [ + "github.pull_request_read", + "github.list_pull_requests", + "github.merge_pull_request", + "github.create_pull_request", + "github.update_pull_request", + "github.search_pull_requests", + "github.pull_request_review_write", + "gitlab.gitlab_get_merge_request", + "gitlab.gitlab_list_merge_requests", + "github.update_pull_request_branch" + ], + "hit1": "github.pull_request_read", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 272, + "k3_calls": 91, + "k3_tokens": 21988, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 55, + "k5_tokens": 21340, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 28, + "k10_tokens": 20983, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "kind": "verb_product", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "ranking": [ + "gmail.gmail_create_draft", + "filesystem.read_file", + "gmail.gmail_trash_email", + "jira.create_filter", + "gmail.gmail_send_email", + "jira.update_filter", + "gmail.gmail_modify_email", + "memory.delete_relations", + "gmail.gmail_get_email", + "github.delete_artifact" + ], + "hit1": "gmail.gmail_create_draft", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "kind": "verb_product", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "ranking": [ + "filesystem.read_file", + "filesystem.move_file", + "filesystem.write_file", + "filesystem.edit_file", + "slack.get_file", + "github.delete_file", + "slack.create_file", + "slack.delete_file", + "slack.list_files", + "jira.get_filter" + ], + "hit1": "filesystem.read_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 261, + "k3_calls": 87, + "k3_tokens": 20738, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 53, + "k5_tokens": 20393, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 27, + "k10_tokens": 20238, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "kind": "verb_product", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_list_events", + "google_calendar.gcal_create_event", + "sentry.sentry_list_events", + "google_calendar.gcal_update_event", + "google_calendar.gcal_delete_event", + "github.create_environment", + "kubernetes.k8s_get_events", + "github.list_environments", + "google_calendar.gcal_list_calendars", + "memory.create_entities" + ], + "hit1": "google_calendar.gcal_list_events", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 301, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 461, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 771, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_create_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_update_event", + "google_calendar.gcal_delete_event", + "sentry.sentry_list_events", + "github.create_environment", + "kubernetes.k8s_get_events", + "github.get_environment", + "kubernetes.k8s_scale_deployment", + "github.list_environments" + ], + "hit1": "google_calendar.gcal_create_event", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 576, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 461, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 834, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_projects", + "linear.linear_list_projects", + "jira.jira_list_projects", + "sentry.sentry_list_projects", + "gitlab.gitlab_list_projects", + "vercel.vercel_create_project", + "linear.linear_create_project", + "asana.asana_create_project", + "github.projects_get", + "github.projects_list" + ], + "hit1": "vercel.vercel_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 139, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 241, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 585, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "ranking": [ + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update", + "supabase.supabase_invoke_fn", + "supabase.supabase_list_tables", + "slack.update_canvase", + "notion.notion_query_database", + "gitlab.update_release", + "linear.update_user", + "jira.update_board" + ], + "hit1": "supabase.supabase_select", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 711, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "kind": "verb_product", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "ranking": [ + "linear.get_comment", + "notion.get_comment", + "linear.list_comments", + "notion.list_comments", + "figma.figma_get_comments", + "linear.create_comment", + "notion.create_comment", + "figma.figma_post_comment", + "jira.jira_add_comment", + "linear.update_comment" + ], + "hit1": "linear.get_comment", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.125, + "ndcg@10": 0.31546487678572877, + "hit_rank": 8, + "k3_calls": 3, + "k3_tokens": 704, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 748, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "kind": "verb_product", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "ranking": [ + "linear.list_users", + "okta.okta_list_users", + "linear.get_user", + "github.search_users", + "slack.slack_get_users", + "notion.get_users_token", + "slack.slack_list_users", + "notion.notion_list_users", + "slack.list_usergroups", + "linear.create_user" + ], + "hit1": "linear.list_users", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 646, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_projects", + "linear.create_user", + "datadog.datadog_query_metrics", + "github.projects_get", + "prometheus.prom_list_metrics", + "github.projects_list", + "github.projects_write", + "okta.okta_create_user", + "slack.create_usergroup", + "prometheus.prom_query_range" + ], + "hit1": "sentry.sentry_list_projects", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.1, + "ndcg@10": 0.2890648263178879, + "hit_rank": 10, + "k3_calls": 4, + "k3_tokens": 940, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 2, + "k5_tokens": 808, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 788, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "kind": "verb_product", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "ranking": [ + "grafana.grafana_create_annotation", + "github.mark_all_notifications_read", + "github.list_notifications", + "grafana.grafana_get_dashboard", + "github.dismiss_notification", + "memory.add_observations", + "grafana.grafana_search_dashboards", + "notion.notion_search", + "github.get_notification_details", + "memory.create_relations" + ], + "hit1": "grafana.grafana_create_annotation", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 209, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 602, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "kind": "identifier", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "ranking": [ + "playwright.browser_wait_for", + "playwright.browser_fill_form", + "trello.trello_create_board", + "linear.linear_create_comment", + "github.list_repository_collaborators", + "stripe.stripe_create_customer", + "salesforce.salesforce_create_record" + ], + "hit1": "playwright.browser_wait_for", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 334, + "k3_calls": 112, + "k3_tokens": 26006, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 67, + "k5_tokens": 25091, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 34, + "k10_tokens": 24881, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "kind": "verb_product", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "ranking": [ + "asana.asana_list_tasks", + "asana.asana_get_task", + "clickup.clickup_list_tasks", + "asana.asana_create_task", + "asana.asana_update_task", + "clickup.clickup_get_task", + "grafana.grafana_get_dashboard", + "github.list_tags", + "asana.asana_create_project", + "slack.list_canvases" + ], + "hit1": "asana.asana_list_tasks", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "ranking": [ + "trello.trello_list_cards", + "trello.trello_create_card", + "trello.trello_create_board", + "stripe.stripe_create_refund", + "stripe.stripe_create_product", + "jira.create_board", + "slack.create_reminder", + "trello.trello_add_comment", + "jira.get_board", + "github.create_runner" + ], + "hit1": "trello.trello_list_cards", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.125, + "ndcg@10": 0.31546487678572877, + "hit_rank": 8, + "k3_calls": 3, + "k3_tokens": 746, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 819, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_get_task", + "asana.asana_update_task", + "asana.asana_get_task", + "asana.asana_list_tasks", + "clickup.clickup_list_tasks", + "clickup.clickup_update_task", + "clickup.clickup_create_task", + "asana.asana_create_task", + "notion.update_data_source", + "github.list_tags" + ], + "hit1": "clickup.clickup_get_task", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.16666666666666666, + "ndcg@10": 0.3562071871080222, + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 448, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 785, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "kind": "verb_product", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "ranking": [ + "confluence.confluence_search", + "hubspot.hubspot_search_contacts", + "confluence.confluence_get_page", + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "cloudflare.cf_purge_cache", + "github.actions_run_trigger", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "github.add_comment_to_pending_review" + ], + "hit1": "confluence.confluence_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 187, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 369, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 848, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_deal", + "github.get_dependabot_alert", + "gitlab.delete_label", + "linear.delete_label", + "slack.slack_post_message", + "postgres.describe_table", + "sqlite.sqlite_describe_table", + "gitlab.delete_job", + "github.list_dependabot_alerts", + "slack.delete_pin" + ], + "hit1": "hubspot.hubspot_create_deal", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 676, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_update_ticket", + "pagerduty.pagerduty_list_incidents", + "aws_s3.s3_list_buckets", + "clickup.clickup_get_task", + "zendesk.zendesk_search", + "github.get_file_contents", + "pagerduty.pagerduty_resolve_incident" + ], + "hit1": "zendesk.zendesk_list_tickets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 220, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "kind": "verb_product", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "ranking": [ + "discord.discord_list_channels", + "slack.slack_list_channels", + "discord.discord_get_channel", + "slack.slack_join_channel", + "slack.slack_create_channel", + "slack.slack_leave_channel", + "slack.slack_get_channel_history", + "slack.slack_set_channel_topic", + "discord.discord_send_message", + "discord.discord_list_members" + ], + "hit1": "discord.discord_list_channels", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 308, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 622, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "ranking": [ + "redis.redis_keys", + "playwright.browser_press_key", + "postgres.query", + "redis.redis_get", + "redis.redis_set", + "kubernetes.k8s_delete", + "redis.redis_delete", + "kubernetes.k8s_apply", + "kubernetes.k8s_list_deployments", + "filesystem.read_media_file" + ], + "hit1": "redis.redis_keys", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.2, + "ndcg@10": 0.38685280723454163, + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 397, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 289, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 602, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "ranking": [ + "github.get_discussion_comments", + "kubernetes.k8s_list_deployments", + "datadog.datadog_query_metrics", + "vercel.vercel_list_deployments", + "linear.list_comments", + "notion.list_comments", + "docker.docker_images", + "linear.delete_comment", + "notion.delete_comment", + "notion.notion_get_comments" + ], + "hit1": "github.get_discussion_comments", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 287, + "k3_calls": 96, + "k3_tokens": 22835, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 58, + "k5_tokens": 22198, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 29, + "k10_tokens": 21618, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "kind": "verb_product", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "ranking": [ + "mysql.mysql_query", + "prometheus.prom_query", + "sqlite.sqlite_query", + "salesforce.salesforce_query", + "postgres.explain_query", + "prometheus.prom_query_range", + "datadog.datadog_query_metrics", + "notion.notion_query_database", + "github.pull_request_read", + "postgres.query" + ], + "hit1": "mysql.mysql_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 293, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "ranking": [ + "linear.linear_search_issues", + "elasticsearch.es_search", + "notion.notion_search", + "brave_search.brave_local_search", + "zendesk.zendesk_search", + "slack.slack_search_messages", + "confluence.confluence_search", + "gmail.gmail_search_emails", + "stripe.stripe_create_price", + "stripe.stripe_create_product" + ], + "hit1": "linear.linear_search_issues", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 293, + "k3_calls": 98, + "k3_tokens": 23327, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 59, + "k5_tokens": 22549, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 30, + "k10_tokens": 22277, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_get_object", + "aws_s3.s3_put_object", + "aws_s3.s3_list_objects", + "aws_s3.s3_delete_object", + "asana.asana_create_project", + "jira.jira_list_projects", + "linear.linear_list_projects", + "gitlab.gitlab_list_projects", + "sentry.sentry_list_projects", + "memory.add_observations" + ], + "hit1": "aws_s3.s3_get_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 392, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 630, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones", + "memory.open_nodes", + "zendesk.zendesk_search", + "github.list_milestones", + "gitlab.list_milestones", + "zendesk.zendesk_get_ticket", + "confluence.confluence_search", + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_update_ticket" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 164, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "ranking": [ + "stripe.stripe_list_invoices", + "stripe.stripe_create_invoice", + "supabase.supabase_invoke_fn", + "notion.notion_create_page", + "stripe.stripe_create_price", + "stripe.stripe_list_customers", + "notion.notion_get_comments", + "notion.notion_create_database", + "stripe.stripe_create_refund", + "github.list_notifications" + ], + "hit1": "stripe.stripe_list_invoices", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 361, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 701, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "kind": "verb_product", + "gold": [ + "github.actions_list" + ], + "alts": [], + "ranking": [ + "jira.jira_add_worklog", + "github.actions_get_job_logs" + ], + "hit1": "jira.jira_add_worklog", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 62, + "k3_calls": 21, + "k3_tokens": 6073, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 13, + "k5_tokens": 6075, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 7, + "k10_tokens": 6292, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "kind": "verb_product", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "ranking": [ + "docker.docker_logs", + "kubernetes.k8s_pod_logs", + "github.actions_get_job_logs", + "datadog.datadog_list_logs", + "git.git_log", + "github.list_gists", + "playwright.browser_tab_close", + "github.list_tags", + "brave_search.brave_local_search", + "gitlab.list_jobs" + ], + "hit1": "docker.docker_logs", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 276, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "kind": "verb_product", + "gold": [ + "github.create_repository" + ], + "alts": [], + "ranking": [ + "github.fork_repository", + "github.star_repository", + "github.unstar_repository", + "github.create_repository", + "github.delete_repository", + "github.get_repository_tree", + "github.search_repositories", + "github.list_repository_collaborators", + "github.list_starred_repositories", + "filesystem.create_directory" + ], + "hit1": "github.fork_repository", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 515, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 773, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "kind": "verb_product", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "ranking": [ + "git.git_branch", + "github.list_branches", + "github.create_branch", + "git.git_create_branch", + "github.update_pull_request_branch", + "google_drive.gdrive_search", + "git.git_checkout", + "brave_search.brave_web_search", + "jira.get_attachment", + "brave_search.brave_local_search" + ], + "hit1": "git.git_branch", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.14285714285714285, + "ndcg@10": 0.3333333333333333, + "hit_rank": 7, + "k3_calls": 3, + "k3_tokens": 714, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 770, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 750, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "kind": "identifier", + "gold": [ + "git.git_log" + ], + "alts": [], + "ranking": [ + "git.git_log", + "datadog.datadog_list_logs", + "github.actions_get_job_logs", + "docker.docker_logs", + "git.git_show", + "kubernetes.k8s_pod_logs", + "git.git_commit", + "github.list_gists", + "git.git_checkout", + "jira.jira_add_worklog" + ], + "hit1": "git.git_log", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 274, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 762, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "kind": "verb_product", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "ranking": [ + "postgres.list_schemas", + "gmail.gmail_search_emails", + "filesystem.search_files", + "memory.search_nodes", + "github.search_users", + "github.search_issues", + "slack.slack_search_messages", + "github.search_commits", + "github.list_branches", + "docker.docker_images" + ], + "hit1": "postgres.list_schemas", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 615, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "kind": "verb_product", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "ranking": [ + "memory.read_graph", + "pagerduty.pagerduty_acknowledge_incident", + "git.git_branch", + "google_drive.gdrive_search", + "gitlab.get_release", + "gmail.gmail_trash_email", + "playwright.browser_snapshot", + "git.git_create_branch", + "slack.create_pin", + "gitlab.gitlab_create_pipeline" + ], + "hit1": "memory.read_graph", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 106, + "k3_calls": 36, + "k3_tokens": 9236, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 22, + "k5_tokens": 9029, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 11, + "k10_tokens": 8787, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "kind": "paraphrase", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "ranking": [ + "github.create_milestone", + "gitlab.create_milestone", + "github.delete_milestone", + "gitlab.delete_milestone", + "github.update_milestone", + "gitlab.update_milestone", + "github.get_milestone", + "gitlab.get_milestone", + "gmail.gmail_get_email", + "linear.linear_list_teams" + ], + "hit1": "github.create_milestone", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 97, + "k3_calls": 33, + "k3_tokens": 8765, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 20, + "k5_tokens": 8530, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 10, + "k10_tokens": 8310, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "kind": "paraphrase", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "ranking": [ + "slack.list_files", + "filesystem.move_file", + "filesystem.read_file", + "slack.create_file", + "jira.create_filter", + "github.delete_file", + "slack.delete_file", + "jira.list_filters", + "slack.update_file", + "filesystem.search_files" + ], + "hit1": "slack.list_files", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 87, + "k3_calls": 29, + "k3_tokens": 7978, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7948, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7759, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "kind": "verb_product", + "gold": [ + "github.search_issues" + ], + "alts": [], + "ranking": [ + "github.list_issues", + "github.search_issues", + "jira.jira_search_issues", + "gitlab.gitlab_list_issues", + "github.create_issue", + "linear.linear_search_issues", + "linear.linear_list_issues", + "sentry.sentry_list_issues", + "gitlab.gitlab_get_issue", + "jira.jira_get_issue" + ], + "hit1": "github.list_issues", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 280, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 484, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 844, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "kind": "paraphrase", + "gold": [ + "github.get_me" + ], + "alts": [], + "ranking": [ + "linear.get_user", + "linear.update_user", + "linear.list_users", + "linear.create_user", + "github.search_users", + "linear.delete_user", + "slack.get_usergroup", + "okta.okta_get_user", + "slack.update_usergroup", + "notion.get_users_token" + ], + "hit1": "linear.get_user", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 11, + "k3_calls": 4, + "k3_tokens": 802, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 3, + "k5_tokens": 1045, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 1.0, + "k10_calls": 2, + "k10_tokens": 1428, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "cost": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 15.006666666666666, + "p95_calls": 85.64999999999995, + "success_within_1": 0.5333333333333333, + "success_within_2": 0.62, + "success_within_3": 0.7, + "mean_tokens": 3683.14, + "p95_tokens": 20395.54999999999, + "total_tokens": 552471, + "tokens_per_successful_query": 3683.14, + "total_tokens_successful": 552471 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 9.293333333333333, + "p95_calls": 52.099999999999966, + "success_within_1": 0.6133333333333333, + "success_within_2": 0.7266666666666667, + "success_within_3": 0.7333333333333333, + "mean_tokens": 3667.9533333333334, + "p95_tokens": 20001.049999999985, + "total_tokens": 550193, + "tokens_per_successful_query": 3667.9533333333334, + "total_tokens_successful": 550193 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 5.026666666666666, + "p95_calls": 26.549999999999983, + "success_within_1": 0.7266666666666667, + "success_within_2": 0.7333333333333333, + "success_within_3": 0.74, + "mean_tokens": 3850.9733333333334, + "p95_tokens": 19869.449999999983, + "total_tokens": 577646, + "tokens_per_successful_query": 3850.9733333333334, + "total_tokens_successful": 577646 + } + } +} diff --git a/experiments/mcp-tool-search/results/codedb_trigram_tokens.csv b/experiments/mcp-tool-search/results/codedb_trigram_tokens.csv new file mode 100644 index 00000000..9a1ea40d --- /dev/null +++ b/experiments/mcp-tool-search/results/codedb_trigram_tokens.csv @@ -0,0 +1,151 @@ +id,query,kind,hit1,recall@1,recall@3,recall@5,mrr@10,ndcg@10,hit_rank,k3_calls,k3_tokens,k3_success1,k5_calls,k5_tokens,k5_success1,k10_calls,k10_tokens,k10_success1,ranking +q001,Create a GitHub issue,verb_product,github.issue_write,1.0,1.0,1.0,1.0000,0.3801,1,1,270,1.0,1,370,1.0,1,704,1.0,github.issue_write git.git_show jira.jira_get_issue git.git_branch git.git_diff_unstaged gitlab.gitlab_get_issue git.git_status sentry.sentry_update_issue git.git_diff_staged sentry.sentry_get_issue +q002,Create a Jira ticket,verb_product,jira.jira_search_issues,0.0,0.0,0.0,0.1000,0.2891,10,4,839,0.0,2,707,0.0,1,687,1.0,jira.jira_search_issues zendesk.zendesk_create_ticket jira.jira_get_issue zendesk.zendesk_get_ticket jira.jira_list_projects zendesk.zendesk_update_ticket jira.jira_add_worklog jira.jira_add_comment id_map.json jira.jira_create_issue +q003,Send a Slack message,verb_product,slack.slack_post_message,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,381,1.0,1,839,1.0,slack.slack_post_message slack.slack_add_reaction slack.slack_search_messages discord.discord_send_message slack.slack_list_users discord.discord_add_reaction slack.slack_reply_to_thread github.delete_file figma.figma_post_comment gmail.gmail_modify_email +q004,List open pull requests,resource,github.pull_request_review_write,0.0,1.0,1.0,0.5000,0.6433,2,1,393,1.0,1,536,1.0,1,885,1.0,github.pull_request_review_write github.list_pull_requests github.update_pull_request github.search_pull_requests github.pull_request_read gitlab.gitlab_list_merge_requests github.update_pull_request_branch github.merge_pull_request id_map.json bitbucket.bitbucket_list_prs +q005,Open a pull request on GitHub,verb_product,github.pull_request_review_write,0.0,1.0,1.0,0.5000,0.6309,2,1,395,1.0,1,598,1.0,1,962,1.0,github.pull_request_review_write github.create_pull_request gitlab.gitlab_get_merge_request git.git_show github.update_pull_request gitlab.gitlab_merge_merge_request git.git_branch github.pull_request_read git.git_diff_unstaged github.update_pull_request_branch +q006,Run a SQL query against postgres,verb_product,postgres.query,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,272,1.0,1,546,1.0,postgres.query postgres.explain_query mysql.mysql_query sqlite.sqlite_query postgres.list_schemas postgres.describe_table postgres.list_tables salesforce.salesforce_query mysql.mysql_list_tables +q007,Read a local file from disk,paraphrase,brave_search.brave_local_search,0.0,0.0,0.0,0.1000,0.2197,10,4,937,0.0,2,801,0.0,1,779,1.0,brave_search.brave_local_search google_drive.gdrive_get_file google_drive.gdrive_update_file filesystem.read_media_file gitlab.gitlab_get_file figma.figma_get_file google_drive.gdrive_share_file github.delete_file mongodb.mongo_delete filesystem.read_file +q008,What files are in this folder,paraphrase,github.push_files,0.0,0.0,0.0,0.0000,0.0000,91,31,8425,0.0,19,8277,0.0,10,8326,0.0,github.push_files filesystem.search_files slack.list_files jira.update_sprint linear.list_users gitlab.get_pipeline elasticsearch.es_index_doc linear.update_label git.git_add linear.create_comment +q009,Remember this fact in the knowledge graph,paraphrase,github.create_artifact,0.0,0.0,0.0,0.0000,0.0000,99,33,8757,0.0,20,8509,0.0,10,8279,0.0,github.create_artifact github.delete_artifact github.get_artifact github.list_artifacts github.update_artifact id_map.json jira.update_sprint linear.list_users gitlab.get_pipeline elasticsearch.es_index_doc +q010,Search the web with Brave,verb_product,brave_search.brave_web_search,1.0,1.0,1.0,1.0000,1.0000,1,1,169,1.0,1,298,1.0,1,600,1.0,brave_search.brave_web_search id_map.json filesystem.list_directory_with_sizes brave_search.brave_local_search jira.delete_webhook jira.get_webhook jira.list_webhooks jira.update_webhook playwright.browser_navigate memory.delete_entities +q011,Fetch the contents of a URL,verb_product,fetch.fetch,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,370,1.0,1,671,1.0,fetch.fetch playwright.browser_navigate github.get_file_contents slack.slack_set_channel_topic playwright.browser_snapshot puppeteer.puppeteer_navigate filesystem.list_directory playwright.browser_tab_new memory.delete_entities twilio.twilio_make_call +q012,Show git status,identifier,git.git_status,1.0,1.0,1.0,1.0000,1.0000,1,1,178,1.0,1,320,1.0,1,681,1.0,git.git_status git.git_show git.git_branch sentry.sentry_update_issue git.git_diff_unstaged git.git_diff_staged github.list_check_runs gitlab.list_jobs slack.list_bookmarks jira.list_attachments +q013,Commit staged changes,paraphrase,git.git_commit,1.0,1.0,1.0,1.0000,1.0000,1,1,134,1.0,1,236,1.0,1,611,1.0,git.git_commit id_map.json git.git_diff_staged git.git_diff_unstaged git.git_reset github.get_commit github.search_commits git.git_log git.git_show github.push_files +q014,Create a Linear issue,verb_product,linear.linear_create_project,0.0,0.0,0.0,0.1429,0.3333,7,3,860,0.0,2,910,0.0,1,890,1.0,linear.linear_create_project jira.jira_get_issue linear.linear_get_issue github.issue_write linear.linear_update_issue gitlab.gitlab_get_issue linear.linear_create_issue sentry.sentry_update_issue linear.linear_create_comment sentry.sentry_get_issue +q015,Make a task in my project tracker Linear,paraphrase,asana.asana_get_task,0.0,0.0,0.0,0.0000,0.0000,145,49,12381,0.0,29,11702,0.0,15,11743,0.0,asana.asana_get_task linear.linear_create_project mysql.mysql_list_tables mysql.mysql_query clickup.clickup_update_task vercel.vercel_create_project clickup.clickup_get_task asana.asana_create_project asana.asana_update_task github.projects_write +q016,Post a comment on a GitHub issue,verb_product,figma.figma_post_comment,0.0,0.0,0.0,0.0000,0.0000,44,15,4304,0.0,9,4160,0.0,5,4542,0.0,figma.figma_post_comment github.update_issue_comment git.git_show github.get_discussion_comments notion.create_comment git.git_branch git.git_diff_unstaged linear.create_comment git.git_status github.add_comment_to_pending_review +q017,Merge the pull request,ambiguous,gitlab.gitlab_get_merge_request,0.0,1.0,1.0,0.3333,0.5174,3,1,321,1.0,1,543,1.0,1,1093,1.0,gitlab.gitlab_get_merge_request github.pull_request_review_write gitlab.gitlab_merge_merge_request github.merge_pull_request gitlab.gitlab_create_merge_request github.update_pull_request github.pull_request_read github.update_pull_request_branch gitlab.gitlab_list_merge_requests github.create_pull_request +q018,Trigger a GitHub Actions workflow,verb_product,github.actions_run_trigger,1.0,1.0,1.0,1.0000,1.0000,1,1,272,1.0,1,412,1.0,1,722,1.0,github.actions_run_trigger circleci.circleci_trigger_pipeline git.git_show git.git_branch github.actions_get_job_logs git.git_diff_unstaged github.actions_list git.git_status github.actions_get git.git_diff_staged +q019,Get Dependabot alerts,resource,github.list_dependabot_alerts,1.0,1.0,1.0,1.0000,0.9502,1,1,189,1.0,1,367,1.0,1,724,1.0,github.list_dependabot_alerts id_map.json github.get_dependabot_alert github.list_code_scanning_alerts github.list_secret_scanning_alerts +q020,List Sentry issues,verb_product,sentry.sentry_list_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,428,1.0,1,795,1.0,sentry.sentry_list_issues sentry.sentry_list_projects jira.jira_search_issues github.list_issues sentry.sentry_update_issue sentry.sentry_get_issue github.search_issues sentry.sentry_list_events linear.linear_list_issues +q021,Create a Stripe customer,verb_product,stripe.stripe_create_invoice,0.0,1.0,1.0,0.3333,0.5000,3,1,201,1.0,1,309,1.0,1,593,1.0,stripe.stripe_create_invoice stripe.stripe_list_customers stripe.stripe_create_customer stripe.stripe_get_payment_intent stripe.stripe_get_customer stripe.stripe_create_price stripe.stripe_create_product id_map.json stripe.stripe_create_payment_intent stripe.stripe_list_invoices +q022,Refund a Stripe payment,paraphrase,stripe.stripe_get_payment_intent,0.0,1.0,1.0,0.5000,0.6309,2,1,212,1.0,1,379,1.0,1,587,1.0,stripe.stripe_get_payment_intent stripe.stripe_create_refund stripe.stripe_list_customers stripe.stripe_create_payment_intent stripe.stripe_create_price stripe.stripe_create_invoice stripe.stripe_create_product id_map.json +q023,Send an email with Gmail,verb_product,gmail.gmail_send_email,1.0,1.0,1.0,1.0000,1.0000,1,1,264,1.0,1,377,1.0,1,710,1.0,gmail.gmail_send_email gmail.gmail_modify_email gmail.gmail_trash_email gmail.gmail_get_email filesystem.list_directory_with_sizes puppeteer.puppeteer_click gmail.gmail_list_labels asana.asana_get_task asana.asana_update_task asana.asana_create_task +q024,Search my Gmail inbox,verb_product,gmail.gmail_send_email,0.0,0.0,0.0,0.0000,0.0000,251,84,19910,0.0,51,19469,0.0,26,19411,0.0,gmail.gmail_send_email mysql.mysql_list_tables mysql.mysql_query gmail.gmail_modify_email gmail.gmail_trash_email gmail.gmail_get_email gmail.gmail_list_labels +q025,Upload a file to Google Drive,paraphrase,google_drive.gdrive_get_file,0.0,0.0,0.0,0.1000,0.2891,10,4,1083,0.0,2,947,0.0,1,925,1.0,google_drive.gdrive_get_file slack.slack_upload_file github.add_comment_to_pending_review google_calendar.gcal_delete_event google_drive.gdrive_update_file slack.slack_reply_to_thread google_drive.gdrive_search gitlab.gitlab_get_file github.assign_copilot_to_issue google_drive.gdrive_create_file +q026,Create a calendar event,resource,google_calendar.gcal_delete_event,0.0,1.0,1.0,0.5000,0.6309,2,1,300,1.0,1,427,1.0,1,748,1.0,google_calendar.gcal_delete_event google_calendar.gcal_create_event google_calendar.gcal_update_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars id_map.json kubernetes.k8s_get_events github.pull_request_review_write sentry.sentry_list_events +q027,List Kubernetes pods,verb_product,kubernetes.k8s_scale_deployment,0.0,1.0,1.0,0.5000,0.6309,2,1,262,1.0,1,416,1.0,1,627,1.0,kubernetes.k8s_scale_deployment kubernetes.k8s_list_pods kubernetes.k8s_delete kubernetes.k8s_list_deployments kubernetes.k8s_pod_logs kubernetes.k8s_list_services id_map.json +q028,Get logs from a k8s pod,verb_product,kubernetes.k8s_pod_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,273,1.0,1,450,1.0,1,801,1.0,kubernetes.k8s_pod_logs kubernetes.k8s_list_pods github.actions_get_job_logs kubernetes.k8s_get_pod datadog.datadog_list_logs docker.docker_logs id_map.json github.delete_file mongodb.mongo_delete datadog.datadog_query_metrics +q029,Run a docker container,verb_product,docker.docker_rm,0.0,1.0,1.0,0.3333,0.5000,3,1,219,1.0,1,348,1.0,1,646,1.0,docker.docker_rm docker.docker_stop docker.docker_run docker.docker_build docker.docker_exec docker.docker_logs docker.docker_ps kubernetes.k8s_pod_logs +q030,Show docker container logs,verb_product,docker.docker_rm,0.0,1.0,1.0,0.3333,0.5000,3,1,188,1.0,1,379,1.0,1,747,1.0,docker.docker_rm docker.docker_stop docker.docker_logs docker.docker_run kubernetes.k8s_pod_logs github.actions_get_job_logs docker.docker_build docker.docker_exec datadog.datadog_list_logs docker.docker_ps +q031,Navigate the browser to a URL,ambiguous,playwright.browser_navigate,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,423,1.0,1,731,1.0,playwright.browser_navigate puppeteer.puppeteer_navigate playwright.browser_tab_new github.add_comment_to_pending_review slack.slack_reply_to_thread playwright.browser_tab_list github.assign_copilot_to_issue playwright.browser_tab_close playwright.browser_wait_for memory.delete_entities +q032,Take a screenshot of the page,ambiguous,playwright.browser_take_screenshot,1.0,1.0,1.0,1.0000,1.0000,1,1,246,1.0,1,285,1.0,1,635,1.0,playwright.browser_take_screenshot puppeteer.puppeteer_screenshot slack.slack_set_channel_topic playwright.browser_snapshot id_map.json filesystem.list_directory playwright.browser_navigate memory.delete_entities github.update_pull_request memory.create_relations +q033,Create a GitLab merge request,verb_product,gitlab.gitlab_get_merge_request,0.0,1.0,1.0,0.3333,0.5000,3,1,297,1.0,1,431,1.0,1,831,1.0,gitlab.gitlab_get_merge_request gitlab.gitlab_merge_merge_request gitlab.gitlab_create_merge_request gitlab.gitlab_list_merge_requests git.git_show git.git_branch git.git_diff_unstaged github.merge_pull_request github.pull_request_review_write git.git_status +q034,Search Notion pages,verb_product,notion.notion_search,1.0,1.0,1.0,1.0000,1.0000,1,1,215,1.0,1,386,1.0,1,743,1.0,notion.notion_search notion.notion_append_block notion.notion_get_comments notion.notion_create_database notion.notion_query_database +q035,Create a Notion page,verb_product,notion.notion_append_block,0.0,0.0,0.0,0.0000,0.0000,166,56,13846,0.0,34,13474,0.0,17,13117,0.0,notion.notion_append_block notion.notion_update_page notion.notion_search notion.notion_get_page notion.notion_get_comments confluence.confluence_get_page notion.notion_create_database confluence.confluence_create_page notion.notion_query_database confluence.confluence_update_page +q036,Query a Notion database,verb_product,notion.notion_query_database,1.0,1.0,1.0,1.0000,1.0000,1,1,263,1.0,1,387,1.0,1,711,1.0,notion.notion_query_database notion.notion_create_database notion.notion_append_block notion.notion_search notion.notion_get_comments mongodb.mongo_delete mongodb.mongo_find mongodb.mongo_update mongodb.mongo_list_collections postgres.list_schemas +q037,Create a Zendesk ticket,verb_product,zendesk.zendesk_get_ticket,0.0,1.0,1.0,0.5000,0.6309,2,1,257,1.0,1,368,1.0,1,642,1.0,zendesk.zendesk_get_ticket zendesk.zendesk_create_ticket zendesk.zendesk_update_ticket zendesk.zendesk_search zendesk.zendesk_list_tickets id_map.json jira.jira_create_issue +q038,Create a HubSpot contact,verb_product,hubspot.hubspot_create_contact,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,364,1.0,1,801,1.0,hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_list_contacts hubspot.hubspot_search_contacts hubspot.hubspot_create_deal id_map.json github.delete_file github.pull_request_review_write github.get_discussion_comments github.get_discussion +q039,Create a Salesforce lead,verb_product,salesforce.salesforce_create_lead,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,423,1.0,1,634,1.0,salesforce.salesforce_create_lead salesforce.salesforce_get_record salesforce.salesforce_convert_lead salesforce.salesforce_update_record salesforce.salesforce_delete_record salesforce.salesforce_list_opportunities id_map.json +q040,Run a SOQL query,identifier,salesforce.salesforce_query,1.0,1.0,1.0,1.0000,1.0000,1,1,173,1.0,1,331,1.0,1,746,1.0,salesforce.salesforce_query +q041,List Shopify orders,verb_product,shopify.shopify_list_orders,1.0,1.0,1.0,1.0000,1.0000,1,1,230,1.0,1,376,1.0,1,626,1.0,shopify.shopify_list_orders shopify.shopify_list_products shopify.shopify_update_product shopify.shopify_create_product shopify.shopify_get_product id_map.json +q042,Create a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,0.0,0.0,0.0,0.1667,0.3562,6,2,486,0.0,2,736,0.0,1,715,1.0,pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls pagerduty.pagerduty_create_incident +q043,Who is on call in PagerDuty,paraphrase,pagerduty.pagerduty_list_oncalls,1.0,1.0,1.0,1.0000,1.0000,1,1,213,1.0,1,378,1.0,1,712,1.0,pagerduty.pagerduty_list_oncalls filesystem.list_allowed_directories twilio.twilio_make_call github.get_discussion_comments slack.slack_create_channel notion.notion_get_comments figma.figma_post_comment figma.figma_get_comments id_map.json github.create_pull_request +q044,Query Datadog metrics,verb_product,datadog.datadog_query_metrics,1.0,1.0,1.0,1.0000,1.0000,1,1,287,1.0,1,449,1.0,1,667,1.0,datadog.datadog_query_metrics datadog.datadog_create_monitor prometheus.prom_list_metrics datadog.datadog_list_logs datadog.datadog_list_monitors datadog.datadog_get_monitor id_map.json +q045,Search Datadog logs,verb_product,datadog.datadog_list_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,315,1.0,1,528,1.0,1,828,1.0,datadog.datadog_list_logs datadog.datadog_query_metrics github.actions_get_job_logs datadog.datadog_create_monitor kubernetes.k8s_pod_logs datadog.datadog_list_monitors docker.docker_logs datadog.datadog_get_monitor git.git_log +q046,List S3 buckets,verb_product,aws_s3.s3_list_buckets,1.0,1.0,1.0,1.0000,1.0000,1,1,115,1.0,1,220,1.0,1,642,1.0,aws_s3.s3_list_buckets id_map.json +q047,Upload an object to S3,paraphrase,aws_s3.s3_put_object,1.0,1.0,1.0,1.0000,1.0000,1,1,346,1.0,1,505,1.0,1,822,1.0,aws_s3.s3_put_object slack.slack_upload_file github.add_comment_to_pending_review aws_s3.s3_delete_object slack.slack_reply_to_thread aws_s3.s3_get_object github.assign_copilot_to_issue slack.slack_add_reaction id_map.json asana.asana_get_task +q048,Purge Cloudflare cache,verb_product,cloudflare.cf_purge_cache,1.0,1.0,1.0,1.0000,1.0000,1,1,231,1.0,1,286,1.0,1,643,1.0,cloudflare.cf_purge_cache cloudflare.cf_list_zones cloudflare.cf_create_dns cloudflare.cf_list_dns id_map.json +q049,Create a Cloudflare DNS record,verb_product,cloudflare.cf_create_dns,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,363,1.0,1,660,1.0,cloudflare.cf_create_dns cloudflare.cf_list_dns cloudflare.cf_list_zones salesforce.salesforce_get_record cloudflare.cf_purge_cache salesforce.salesforce_update_record salesforce.salesforce_delete_record salesforce.salesforce_create_record id_map.json git.git_commit +q050,List Vercel deployments,verb_product,vercel.vercel_list_deployments,1.0,1.0,1.0,1.0000,1.0000,1,1,173,1.0,1,291,1.0,1,541,1.0,vercel.vercel_list_deployments vercel.vercel_list_projects kubernetes.k8s_list_deployments vercel.vercel_get_deployment vercel.vercel_create_project id_map.json +q051,Insert a row into Supabase,verb_product,supabase.supabase_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,245,1.0,1,352,1.0,1,620,1.0,supabase.supabase_insert mongodb.mongo_insert supabase.supabase_update supabase.supabase_select supabase.supabase_list_tables supabase.supabase_invoke_fn playwright.browser_type id_map.json playwright.browser_click playwright.browser_evaluate +q052,Export images from Figma,paraphrase,figma.figma_get_images,1.0,1.0,1.0,1.0000,1.0000,1,1,215,1.0,1,325,1.0,1,797,1.0,figma.figma_get_images docker.docker_images figma.figma_post_comment figma.figma_get_comments figma.figma_get_file figma.figma_get_file_nodes github.delete_file mongodb.mongo_delete datadog.datadog_query_metrics github.get_file_contents +q053,Send an SMS with Twilio,verb_product,twilio.twilio_send_sms,1.0,1.0,1.0,1.0000,1.0000,1,1,161,1.0,1,308,1.0,1,703,1.0,twilio.twilio_send_sms twilio.twilio_list_messages id_map.json filesystem.list_directory_with_sizes twilio.twilio_make_call puppeteer.puppeteer_click asana.asana_get_task asana.asana_update_task asana.asana_create_task github.update_pull_request_branch +q054,Create an Okta user,verb_product,okta.okta_deactivate_user,0.0,1.0,1.0,0.5000,0.6309,2,1,183,1.0,1,289,1.0,1,657,1.0,okta.okta_deactivate_user okta.okta_create_user okta.okta_get_user slack.slack_get_user_profile okta.okta_list_users linear.get_user asana.asana_get_task asana.asana_update_task asana.asana_create_task asana.asana_create_project +q055,Instant query Prometheus,verb_product,prometheus.prom_query,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,275,1.0,1,681,1.0,prometheus.prom_query prometheus.prom_list_metrics prometheus.prom_query_range id_map.json +q056,Search Grafana dashboards,verb_product,grafana.grafana_search_dashboards,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,271,1.0,1,677,1.0,grafana.grafana_search_dashboards grafana.grafana_create_annotation grafana.grafana_get_dashboard id_map.json +q057,Trigger a CircleCI pipeline,verb_product,circleci.circleci_trigger_pipeline,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,271,1.0,1,634,1.0,circleci.circleci_trigger_pipeline circleci.circleci_get_pipeline circleci.circleci_list_pipelines gitlab.gitlab_create_pipeline id_map.json github.actions_run_trigger gitlab.get_pipeline gitlab.create_pipeline +q058,Run terraform plan,identifier,terraform.tf_plan,1.0,1.0,1.0,1.0000,1.0000,1,1,172,1.0,1,227,1.0,1,584,1.0,terraform.tf_plan terraform.tf_apply terraform.tf_state_list id_map.json postgres.explain_query +q059,Create an Asana task,verb_product,asana.asana_get_task,0.0,1.0,1.0,0.3333,0.5000,3,1,267,1.0,1,405,1.0,1,824,1.0,asana.asana_get_task asana.asana_update_task asana.asana_create_task asana.asana_create_project asana.asana_list_tasks clickup.clickup_update_task clickup.clickup_get_task clickup.clickup_create_task slack.slack_add_reaction gmail.gmail_send_email +q060,Create a Trello card,verb_product,trello.trello_list_cards,0.0,1.0,1.0,0.5000,0.6309,2,1,259,1.0,1,316,1.0,1,673,1.0,trello.trello_list_cards trello.trello_create_card trello.trello_add_comment trello.trello_create_board id_map.json +q061,Create a ClickUp task,verb_product,clickup.clickup_update_task,0.0,1.0,1.0,0.5000,0.6309,2,1,257,1.0,1,362,1.0,1,660,1.0,clickup.clickup_update_task clickup.clickup_create_task clickup.clickup_get_task clickup.clickup_list_tasks asana.asana_get_task asana.asana_update_task playwright.browser_click puppeteer.puppeteer_click +q062,Create a Confluence page,verb_product,confluence.confluence_get_page,0.0,1.0,1.0,0.5000,0.6309,2,1,260,1.0,1,382,1.0,1,686,1.0,confluence.confluence_get_page confluence.confluence_create_page confluence.confluence_update_page notion.notion_update_page notion.notion_get_page confluence.confluence_search +q063,Send a Discord message,verb_product,discord.discord_add_reaction,0.0,1.0,1.0,0.5000,0.6309,2,1,245,1.0,1,343,1.0,1,763,1.0,discord.discord_add_reaction discord.discord_send_message slack.slack_post_message discord.discord_get_channel discord.discord_list_channels discord.discord_list_members slack.slack_add_reaction github.delete_file figma.figma_post_comment gmail.gmail_modify_email +q064,Search Elasticsearch,verb_product,elasticsearch.es_get_doc,0.0,1.0,1.0,0.3333,0.5000,3,1,239,1.0,1,293,1.0,1,699,1.0,elasticsearch.es_get_doc elasticsearch.es_index_doc elasticsearch.es_search id_map.json +q065,Get a Redis key,verb_product,redis.redis_get,1.0,1.0,1.0,1.0000,1.0000,1,1,185,1.0,1,288,1.0,1,553,1.0,redis.redis_get redis.redis_set redis.redis_keys playwright.browser_press_key jira.jira_get_issue redis.redis_delete figma.figma_post_comment figma.figma_get_comments +q066,Find MongoDB documents,verb_product,mongodb.mongo_insert,0.0,1.0,1.0,0.3333,0.5000,3,1,244,1.0,1,381,1.0,1,738,1.0,mongodb.mongo_insert mongodb.mongo_delete mongodb.mongo_find mongodb.mongo_update mongodb.mongo_list_collections +q067,List SQLite tables,verb_product,sqlite.sqlite_list_tables,1.0,1.0,1.0,1.0000,1.0000,1,1,156,1.0,1,252,1.0,1,451,1.0,sqlite.sqlite_list_tables sqlite.sqlite_describe_table sqlite.sqlite_query mysql.mysql_list_tables postgres.list_tables supabase.supabase_list_tables id_map.json +q068,Describe a postgres table,verb_product,postgres.describe_table,1.0,1.0,1.0,1.0000,1.0000,1,1,170,1.0,1,286,1.0,1,543,1.0,postgres.describe_table sqlite.sqlite_describe_table postgres.list_schemas postgres.list_tables supabase.supabase_insert postgres.query supabase.supabase_update postgres.explain_query supabase.supabase_select id_map.json +q069,github_create_issue,identifier,github.list_check_runs,0.0,0.0,1.0,0.2500,0.1637,4,2,634,0.0,1,542,1.0,1,920,1.0,github.list_check_runs jira.jira_get_issue github.delete_file github.issue_write github.pull_request_review_write gitlab.gitlab_get_issue github.delete_check_run sentry.sentry_update_issue github.get_discussion_comments sentry.sentry_get_issue +q070,createIssue,identifier,jira.jira_get_issue,0.0,0.0,0.0,0.0000,0.0000,34,12,3256,0.0,7,3080,0.0,4,3574,0.0,jira.jira_get_issue github.issue_write gitlab.gitlab_get_issue sentry.sentry_update_issue sentry.sentry_get_issue +q071,list-directory,identifier,filesystem.create_directory,0.0,1.0,1.0,0.5000,0.6309,2,1,182,1.0,1,348,1.0,1,653,1.0,filesystem.create_directory filesystem.list_directory filesystem.directory_tree filesystem.list_directory_with_sizes github.get_file_contents filesystem.get_file_info +q072,sequential thinking next thought,paraphrase,sequential_thinking.sequentialthinking,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,291,1.0,1,713,1.0,sequential_thinking.sequentialthinking id_map.json +q073,What time is it in Tokyo,paraphrase,github.create_hook,0.0,0.0,1.0,0.2500,0.4345,4,2,452,0.0,1,380,1.0,1,726,1.0,github.create_hook gitlab.list_pipelines github.update_hook time.convert_time filesystem.list_allowed_directories time.get_current_time playwright.browser_wait_for github.get_team_members slack.slack_create_channel jira.jira_add_worklog +q074,Convert time between timezones,verb_product,time.convert_time,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,335,1.0,1,650,1.0,time.convert_time salesforce.salesforce_convert_lead time.get_current_time playwright.browser_wait_for git.git_diff memory.create_relations id_map.json jira.jira_add_worklog google_calendar.gcal_list_events grafana.grafana_create_annotation +q075,Star a GitHub repo,verb_product,github.star_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,220,1.0,1,380,1.0,1,773,1.0,github.star_repository git.git_show github.unstar_repository git.git_branch github.delete_file github.pull_request_review_write git.git_diff_unstaged github.get_discussion_comments git.git_status git.git_diff_staged +q076,Fork a repository,resource,github.fork_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,425,1.0,1,675,1.0,github.fork_repository github.star_repository github.unstar_repository github.create_repository github.get_repository_tree id_map.json +q077,Get the latest GitHub release,verb_product,github.get_latest_release,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,306,1.0,1,717,1.0,github.get_latest_release git.git_show git.git_branch git.git_diff_unstaged git.git_status git.git_diff_staged playwright.browser_navigate github.delete_file memory.delete_entities github.pull_request_review_write +q078,Create a gist,resource,github.create_gist,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,280,1.0,1,637,1.0,github.create_gist github.get_gist github.list_gists github.update_gist id_map.json +q079,Search GitHub code,verb_product,git.git_show,0.0,0.0,0.0,0.1667,0.3562,6,2,464,0.0,2,681,0.0,1,661,1.0,git.git_show github.get_code_scanning_alert git.git_branch github.list_code_scanning_alerts git.git_diff_unstaged github.search_code git.git_status git.git_diff_staged id_map.json github.delete_file +q080,List GitHub notifications,verb_product,git.git_show,0.0,1.0,1.0,0.5000,0.6309,2,1,192,1.0,1,284,1.0,1,737,1.0,git.git_show github.list_notifications git.git_branch github.mark_all_notifications_read git.git_diff_unstaged git.git_status git.git_diff_staged github.delete_file github.pull_request_review_write github.get_discussion_comments +q081,Assign Copilot to an issue,verb_product,github.assign_copilot_to_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,347,1.0,1,537,1.0,1,858,1.0,github.assign_copilot_to_issue github.add_comment_to_pending_review jira.jira_assign_issue github.request_copilot_review slack.slack_reply_to_thread asana.asana_create_task linear.linear_assign_issue slack.slack_add_reaction id_map.json asana.asana_get_task +q082,Get secret scanning alerts,resource,github.list_secret_scanning_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,295,1.0,1,373,1.0,1,720,1.0,github.list_secret_scanning_alerts github.get_secret_scanning_alert github.list_code_scanning_alerts id_map.json github.list_dependabot_alerts github.get_code_scanning_alert +q083,Create or update a file on GitHub,verb_product,github.create_or_update_file,1.0,1.0,1.0,1.0000,0.7602,1,1,286,1.0,1,423,1.0,1,768,1.0,github.create_or_update_file google_drive.gdrive_get_file git.git_show google_drive.gdrive_update_file git.git_branch gitlab.gitlab_get_file git.git_diff_unstaged figma.figma_get_file git.git_status gmail.gmail_modify_email +q084,List collaborators on a repo,resource,github.list_repository_collaborators,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,392,1.0,1,764,1.0,github.list_repository_collaborators github.get_discussion_comments github.star_repository pagerduty.pagerduty_list_oncalls github.unstar_repository notion.notion_get_comments github.get_repository_tree figma.figma_post_comment github.fork_repository figma.figma_get_comments +q085,Reply in a Slack thread,verb_product,slack.slack_reply_to_thread,1.0,1.0,1.0,1.0000,1.0000,1,1,256,1.0,1,411,1.0,1,675,1.0,slack.slack_reply_to_thread slack.slack_search_messages slack.slack_get_thread_replies slack.slack_add_reaction slack.slack_list_users github.get_notification_details slack.slack_post_message id_map.json jira.update_sprint linear.list_users +q086,Add an emoji reaction on Slack,verb_product,slack.slack_add_reaction,1.0,1.0,1.0,1.0000,1.0000,1,1,252,1.0,1,434,1.0,1,800,1.0,slack.slack_add_reaction discord.discord_add_reaction asana.asana_get_task github.get_discussion_comments asana.asana_update_task pagerduty.pagerduty_list_oncalls asana.asana_create_task notion.notion_get_comments asana.asana_create_project figma.figma_post_comment +q087,Search Slack messages,verb_product,slack.slack_search_messages,1.0,1.0,1.0,1.0000,1.0000,1,1,230,1.0,1,384,1.0,1,652,1.0,slack.slack_search_messages slack.slack_add_reaction twilio.twilio_list_messages slack.slack_list_users slack.slack_reply_to_thread slack.slack_post_message gmail.gmail_search_emails slack.slack_get_channel_history id_map.json +q088,Create a Slack channel,verb_product,slack.slack_search_messages,0.0,0.0,0.0,0.0000,0.0000,135,45,11143,0.0,27,10783,0.0,14,10993,0.0,slack.slack_search_messages discord.discord_get_channel slack.slack_add_reaction slack.slack_set_channel_topic slack.slack_list_users slack.slack_get_channel_history slack.slack_reply_to_thread slack.slack_leave_channel slack.slack_post_message slack.slack_join_channel +q089,Transition a Jira issue,verb_product,jira.jira_get_issue,0.0,1.0,1.0,0.5000,0.6309,2,1,213,1.0,1,383,1.0,1,772,1.0,jira.jira_get_issue jira.jira_transition_issue jira.jira_search_issues github.issue_write jira.jira_list_projects gitlab.gitlab_get_issue jira.jira_add_worklog sentry.sentry_update_issue jira.jira_add_comment sentry.sentry_get_issue +q090,Search Jira with JQL,identifier,jira.jira_search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,328,1.0,1,670,1.0,jira.jira_search_issues filesystem.list_directory_with_sizes jira.jira_get_issue jira.jira_list_projects jira.jira_add_worklog jira.jira_add_comment github.update_pull_request_branch puppeteer.puppeteer_click puppeteer.puppeteer_evaluate puppeteer.puppeteer_fill +q091,Add a worklog to Jira,verb_product,jira.jira_add_worklog,1.0,1.0,1.0,1.0000,1.0000,1,1,338,1.0,1,479,1.0,1,815,1.0,jira.jira_add_worklog github.add_comment_to_pending_review jira.jira_search_issues slack.slack_reply_to_thread jira.jira_get_issue github.assign_copilot_to_issue jira.jira_list_projects jira.jira_add_comment filesystem.list_allowed_directories slack.slack_add_reaction +q092,List Linear teams,verb_product,linear.linear_create_project,0.0,0.0,1.0,0.2500,0.4307,4,2,494,0.0,1,343,1.0,1,663,1.0,linear.linear_create_project github.get_teams linear.linear_get_issue linear.linear_list_teams linear.linear_update_issue linear.linear_create_issue linear.linear_create_comment id_map.json +q093,Comment on a Linear issue,verb_product,figma.figma_post_comment,0.0,0.0,0.0,0.0000,0.0000,147,49,12287,0.0,30,12042,0.0,15,11727,0.0,figma.figma_post_comment github.update_issue_comment linear.linear_create_project jira.jira_get_issue notion.create_comment linear.linear_get_issue github.issue_write linear.linear_update_issue gitlab.gitlab_get_issue linear.create_comment +q094,Move a local file,paraphrase,filesystem.move_file,1.0,1.0,1.0,1.0000,1.0000,1,1,213,1.0,1,395,1.0,1,746,1.0,filesystem.move_file brave_search.brave_local_search google_drive.gdrive_get_file google_drive.gdrive_update_file gitlab.gitlab_get_file gmail.gmail_modify_email figma.figma_get_file google_drive.gdrive_share_file docker.docker_rm filesystem.read_file +q095,Recursively search files by pattern,paraphrase,filesystem.search_files,1.0,1.0,1.0,1.0000,1.0000,1,1,297,1.0,1,419,1.0,1,748,1.0,filesystem.search_files github.push_files github.get_release_by_tag redis.redis_keys slack.list_files salesforce.salesforce_get_record jira.jira_get_issue linear.linear_get_issue github.search_repositories filesystem.list_directory +q096,Edit a file in place with old and new string,paraphrase,filesystem.edit_file,1.0,1.0,1.0,1.0000,1.0000,1,1,265,1.0,1,410,1.0,1,801,1.0,filesystem.edit_file google_drive.gdrive_get_file google_drive.gdrive_update_file gitlab.gitlab_get_file figma.figma_get_file google_drive.gdrive_share_file jira.update_sprint brave_search.brave_local_search linear.list_users twilio.twilio_make_call +q097,Read the whole knowledge graph,paraphrase,memory.read_graph,1.0,1.0,1.0,1.0000,1.0000,1,1,158,1.0,1,271,1.0,1,539,1.0,memory.read_graph memory.create_relations memory.delete_entities memory.search_nodes memory.add_observations memory.create_entities memory.delete_relations memory.open_nodes playwright.browser_navigate pagerduty.pagerduty_acknowledge_incident +q098,Search memory nodes,verb_product,memory.search_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,305,1.0,1,507,1.0,memory.search_nodes memory.delete_entities figma.figma_get_file_nodes memory.create_relations memory.open_nodes memory.delete_observations memory.add_observations id_map.json +q099,Show unstaged git diff,verb_product,git.git_diff_unstaged,1.0,1.0,1.0,1.0000,1.0000,1,1,187,1.0,1,297,1.0,1,508,1.0,git.git_diff_unstaged git.git_diff_staged git.git_show git.git_diff git.git_branch id_map.json git.git_status +q100,Stage files with git add,identifier,git.git_diff_unstaged,0.0,0.0,0.0,0.0000,0.0000,113,38,9484,0.0,23,9231,0.0,12,9385,0.0,git.git_diff_unstaged git.git_diff_staged filesystem.list_directory_with_sizes github.push_files git.git_show filesystem.search_files git.git_branch slack.list_files git.git_status id_map.json +q101,Create a Shopify product,verb_product,shopify.shopify_update_product,0.0,1.0,1.0,0.5000,0.6309,2,1,240,1.0,1,377,1.0,1,684,1.0,shopify.shopify_update_product shopify.shopify_create_product shopify.shopify_get_product shopify.shopify_list_products shopify.shopify_list_orders stripe.stripe_create_product stripe.stripe_create_price +q102,Cancel a Stripe subscription,verb_product,stripe.stripe_cancel_subscription,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,338,1.0,1,546,1.0,stripe.stripe_cancel_subscription stripe.stripe_list_subscriptions stripe.stripe_list_customers stripe.stripe_get_payment_intent stripe.stripe_create_price stripe.stripe_create_invoice stripe.stripe_create_product id_map.json +q103,Create a payment intent,resource,stripe.stripe_create_payment_intent,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,288,1.0,1,694,1.0,stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent stripe.stripe_create_refund id_map.json +q104,Acknowledge a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,363,1.0,1,626,1.0,pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls pagerduty.pagerduty_create_incident id_map.json +q105,Create a Datadog monitor,verb_product,datadog.datadog_create_monitor,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,459,1.0,1,709,1.0,datadog.datadog_create_monitor datadog.datadog_list_monitors datadog.datadog_get_monitor datadog.datadog_query_metrics datadog.datadog_list_logs id_map.json +q106,Resolve a Sentry issue,paraphrase,sentry.sentry_update_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,245,1.0,1,348,1.0,1,724,1.0,sentry.sentry_update_issue sentry.sentry_get_issue pagerduty.pagerduty_resolve_incident sentry.sentry_list_projects jira.jira_get_issue sentry.sentry_list_issues github.issue_write gitlab.gitlab_get_issue sentry.sentry_list_events id_map.json +q107,Scale a Kubernetes deployment,verb_product,kubernetes.k8s_scale_deployment,1.0,1.0,1.0,1.0000,1.0000,1,1,174,1.0,1,322,1.0,1,622,1.0,kubernetes.k8s_scale_deployment kubernetes.k8s_list_deployments id_map.json kubernetes.k8s_delete vercel.vercel_get_deployment kubernetes.k8s_pod_logs vercel.vercel_list_deployments kubernetes.k8s_list_services +q108,Apply a Kubernetes manifest,verb_product,kubernetes.k8s_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,246,1.0,1,390,1.0,1,632,1.0,kubernetes.k8s_apply kubernetes.k8s_scale_deployment terraform.tf_apply kubernetes.k8s_delete kubernetes.k8s_list_deployments kubernetes.k8s_pod_logs kubernetes.k8s_list_services id_map.json +q109,Stop a docker container,verb_product,docker.docker_stop,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,348,1.0,1,608,1.0,docker.docker_stop docker.docker_rm docker.docker_run docker.docker_build docker.docker_exec docker.docker_logs docker.docker_ps kubernetes.k8s_pod_logs id_map.json +q110,Build a docker image,verb_product,docker.docker_build,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,330,1.0,1,563,1.0,docker.docker_build docker.docker_run docker.docker_rm docker.docker_stop docker.docker_ps docker.docker_images figma.figma_get_images id_map.json filesystem.read_media_file +q111,Click an element in Playwright,verb_product,playwright.browser_click,1.0,1.0,1.0,1.0000,1.0000,1,1,230,1.0,1,374,1.0,1,734,1.0,playwright.browser_click puppeteer.puppeteer_click clickup.clickup_create_task clickup.clickup_list_tasks clickup.clickup_update_task playwright.browser_type asana.asana_get_task playwright.browser_take_screenshot jira.update_sprint asana.asana_update_task +q112,Type text into a browser input,paraphrase,playwright.browser_type,1.0,1.0,1.0,1.0000,0.7602,1,1,252,1.0,1,353,1.0,1,682,1.0,playwright.browser_type playwright.browser_wait_for filesystem.read_text_file playwright.browser_tab_new playwright.browser_navigate playwright.browser_tab_list playwright.browser_tab_close mongodb.mongo_insert cloudflare.cf_create_dns supabase.supabase_insert +q113,List GitLab projects,verb_product,gitlab.gitlab_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,170,1.0,1,267,1.0,1,524,1.0,gitlab.gitlab_list_projects git.git_show vercel.vercel_list_projects git.git_branch sentry.sentry_list_projects git.git_diff_unstaged jira.jira_list_projects git.git_status github.projects_list git.git_diff_staged +q114,Create a Bitbucket pull request,verb_product,github.pull_request_review_write,0.0,0.0,0.0,0.1111,0.3010,9,3,943,0.0,2,1016,0.0,1,994,1.0,github.pull_request_review_write bitbucket.bitbucket_list_prs gitlab.gitlab_get_merge_request bitbucket.bitbucket_create_issue github.update_pull_request gitlab.gitlab_merge_merge_request bitbucket.bitbucket_list_repos github.pull_request_read bitbucket.bitbucket_create_pr github.update_pull_request_branch +q115,Create a Gmail draft,verb_product,gmail.gmail_send_email,0.0,1.0,1.0,0.5000,0.6309,2,1,288,1.0,1,398,1.0,1,673,1.0,gmail.gmail_send_email gmail.gmail_create_draft gmail.gmail_modify_email gmail.gmail_trash_email gmail.gmail_get_email gmail.gmail_list_labels github.create_pull_request id_map.json +q116,Share a Google Drive file,verb_product,google_drive.gdrive_get_file,0.0,0.0,1.0,0.2500,0.4307,4,2,510,0.0,1,417,1.0,1,778,1.0,google_drive.gdrive_get_file google_drive.gdrive_search google_drive.gdrive_create_file google_drive.gdrive_share_file google_drive.gdrive_update_file google_calendar.gcal_delete_event gitlab.gitlab_get_file google_calendar.gcal_update_event google_drive.gdrive_list_folder figma.figma_get_file +q117,List Google Calendar events,verb_product,google_calendar.gcal_delete_event,0.0,1.0,1.0,0.5000,0.6309,2,1,269,1.0,1,376,1.0,1,770,1.0,google_calendar.gcal_delete_event google_calendar.gcal_list_events google_calendar.gcal_update_event google_drive.gdrive_search kubernetes.k8s_get_events google_drive.gdrive_create_file google_calendar.gcal_create_event sentry.sentry_list_events google_drive.gdrive_get_file google_calendar.gcal_list_calendars +q118,Delete a calendar event,resource,google_calendar.gcal_delete_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,427,1.0,1,748,1.0,google_calendar.gcal_delete_event google_calendar.gcal_create_event google_calendar.gcal_update_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars id_map.json kubernetes.k8s_get_events github.pull_request_review_write sentry.sentry_list_events +q119,List Vercel projects,verb_product,vercel.vercel_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,170,1.0,1,262,1.0,1,529,1.0,vercel.vercel_list_projects vercel.vercel_get_deployment sentry.sentry_list_projects vercel.vercel_list_deployments jira.jira_list_projects vercel.vercel_create_project github.projects_list gitlab.gitlab_list_projects +q120,Select rows from Supabase,verb_product,supabase.supabase_select,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,354,1.0,1,766,1.0,supabase.supabase_select supabase.supabase_insert supabase.supabase_update supabase.supabase_list_tables supabase.supabase_invoke_fn github.delete_file mongodb.mongo_delete puppeteer.puppeteer_click datadog.datadog_query_metrics puppeteer.puppeteer_fill +q121,Post a Figma comment,verb_product,figma.figma_post_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,406,1.0,1,873,1.0,figma.figma_post_comment github.update_issue_comment figma.figma_get_comments notion.create_comment figma.figma_get_file figma.figma_get_images linear.create_comment figma.figma_get_file_nodes github.add_comment_to_pending_review +q122,List Okta users,verb_product,okta.okta_create_user,0.0,0.0,1.0,0.2000,0.3869,5,2,417,0.0,1,317,1.0,1,617,1.0,okta.okta_create_user slack.slack_list_users okta.okta_deactivate_user linear.list_users okta.okta_list_users notion.list_users_tokens okta.okta_get_user notion.delete_users_token notion.get_users_token +q123,Range query Prometheus,verb_product,prometheus.prom_query_range,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,275,1.0,1,681,1.0,prometheus.prom_query_range prometheus.prom_list_metrics prometheus.prom_query id_map.json +q124,Create a Grafana annotation,verb_product,grafana.grafana_create_annotation,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,272,1.0,1,678,1.0,grafana.grafana_create_annotation grafana.grafana_get_dashboard grafana.grafana_search_dashboards id_map.json +q125,Apply terraform,identifier,terraform.tf_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,146,1.0,1,238,1.0,1,595,1.0,terraform.tf_apply id_map.json kubernetes.k8s_apply terraform.tf_plan terraform.tf_state_list +q126,List Asana tasks,verb_product,asana.asana_list_tasks,1.0,1.0,1.0,1.0000,1.0000,1,1,192,1.0,1,386,1.0,1,616,1.0,asana.asana_list_tasks asana.asana_get_task clickup.clickup_list_tasks asana.asana_update_task asana.asana_create_task asana.asana_create_project id_map.json +q127,Comment on a Trello card,verb_product,trello.trello_list_cards,0.0,1.0,1.0,0.5000,0.6309,2,1,260,1.0,1,438,1.0,1,934,1.0,trello.trello_list_cards trello.trello_add_comment trello.trello_create_card figma.figma_post_comment github.update_issue_comment trello.trello_create_board notion.create_comment linear.create_comment github.add_comment_to_pending_review github.get_discussion_comments +q128,Update a ClickUp task,verb_product,clickup.clickup_update_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,362,1.0,1,660,1.0,clickup.clickup_update_task clickup.clickup_create_task clickup.clickup_get_task clickup.clickup_list_tasks asana.asana_get_task asana.asana_update_task playwright.browser_click puppeteer.puppeteer_click +q129,Search Confluence,verb_product,confluence.confluence_get_page,0.0,0.0,1.0,0.2500,0.4307,4,2,423,0.0,1,366,1.0,1,772,1.0,confluence.confluence_get_page confluence.confluence_create_page confluence.confluence_update_page confluence.confluence_search +q130,Create a HubSpot deal,verb_product,hubspot.hubspot_create_deal,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,873,1.0,hubspot.hubspot_create_deal hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_list_contacts hubspot.hubspot_search_contacts github.delete_file github.pull_request_review_write github.get_discussion_comments github.get_discussion github.list_discussions +q131,List Zendesk tickets,verb_product,zendesk.zendesk_list_tickets,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,367,1.0,1,617,1.0,zendesk.zendesk_list_tickets zendesk.zendesk_search zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_create_ticket id_map.json +q132,List Discord channels,verb_product,discord.discord_list_channels,1.0,1.0,1.0,1.0000,1.0000,1,1,205,1.0,1,339,1.0,1,547,1.0,discord.discord_list_channels discord.discord_add_reaction discord.discord_get_channel slack.slack_list_channels discord.discord_send_message discord.discord_list_members id_map.json +q133,Set a Redis key,verb_product,redis.redis_get,0.0,1.0,1.0,0.5000,0.6309,2,1,185,1.0,1,288,1.0,1,553,1.0,redis.redis_get redis.redis_set redis.redis_keys playwright.browser_press_key jira.jira_get_issue redis.redis_delete figma.figma_post_comment figma.figma_get_comments +q134,Insert MongoDB documents,verb_product,mongodb.mongo_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,381,1.0,1,607,1.0,mongodb.mongo_insert mongodb.mongo_delete mongodb.mongo_find mongodb.mongo_update mongodb.mongo_list_collections supabase.supabase_insert id_map.json +q135,Run a MySQL query,verb_product,mysql.mysql_query,1.0,1.0,1.0,1.0000,1.0000,1,1,173,1.0,1,281,1.0,1,568,1.0,mysql.mysql_query mysql.mysql_list_tables sqlite.sqlite_query postgres.query postgres.explain_query postgres.list_schemas +q136,Index a document in Elasticsearch,verb_product,elasticsearch.es_index_doc,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,318,1.0,1,661,1.0,elasticsearch.es_index_doc elasticsearch.es_get_doc elasticsearch.es_search mongodb.mongo_find id_map.json mongodb.mongo_insert jira.update_sprint mongodb.mongo_delete linear.list_users gitlab.get_pipeline +q137,Get an S3 object,verb_product,aws_s3.s3_delete_object,0.0,1.0,1.0,0.5000,0.6309,2,1,249,1.0,1,301,1.0,1,728,1.0,aws_s3.s3_delete_object aws_s3.s3_get_object aws_s3.s3_put_object id_map.json asana.asana_get_task aws_s3.s3_list_objects asana.asana_update_task asana.asana_create_task asana.asana_create_project slack.slack_add_reaction +q138,List Cloudflare zones,verb_product,cloudflare.cf_list_zones,1.0,1.0,1.0,1.0000,1.0000,1,1,230,1.0,1,363,1.0,1,613,1.0,cloudflare.cf_list_zones cloudflare.cf_purge_cache cloudflare.cf_create_dns cloudflare.cf_list_dns time.convert_time id_map.json +q139,Create a Stripe invoice,verb_product,stripe.stripe_create_invoice,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,320,1.0,1,547,1.0,stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_list_customers stripe.stripe_get_payment_intent stripe.stripe_create_price stripe.stripe_create_product id_map.json +q140,List GitHub Actions workflows,verb_product,github.actions_list,1.0,1.0,1.0,1.0000,1.0000,1,1,268,1.0,1,408,1.0,1,765,1.0,github.actions_list git.git_show github.actions_run_trigger git.git_branch github.actions_get_job_logs git.git_diff_unstaged git.git_status github.actions_get git.git_diff_staged github.delete_file +q141,Get GitHub Actions job logs,verb_product,github.actions_get_job_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,296,1.0,1,396,1.0,1,723,1.0,github.actions_get_job_logs git.git_show github.actions_run_trigger git.git_branch gitlab.delete_job git.git_diff_unstaged github.actions_list gitlab.update_job git.git_status github.actions_get +q142,Create a GitHub repository,verb_product,github.star_repository,0.0,0.0,0.0,0.1667,0.3562,6,2,421,0.0,2,712,0.0,1,691,1.0,github.star_repository git.git_show git.git_branch github.unstar_repository git.git_diff_unstaged github.create_repository git.git_status github.get_repository_tree git.git_diff_staged github.fork_repository +q143,Checkout a git branch,verb_product,git.git_branch,0.0,1.0,1.0,0.5000,0.6309,2,1,189,1.0,1,373,1.0,1,739,1.0,git.git_branch git.git_checkout git.git_show git.git_create_branch github.create_branch git.git_diff_unstaged github.update_pull_request_branch git.git_status git.git_diff_staged github.delete_file +q144,Show git log,identifier,git.git_show,0.0,0.0,1.0,0.2000,0.3869,5,2,364,0.0,1,294,1.0,1,694,1.0,git.git_show git.git_branch git.git_diff_unstaged git.git_status git.git_log git.git_diff_staged datadog.datadog_list_logs docker.docker_logs jira.jira_add_worklog kubernetes.k8s_pod_logs +q145,List postgres schemas,verb_product,postgres.list_schemas,1.0,1.0,1.0,1.0000,1.0000,1,1,169,1.0,1,277,1.0,1,527,1.0,postgres.list_schemas postgres.describe_table postgres.list_tables postgres.query postgres.explain_query id_map.json +q146,Open nodes in the knowledge graph,verb_product,memory.search_nodes,0.0,1.0,1.0,0.5000,0.6309,2,1,205,1.0,1,308,1.0,1,570,1.0,memory.search_nodes memory.open_nodes figma.figma_get_file_nodes memory.delete_entities memory.create_relations id_map.json jira.update_sprint playwright.browser_navigate pagerduty.pagerduty_acknowledge_incident linear.list_users +q147,Get file metadata from the filesystem,paraphrase,google_drive.gdrive_get_file,0.0,0.0,0.0,0.1250,0.3155,8,3,723,0.0,2,752,0.0,1,730,1.0,google_drive.gdrive_get_file memory.delete_entities google_drive.gdrive_update_file gitlab.gitlab_get_file figma.figma_get_file filesystem.read_media_file google_drive.gdrive_share_file filesystem.get_file_info github.delete_file playwright.browser_navigate +q148,Write a new local file,paraphrase,playwright.browser_tab_new,0.0,0.0,0.0,0.0000,0.0000,87,29,7985,0.0,18,7922,0.0,9,7733,0.0,playwright.browser_tab_new brave_search.brave_local_search google_drive.gdrive_get_file filesystem.edit_file google_drive.gdrive_update_file gitlab.gitlab_get_file figma.figma_get_file google_drive.gdrive_share_file filesystem.read_file filesystem.read_media_file +q149,Search GitHub issues,verb_product,git.git_show,0.0,0.0,0.0,0.1250,0.3155,8,3,679,0.0,2,744,0.0,1,724,1.0,git.git_show jira.jira_search_issues git.git_branch github.list_issues git.git_diff_unstaged sentry.sentry_list_issues git.git_status github.search_issues git.git_diff_staged linear.linear_list_issues +q150,Get the authenticated GitHub user,paraphrase,git.git_show,0.0,0.0,0.0,0.1250,0.3155,8,3,614,0.0,2,644,0.0,1,622,1.0,git.git_show git.git_branch github.list_notifications git.git_diff_unstaged git.git_status git.git_diff_staged playwright.browser_navigate github.get_me github.delete_file memory.delete_entities diff --git a/experiments/mcp-tool-search/results/codedb_trigram_tokens.json b/experiments/mcp-tool-search/results/codedb_trigram_tokens.json new file mode 100644 index 00000000..abf3dfbf --- /dev/null +++ b/experiments/mcp-tool-search/results/codedb_trigram_tokens.json @@ -0,0 +1,6776 @@ +{ + "summary": { + "n": 150, + "index_s": 0.44171834199994464, + "rss_mb": null, + "p50_ms": 30.7823924999866, + "p95_ms": 41.255320850029875, + "recall@1": 0.6066666666666667, + "recall@1_lo": 0.5266666666666666, + "recall@1_hi": 0.6866666666666666, + "recall@3": 0.8066666666666666, + "recall@3_lo": 0.74, + "recall@3_hi": 0.8666666666666667, + "recall@5": 0.8533333333333334, + "recall@5_lo": 0.8, + "recall@5_hi": 0.9066666666666666, + "mrr@10": 0.7183042328042328, + "mrr@10_lo": 0.6582407407407408, + "mrr@10_hi": 0.7776190476190475, + "ndcg@10": 0.759147729800791, + "ndcg@10_lo": 0.7073246254501939, + "ndcg@10_hi": 0.809538220808961 + }, + "meta": { + "name": "codedb_trigram_tokens", + "needs_model": false, + "needs_api": false, + "notes": "per-token codedb_search (single-token trigram path) fused with RRF" + }, + "rows": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.issue_write", + "git.git_show", + "jira.jira_get_issue", + "git.git_branch", + "git.git_diff_unstaged", + "gitlab.gitlab_get_issue", + "git.git_status", + "sentry.sentry_update_issue", + "git.git_diff_staged", + "sentry.sentry_get_issue" + ], + "hit1": "github.issue_write", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.38009376671593426, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 270, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "kind": "verb_product", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_search_issues", + "zendesk.zendesk_create_ticket", + "jira.jira_get_issue", + "zendesk.zendesk_get_ticket", + "jira.jira_list_projects", + "zendesk.zendesk_update_ticket", + "jira.jira_add_worklog", + "jira.jira_add_comment", + "id_map.json", + "jira.jira_create_issue" + ], + "hit1": "jira.jira_search_issues", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.1, + "ndcg@10": 0.2890648263178879, + "hit_rank": 10, + "k3_calls": 4, + "k3_tokens": 839, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 2, + "k5_tokens": 707, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "kind": "verb_product", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "ranking": [ + "slack.slack_post_message", + "slack.slack_add_reaction", + "slack.slack_search_messages", + "discord.discord_send_message", + "slack.slack_list_users", + "discord.discord_add_reaction", + "slack.slack_reply_to_thread", + "github.delete_file", + "figma.figma_post_comment", + "gmail.gmail_modify_email" + ], + "hit1": "slack.slack_post_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 839, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "kind": "resource", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "ranking": [ + "github.pull_request_review_write", + "github.list_pull_requests", + "github.update_pull_request", + "github.search_pull_requests", + "github.pull_request_read", + "gitlab.gitlab_list_merge_requests", + "github.update_pull_request_branch", + "github.merge_pull_request", + "id_map.json", + "bitbucket.bitbucket_list_prs" + ], + "hit1": "github.pull_request_review_write", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6433224083306327, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 393, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 536, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 885, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "ranking": [ + "github.pull_request_review_write", + "github.create_pull_request", + "gitlab.gitlab_get_merge_request", + "git.git_show", + "github.update_pull_request", + "gitlab.gitlab_merge_merge_request", + "git.git_branch", + "github.pull_request_read", + "git.git_diff_unstaged", + "github.update_pull_request_branch" + ], + "hit1": "github.pull_request_review_write", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 395, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 598, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 962, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "kind": "verb_product", + "gold": [ + "postgres.query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "postgres.explain_query", + "mysql.mysql_query", + "sqlite.sqlite_query", + "postgres.list_schemas", + "postgres.describe_table", + "postgres.list_tables", + "salesforce.salesforce_query", + "mysql.mysql_list_tables" + ], + "hit1": "postgres.query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 272, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 546, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "kind": "paraphrase", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "ranking": [ + "brave_search.brave_local_search", + "google_drive.gdrive_get_file", + "google_drive.gdrive_update_file", + "filesystem.read_media_file", + "gitlab.gitlab_get_file", + "figma.figma_get_file", + "google_drive.gdrive_share_file", + "github.delete_file", + "mongodb.mongo_delete", + "filesystem.read_file" + ], + "hit1": "brave_search.brave_local_search", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.1, + "ndcg@10": 0.21974347732050664, + "hit_rank": 10, + "k3_calls": 4, + "k3_tokens": 937, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 2, + "k5_tokens": 801, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 779, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "kind": "paraphrase", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "ranking": [ + "github.push_files", + "filesystem.search_files", + "slack.list_files", + "jira.update_sprint", + "linear.list_users", + "gitlab.get_pipeline", + "elasticsearch.es_index_doc", + "linear.update_label", + "git.git_add", + "linear.create_comment" + ], + "hit1": "github.push_files", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 91, + "k3_calls": 31, + "k3_tokens": 8425, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 19, + "k5_tokens": 8277, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 10, + "k10_tokens": 8326, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "ranking": [ + "github.create_artifact", + "github.delete_artifact", + "github.get_artifact", + "github.list_artifacts", + "github.update_artifact", + "id_map.json", + "jira.update_sprint", + "linear.list_users", + "gitlab.get_pipeline", + "elasticsearch.es_index_doc" + ], + "hit1": "github.create_artifact", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 99, + "k3_calls": 33, + "k3_tokens": 8757, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 20, + "k5_tokens": 8509, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 10, + "k10_tokens": 8279, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "kind": "verb_product", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "ranking": [ + "brave_search.brave_web_search", + "id_map.json", + "filesystem.list_directory_with_sizes", + "brave_search.brave_local_search", + "jira.delete_webhook", + "jira.get_webhook", + "jira.list_webhooks", + "jira.update_webhook", + "playwright.browser_navigate", + "memory.delete_entities" + ], + "hit1": "brave_search.brave_web_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 169, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 298, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "kind": "verb_product", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "ranking": [ + "fetch.fetch", + "playwright.browser_navigate", + "github.get_file_contents", + "slack.slack_set_channel_topic", + "playwright.browser_snapshot", + "puppeteer.puppeteer_navigate", + "filesystem.list_directory", + "playwright.browser_tab_new", + "memory.delete_entities", + "twilio.twilio_make_call" + ], + "hit1": "fetch.fetch", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 671, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "kind": "identifier", + "gold": [ + "git.git_status" + ], + "alts": [], + "ranking": [ + "git.git_status", + "git.git_show", + "git.git_branch", + "sentry.sentry_update_issue", + "git.git_diff_unstaged", + "git.git_diff_staged", + "github.list_check_runs", + "gitlab.list_jobs", + "slack.list_bookmarks", + "jira.list_attachments" + ], + "hit1": "git.git_status", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 681, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "kind": "paraphrase", + "gold": [ + "git.git_commit" + ], + "alts": [], + "ranking": [ + "git.git_commit", + "id_map.json", + "git.git_diff_staged", + "git.git_diff_unstaged", + "git.git_reset", + "github.get_commit", + "github.search_commits", + "git.git_log", + "git.git_show", + "github.push_files" + ], + "hit1": "git.git_commit", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 134, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 236, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 611, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_project", + "jira.jira_get_issue", + "linear.linear_get_issue", + "github.issue_write", + "linear.linear_update_issue", + "gitlab.gitlab_get_issue", + "linear.linear_create_issue", + "sentry.sentry_update_issue", + "linear.linear_create_comment", + "sentry.sentry_get_issue" + ], + "hit1": "linear.linear_create_project", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.14285714285714285, + "ndcg@10": 0.3333333333333333, + "hit_rank": 7, + "k3_calls": 3, + "k3_tokens": 860, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 910, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 890, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "kind": "paraphrase", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "asana.asana_get_task", + "linear.linear_create_project", + "mysql.mysql_list_tables", + "mysql.mysql_query", + "clickup.clickup_update_task", + "vercel.vercel_create_project", + "clickup.clickup_get_task", + "asana.asana_create_project", + "asana.asana_update_task", + "github.projects_write" + ], + "hit1": "asana.asana_get_task", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 145, + "k3_calls": 49, + "k3_tokens": 12381, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 29, + "k5_tokens": 11702, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11743, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "github.update_issue_comment", + "git.git_show", + "github.get_discussion_comments", + "notion.create_comment", + "git.git_branch", + "git.git_diff_unstaged", + "linear.create_comment", + "git.git_status", + "github.add_comment_to_pending_review" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 44, + "k3_calls": 15, + "k3_tokens": 4304, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 9, + "k5_tokens": 4160, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 5, + "k10_tokens": 4542, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "kind": "ambiguous", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "ranking": [ + "gitlab.gitlab_get_merge_request", + "github.pull_request_review_write", + "gitlab.gitlab_merge_merge_request", + "github.merge_pull_request", + "gitlab.gitlab_create_merge_request", + "github.update_pull_request", + "github.pull_request_read", + "github.update_pull_request_branch", + "gitlab.gitlab_list_merge_requests", + "github.create_pull_request" + ], + "hit1": "gitlab.gitlab_get_merge_request", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5174418337467067, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 321, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 543, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1093, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "kind": "verb_product", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "ranking": [ + "github.actions_run_trigger", + "circleci.circleci_trigger_pipeline", + "git.git_show", + "git.git_branch", + "github.actions_get_job_logs", + "git.git_diff_unstaged", + "github.actions_list", + "git.git_status", + "github.actions_get", + "git.git_diff_staged" + ], + "hit1": "github.actions_run_trigger", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 272, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 722, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "kind": "resource", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "ranking": [ + "github.list_dependabot_alerts", + "id_map.json", + "github.get_dependabot_alert", + "github.list_code_scanning_alerts", + "github.list_secret_scanning_alerts" + ], + "hit1": "github.list_dependabot_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "kind": "verb_product", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_issues", + "sentry.sentry_list_projects", + "jira.jira_search_issues", + "github.list_issues", + "sentry.sentry_update_issue", + "sentry.sentry_get_issue", + "github.search_issues", + "sentry.sentry_list_events", + "linear.linear_list_issues" + ], + "hit1": "sentry.sentry_list_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 795, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_list_customers", + "stripe.stripe_create_customer", + "stripe.stripe_get_payment_intent", + "stripe.stripe_get_customer", + "stripe.stripe_create_price", + "stripe.stripe_create_product", + "id_map.json", + "stripe.stripe_create_payment_intent", + "stripe.stripe_list_invoices" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 201, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 309, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 593, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "kind": "paraphrase", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "ranking": [ + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_refund", + "stripe.stripe_list_customers", + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_price", + "stripe.stripe_create_invoice", + "stripe.stripe_create_product", + "id_map.json" + ], + "hit1": "stripe.stripe_get_payment_intent", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 587, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "kind": "verb_product", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "ranking": [ + "gmail.gmail_send_email", + "gmail.gmail_modify_email", + "gmail.gmail_trash_email", + "gmail.gmail_get_email", + "filesystem.list_directory_with_sizes", + "puppeteer.puppeteer_click", + "gmail.gmail_list_labels", + "asana.asana_get_task", + "asana.asana_update_task", + "asana.asana_create_task" + ], + "hit1": "gmail.gmail_send_email", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 264, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 710, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "kind": "verb_product", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "ranking": [ + "gmail.gmail_send_email", + "mysql.mysql_list_tables", + "mysql.mysql_query", + "gmail.gmail_modify_email", + "gmail.gmail_trash_email", + "gmail.gmail_get_email", + "gmail.gmail_list_labels" + ], + "hit1": "gmail.gmail_send_email", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 251, + "k3_calls": 84, + "k3_tokens": 19910, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 51, + "k5_tokens": 19469, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 26, + "k10_tokens": 19411, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "kind": "paraphrase", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_get_file", + "slack.slack_upload_file", + "github.add_comment_to_pending_review", + "google_calendar.gcal_delete_event", + "google_drive.gdrive_update_file", + "slack.slack_reply_to_thread", + "google_drive.gdrive_search", + "gitlab.gitlab_get_file", + "github.assign_copilot_to_issue", + "google_drive.gdrive_create_file" + ], + "hit1": "google_drive.gdrive_get_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.1, + "ndcg@10": 0.2890648263178879, + "hit_rank": 10, + "k3_calls": 4, + "k3_tokens": 1083, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 2, + "k5_tokens": 947, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 925, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "id_map.json", + "kubernetes.k8s_get_events", + "github.pull_request_review_write", + "sentry.sentry_list_events" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 748, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_delete", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_list_services", + "id_map.json" + ], + "hit1": "kubernetes.k8s_scale_deployment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 627, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_list_pods", + "github.actions_get_job_logs", + "kubernetes.k8s_get_pod", + "datadog.datadog_list_logs", + "docker.docker_logs", + "id_map.json", + "github.delete_file", + "mongodb.mongo_delete", + "datadog.datadog_query_metrics" + ], + "hit1": "kubernetes.k8s_pod_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 273, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 450, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 801, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "ranking": [ + "docker.docker_rm", + "docker.docker_stop", + "docker.docker_run", + "docker.docker_build", + "docker.docker_exec", + "docker.docker_logs", + "docker.docker_ps", + "kubernetes.k8s_pod_logs" + ], + "hit1": "docker.docker_rm", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 646, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "kind": "verb_product", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "ranking": [ + "docker.docker_rm", + "docker.docker_stop", + "docker.docker_logs", + "docker.docker_run", + "kubernetes.k8s_pod_logs", + "github.actions_get_job_logs", + "docker.docker_build", + "docker.docker_exec", + "datadog.datadog_list_logs", + "docker.docker_ps" + ], + "hit1": "docker.docker_rm", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 747, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "kind": "ambiguous", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "ranking": [ + "playwright.browser_navigate", + "puppeteer.puppeteer_navigate", + "playwright.browser_tab_new", + "github.add_comment_to_pending_review", + "slack.slack_reply_to_thread", + "playwright.browser_tab_list", + "github.assign_copilot_to_issue", + "playwright.browser_tab_close", + "playwright.browser_wait_for", + "memory.delete_entities" + ], + "hit1": "playwright.browser_navigate", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 423, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 731, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "kind": "ambiguous", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "ranking": [ + "playwright.browser_take_screenshot", + "puppeteer.puppeteer_screenshot", + "slack.slack_set_channel_topic", + "playwright.browser_snapshot", + "id_map.json", + "filesystem.list_directory", + "playwright.browser_navigate", + "memory.delete_entities", + "github.update_pull_request", + "memory.create_relations" + ], + "hit1": "playwright.browser_take_screenshot", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 246, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 285, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 635, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_get_merge_request", + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_create_merge_request", + "gitlab.gitlab_list_merge_requests", + "git.git_show", + "git.git_branch", + "git.git_diff_unstaged", + "github.merge_pull_request", + "github.pull_request_review_write", + "git.git_status" + ], + "hit1": "gitlab.gitlab_get_merge_request", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 431, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 831, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "kind": "verb_product", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "ranking": [ + "notion.notion_search", + "notion.notion_append_block", + "notion.notion_get_comments", + "notion.notion_create_database", + "notion.notion_query_database" + ], + "hit1": "notion.notion_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 743, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "kind": "verb_product", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "ranking": [ + "notion.notion_append_block", + "notion.notion_update_page", + "notion.notion_search", + "notion.notion_get_page", + "notion.notion_get_comments", + "confluence.confluence_get_page", + "notion.notion_create_database", + "confluence.confluence_create_page", + "notion.notion_query_database", + "confluence.confluence_update_page" + ], + "hit1": "notion.notion_append_block", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 166, + "k3_calls": 56, + "k3_tokens": 13846, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 34, + "k5_tokens": 13474, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 17, + "k10_tokens": 13117, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "kind": "verb_product", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "ranking": [ + "notion.notion_query_database", + "notion.notion_create_database", + "notion.notion_append_block", + "notion.notion_search", + "notion.notion_get_comments", + "mongodb.mongo_delete", + "mongodb.mongo_find", + "mongodb.mongo_update", + "mongodb.mongo_list_collections", + "postgres.list_schemas" + ], + "hit1": "notion.notion_query_database", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 263, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 387, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 711, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_search", + "zendesk.zendesk_list_tickets", + "id_map.json", + "jira.jira_create_issue" + ], + "hit1": "zendesk.zendesk_get_ticket", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 642, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_search_contacts", + "hubspot.hubspot_create_deal", + "id_map.json", + "github.delete_file", + "github.pull_request_review_write", + "github.get_discussion_comments", + "github.get_discussion" + ], + "hit1": "hubspot.hubspot_create_contact", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 801, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "kind": "verb_product", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_create_lead", + "salesforce.salesforce_get_record", + "salesforce.salesforce_convert_lead", + "salesforce.salesforce_update_record", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_list_opportunities", + "id_map.json" + ], + "hit1": "salesforce.salesforce_create_lead", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 423, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 634, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "kind": "identifier", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_query" + ], + "hit1": "salesforce.salesforce_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 173, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "kind": "verb_product", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "ranking": [ + "shopify.shopify_list_orders", + "shopify.shopify_list_products", + "shopify.shopify_update_product", + "shopify.shopify_create_product", + "shopify.shopify_get_product", + "id_map.json" + ], + "hit1": "shopify.shopify_list_orders", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 376, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 626, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "pagerduty.pagerduty_create_incident" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.16666666666666666, + "ndcg@10": 0.3562071871080222, + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 486, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 736, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 715, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "kind": "paraphrase", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_list_oncalls", + "filesystem.list_allowed_directories", + "twilio.twilio_make_call", + "github.get_discussion_comments", + "slack.slack_create_channel", + "notion.notion_get_comments", + "figma.figma_post_comment", + "figma.figma_get_comments", + "id_map.json", + "github.create_pull_request" + ], + "hit1": "pagerduty.pagerduty_list_oncalls", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 712, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "kind": "verb_product", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "ranking": [ + "datadog.datadog_query_metrics", + "datadog.datadog_create_monitor", + "prometheus.prom_list_metrics", + "datadog.datadog_list_logs", + "datadog.datadog_list_monitors", + "datadog.datadog_get_monitor", + "id_map.json" + ], + "hit1": "datadog.datadog_query_metrics", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 287, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 449, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "kind": "verb_product", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "ranking": [ + "datadog.datadog_list_logs", + "datadog.datadog_query_metrics", + "github.actions_get_job_logs", + "datadog.datadog_create_monitor", + "kubernetes.k8s_pod_logs", + "datadog.datadog_list_monitors", + "docker.docker_logs", + "datadog.datadog_get_monitor", + "git.git_log" + ], + "hit1": "datadog.datadog_list_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 315, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 528, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 828, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_list_buckets", + "id_map.json" + ], + "hit1": "aws_s3.s3_list_buckets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 115, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 220, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 642, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "kind": "paraphrase", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_put_object", + "slack.slack_upload_file", + "github.add_comment_to_pending_review", + "aws_s3.s3_delete_object", + "slack.slack_reply_to_thread", + "aws_s3.s3_get_object", + "github.assign_copilot_to_issue", + "slack.slack_add_reaction", + "id_map.json", + "asana.asana_get_task" + ], + "hit1": "aws_s3.s3_put_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 346, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 505, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 822, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "id_map.json" + ], + "hit1": "cloudflare.cf_purge_cache", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 286, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 643, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "cloudflare.cf_list_zones", + "salesforce.salesforce_get_record", + "cloudflare.cf_purge_cache", + "salesforce.salesforce_update_record", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_create_record", + "id_map.json", + "git.git_commit" + ], + "hit1": "cloudflare.cf_create_dns", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_deployments", + "vercel.vercel_list_projects", + "kubernetes.k8s_list_deployments", + "vercel.vercel_get_deployment", + "vercel.vercel_create_project", + "id_map.json" + ], + "hit1": "vercel.vercel_list_deployments", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 173, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 541, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "ranking": [ + "supabase.supabase_insert", + "mongodb.mongo_insert", + "supabase.supabase_update", + "supabase.supabase_select", + "supabase.supabase_list_tables", + "supabase.supabase_invoke_fn", + "playwright.browser_type", + "id_map.json", + "playwright.browser_click", + "playwright.browser_evaluate" + ], + "hit1": "supabase.supabase_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 620, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "kind": "paraphrase", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "ranking": [ + "figma.figma_get_images", + "docker.docker_images", + "figma.figma_post_comment", + "figma.figma_get_comments", + "figma.figma_get_file", + "figma.figma_get_file_nodes", + "github.delete_file", + "mongodb.mongo_delete", + "datadog.datadog_query_metrics", + "github.get_file_contents" + ], + "hit1": "figma.figma_get_images", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 325, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "kind": "verb_product", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "ranking": [ + "twilio.twilio_send_sms", + "twilio.twilio_list_messages", + "id_map.json", + "filesystem.list_directory_with_sizes", + "twilio.twilio_make_call", + "puppeteer.puppeteer_click", + "asana.asana_get_task", + "asana.asana_update_task", + "asana.asana_create_task", + "github.update_pull_request_branch" + ], + "hit1": "twilio.twilio_send_sms", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 161, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 308, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "kind": "verb_product", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "ranking": [ + "okta.okta_deactivate_user", + "okta.okta_create_user", + "okta.okta_get_user", + "slack.slack_get_user_profile", + "okta.okta_list_users", + "linear.get_user", + "asana.asana_get_task", + "asana.asana_update_task", + "asana.asana_create_task", + "asana.asana_create_project" + ], + "hit1": "okta.okta_deactivate_user", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 289, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 657, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query", + "prometheus.prom_list_metrics", + "prometheus.prom_query_range", + "id_map.json" + ], + "hit1": "prometheus.prom_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 275, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 681, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "kind": "verb_product", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "ranking": [ + "grafana.grafana_search_dashboards", + "grafana.grafana_create_annotation", + "grafana.grafana_get_dashboard", + "id_map.json" + ], + "hit1": "grafana.grafana_search_dashboards", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 271, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 677, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "kind": "verb_product", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "ranking": [ + "circleci.circleci_trigger_pipeline", + "circleci.circleci_get_pipeline", + "circleci.circleci_list_pipelines", + "gitlab.gitlab_create_pipeline", + "id_map.json", + "github.actions_run_trigger", + "gitlab.get_pipeline", + "gitlab.create_pipeline" + ], + "hit1": "circleci.circleci_trigger_pipeline", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 271, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 634, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "kind": "identifier", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "ranking": [ + "terraform.tf_plan", + "terraform.tf_apply", + "terraform.tf_state_list", + "id_map.json", + "postgres.explain_query" + ], + "hit1": "terraform.tf_plan", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 227, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 584, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "kind": "verb_product", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "ranking": [ + "asana.asana_get_task", + "asana.asana_update_task", + "asana.asana_create_task", + "asana.asana_create_project", + "asana.asana_list_tasks", + "clickup.clickup_update_task", + "clickup.clickup_get_task", + "clickup.clickup_create_task", + "slack.slack_add_reaction", + "gmail.gmail_send_email" + ], + "hit1": "asana.asana_get_task", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 824, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "ranking": [ + "trello.trello_list_cards", + "trello.trello_create_card", + "trello.trello_add_comment", + "trello.trello_create_board", + "id_map.json" + ], + "hit1": "trello.trello_list_cards", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_update_task", + "clickup.clickup_create_task", + "clickup.clickup_get_task", + "clickup.clickup_list_tasks", + "asana.asana_get_task", + "asana.asana_update_task", + "playwright.browser_click", + "puppeteer.puppeteer_click" + ], + "hit1": "clickup.clickup_update_task", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "kind": "verb_product", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "ranking": [ + "confluence.confluence_get_page", + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "notion.notion_update_page", + "notion.notion_get_page", + "confluence.confluence_search" + ], + "hit1": "confluence.confluence_get_page", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "kind": "verb_product", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "ranking": [ + "discord.discord_add_reaction", + "discord.discord_send_message", + "slack.slack_post_message", + "discord.discord_get_channel", + "discord.discord_list_channels", + "discord.discord_list_members", + "slack.slack_add_reaction", + "github.delete_file", + "figma.figma_post_comment", + "gmail.gmail_modify_email" + ], + "hit1": "discord.discord_add_reaction", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_get_doc", + "elasticsearch.es_index_doc", + "elasticsearch.es_search", + "id_map.json" + ], + "hit1": "elasticsearch.es_get_doc", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 293, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "redis.redis_set", + "redis.redis_keys", + "playwright.browser_press_key", + "jira.jira_get_issue", + "redis.redis_delete", + "figma.figma_post_comment", + "figma.figma_get_comments" + ], + "hit1": "redis.redis_get", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 288, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 553, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert", + "mongodb.mongo_delete", + "mongodb.mongo_find", + "mongodb.mongo_update", + "mongodb.mongo_list_collections" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "kind": "verb_product", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "ranking": [ + "sqlite.sqlite_list_tables", + "sqlite.sqlite_describe_table", + "sqlite.sqlite_query", + "mysql.mysql_list_tables", + "postgres.list_tables", + "supabase.supabase_list_tables", + "id_map.json" + ], + "hit1": "sqlite.sqlite_list_tables", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 156, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 252, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 451, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "kind": "verb_product", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "ranking": [ + "postgres.describe_table", + "sqlite.sqlite_describe_table", + "postgres.list_schemas", + "postgres.list_tables", + "supabase.supabase_insert", + "postgres.query", + "supabase.supabase_update", + "postgres.explain_query", + "supabase.supabase_select", + "id_map.json" + ], + "hit1": "postgres.describe_table", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 170, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 286, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 543, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.list_check_runs", + "jira.jira_get_issue", + "github.delete_file", + "github.issue_write", + "github.pull_request_review_write", + "gitlab.gitlab_get_issue", + "github.delete_check_run", + "sentry.sentry_update_issue", + "github.get_discussion_comments", + "sentry.sentry_get_issue" + ], + "hit1": "github.list_check_runs", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.16369747519436975, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 634, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 542, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 920, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "ranking": [ + "jira.jira_get_issue", + "github.issue_write", + "gitlab.gitlab_get_issue", + "sentry.sentry_update_issue", + "sentry.sentry_get_issue" + ], + "hit1": "jira.jira_get_issue", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 34, + "k3_calls": 12, + "k3_tokens": 3256, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 7, + "k5_tokens": 3080, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3574, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q071", + "query": "list-directory", + "kind": "identifier", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "ranking": [ + "filesystem.create_directory", + "filesystem.list_directory", + "filesystem.directory_tree", + "filesystem.list_directory_with_sizes", + "github.get_file_contents", + "filesystem.get_file_info" + ], + "hit1": "filesystem.create_directory", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 653, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "kind": "paraphrase", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "ranking": [ + "sequential_thinking.sequentialthinking", + "id_map.json" + ], + "hit1": "sequential_thinking.sequentialthinking", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "kind": "paraphrase", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "ranking": [ + "github.create_hook", + "gitlab.list_pipelines", + "github.update_hook", + "time.convert_time", + "filesystem.list_allowed_directories", + "time.get_current_time", + "playwright.browser_wait_for", + "github.get_team_members", + "slack.slack_create_channel", + "jira.jira_add_worklog" + ], + "hit1": "github.create_hook", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.4344817381527212, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 452, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "kind": "verb_product", + "gold": [ + "time.convert_time" + ], + "alts": [], + "ranking": [ + "time.convert_time", + "salesforce.salesforce_convert_lead", + "time.get_current_time", + "playwright.browser_wait_for", + "git.git_diff", + "memory.create_relations", + "id_map.json", + "jira.jira_add_worklog", + "google_calendar.gcal_list_events", + "grafana.grafana_create_annotation" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 650, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "kind": "verb_product", + "gold": [ + "github.star_repository" + ], + "alts": [], + "ranking": [ + "github.star_repository", + "git.git_show", + "github.unstar_repository", + "git.git_branch", + "github.delete_file", + "github.pull_request_review_write", + "git.git_diff_unstaged", + "github.get_discussion_comments", + "git.git_status", + "git.git_diff_staged" + ], + "hit1": "github.star_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 220, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 773, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "kind": "resource", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "ranking": [ + "github.fork_repository", + "github.star_repository", + "github.unstar_repository", + "github.create_repository", + "github.get_repository_tree", + "id_map.json" + ], + "hit1": "github.fork_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 425, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "kind": "verb_product", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "ranking": [ + "github.get_latest_release", + "git.git_show", + "git.git_branch", + "git.git_diff_unstaged", + "git.git_status", + "git.git_diff_staged", + "playwright.browser_navigate", + "github.delete_file", + "memory.delete_entities", + "github.pull_request_review_write" + ], + "hit1": "github.get_latest_release", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 306, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "kind": "resource", + "gold": [ + "github.create_gist" + ], + "alts": [], + "ranking": [ + "github.create_gist", + "github.get_gist", + "github.list_gists", + "github.update_gist", + "id_map.json" + ], + "hit1": "github.create_gist", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 637, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "kind": "verb_product", + "gold": [ + "github.search_code" + ], + "alts": [], + "ranking": [ + "git.git_show", + "github.get_code_scanning_alert", + "git.git_branch", + "github.list_code_scanning_alerts", + "git.git_diff_unstaged", + "github.search_code", + "git.git_status", + "git.git_diff_staged", + "id_map.json", + "github.delete_file" + ], + "hit1": "git.git_show", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.16666666666666666, + "ndcg@10": 0.3562071871080222, + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 464, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 681, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 661, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "kind": "verb_product", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "ranking": [ + "git.git_show", + "github.list_notifications", + "git.git_branch", + "github.mark_all_notifications_read", + "git.git_diff_unstaged", + "git.git_status", + "git.git_diff_staged", + "github.delete_file", + "github.pull_request_review_write", + "github.get_discussion_comments" + ], + "hit1": "git.git_show", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 284, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "kind": "verb_product", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "ranking": [ + "github.assign_copilot_to_issue", + "github.add_comment_to_pending_review", + "jira.jira_assign_issue", + "github.request_copilot_review", + "slack.slack_reply_to_thread", + "asana.asana_create_task", + "linear.linear_assign_issue", + "slack.slack_add_reaction", + "id_map.json", + "asana.asana_get_task" + ], + "hit1": "github.assign_copilot_to_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 347, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 537, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 858, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "kind": "resource", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "ranking": [ + "github.list_secret_scanning_alerts", + "github.get_secret_scanning_alert", + "github.list_code_scanning_alerts", + "id_map.json", + "github.list_dependabot_alerts", + "github.get_code_scanning_alert" + ], + "hit1": "github.list_secret_scanning_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 720, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "ranking": [ + "github.create_or_update_file", + "google_drive.gdrive_get_file", + "git.git_show", + "google_drive.gdrive_update_file", + "git.git_branch", + "gitlab.gitlab_get_file", + "git.git_diff_unstaged", + "figma.figma_get_file", + "git.git_status", + "gmail.gmail_modify_email" + ], + "hit1": "github.create_or_update_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 286, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 423, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 768, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "kind": "resource", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "ranking": [ + "github.list_repository_collaborators", + "github.get_discussion_comments", + "github.star_repository", + "pagerduty.pagerduty_list_oncalls", + "github.unstar_repository", + "notion.notion_get_comments", + "github.get_repository_tree", + "figma.figma_post_comment", + "github.fork_repository", + "figma.figma_get_comments" + ], + "hit1": "github.list_repository_collaborators", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 392, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "kind": "verb_product", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "ranking": [ + "slack.slack_reply_to_thread", + "slack.slack_search_messages", + "slack.slack_get_thread_replies", + "slack.slack_add_reaction", + "slack.slack_list_users", + "github.get_notification_details", + "slack.slack_post_message", + "id_map.json", + "jira.update_sprint", + "linear.list_users" + ], + "hit1": "slack.slack_reply_to_thread", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 411, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "kind": "verb_product", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "ranking": [ + "slack.slack_add_reaction", + "discord.discord_add_reaction", + "asana.asana_get_task", + "github.get_discussion_comments", + "asana.asana_update_task", + "pagerduty.pagerduty_list_oncalls", + "asana.asana_create_task", + "notion.notion_get_comments", + "asana.asana_create_project", + "figma.figma_post_comment" + ], + "hit1": "slack.slack_add_reaction", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 434, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 800, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "kind": "verb_product", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "ranking": [ + "slack.slack_search_messages", + "slack.slack_add_reaction", + "twilio.twilio_list_messages", + "slack.slack_list_users", + "slack.slack_reply_to_thread", + "slack.slack_post_message", + "gmail.gmail_search_emails", + "slack.slack_get_channel_history", + "id_map.json" + ], + "hit1": "slack.slack_search_messages", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 652, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "kind": "verb_product", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "ranking": [ + "slack.slack_search_messages", + "discord.discord_get_channel", + "slack.slack_add_reaction", + "slack.slack_set_channel_topic", + "slack.slack_list_users", + "slack.slack_get_channel_history", + "slack.slack_reply_to_thread", + "slack.slack_leave_channel", + "slack.slack_post_message", + "slack.slack_join_channel" + ], + "hit1": "slack.slack_search_messages", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 135, + "k3_calls": 45, + "k3_tokens": 11143, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 27, + "k5_tokens": 10783, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 14, + "k10_tokens": 10993, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "kind": "verb_product", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_get_issue", + "jira.jira_transition_issue", + "jira.jira_search_issues", + "github.issue_write", + "jira.jira_list_projects", + "gitlab.gitlab_get_issue", + "jira.jira_add_worklog", + "sentry.sentry_update_issue", + "jira.jira_add_comment", + "sentry.sentry_get_issue" + ], + "hit1": "jira.jira_get_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 383, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 772, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "kind": "identifier", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "ranking": [ + "jira.jira_search_issues", + "filesystem.list_directory_with_sizes", + "jira.jira_get_issue", + "jira.jira_list_projects", + "jira.jira_add_worklog", + "jira.jira_add_comment", + "github.update_pull_request_branch", + "puppeteer.puppeteer_click", + "puppeteer.puppeteer_evaluate", + "puppeteer.puppeteer_fill" + ], + "hit1": "jira.jira_search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 670, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "kind": "verb_product", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "ranking": [ + "jira.jira_add_worklog", + "github.add_comment_to_pending_review", + "jira.jira_search_issues", + "slack.slack_reply_to_thread", + "jira.jira_get_issue", + "github.assign_copilot_to_issue", + "jira.jira_list_projects", + "jira.jira_add_comment", + "filesystem.list_allowed_directories", + "slack.slack_add_reaction" + ], + "hit1": "jira.jira_add_worklog", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 338, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 479, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 815, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "kind": "verb_product", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "ranking": [ + "linear.linear_create_project", + "github.get_teams", + "linear.linear_get_issue", + "linear.linear_list_teams", + "linear.linear_update_issue", + "linear.linear_create_issue", + "linear.linear_create_comment", + "id_map.json" + ], + "hit1": "linear.linear_create_project", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 494, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "github.update_issue_comment", + "linear.linear_create_project", + "jira.jira_get_issue", + "notion.create_comment", + "linear.linear_get_issue", + "github.issue_write", + "linear.linear_update_issue", + "gitlab.gitlab_get_issue", + "linear.create_comment" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 147, + "k3_calls": 49, + "k3_tokens": 12287, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 30, + "k5_tokens": 12042, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11727, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q094", + "query": "Move a local file", + "kind": "paraphrase", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "ranking": [ + "filesystem.move_file", + "brave_search.brave_local_search", + "google_drive.gdrive_get_file", + "google_drive.gdrive_update_file", + "gitlab.gitlab_get_file", + "gmail.gmail_modify_email", + "figma.figma_get_file", + "google_drive.gdrive_share_file", + "docker.docker_rm", + "filesystem.read_file" + ], + "hit1": "filesystem.move_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 395, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "kind": "paraphrase", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "ranking": [ + "filesystem.search_files", + "github.push_files", + "github.get_release_by_tag", + "redis.redis_keys", + "slack.list_files", + "salesforce.salesforce_get_record", + "jira.jira_get_issue", + "linear.linear_get_issue", + "github.search_repositories", + "filesystem.list_directory" + ], + "hit1": "filesystem.search_files", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 419, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 748, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "kind": "paraphrase", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "ranking": [ + "filesystem.edit_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_update_file", + "gitlab.gitlab_get_file", + "figma.figma_get_file", + "google_drive.gdrive_share_file", + "jira.update_sprint", + "brave_search.brave_local_search", + "linear.list_users", + "twilio.twilio_make_call" + ], + "hit1": "filesystem.edit_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 265, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 410, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 801, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "ranking": [ + "memory.read_graph", + "memory.create_relations", + "memory.delete_entities", + "memory.search_nodes", + "memory.add_observations", + "memory.create_entities", + "memory.delete_relations", + "memory.open_nodes", + "playwright.browser_navigate", + "pagerduty.pagerduty_acknowledge_incident" + ], + "hit1": "memory.read_graph", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 158, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 271, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 539, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "kind": "verb_product", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "memory.delete_entities", + "figma.figma_get_file_nodes", + "memory.create_relations", + "memory.open_nodes", + "memory.delete_observations", + "memory.add_observations", + "id_map.json" + ], + "hit1": "memory.search_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 305, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 507, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "kind": "verb_product", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "ranking": [ + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_show", + "git.git_diff", + "git.git_branch", + "id_map.json", + "git.git_status" + ], + "hit1": "git.git_diff_unstaged", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 187, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 297, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 508, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "kind": "identifier", + "gold": [ + "git.git_add" + ], + "alts": [], + "ranking": [ + "git.git_diff_unstaged", + "git.git_diff_staged", + "filesystem.list_directory_with_sizes", + "github.push_files", + "git.git_show", + "filesystem.search_files", + "git.git_branch", + "slack.list_files", + "git.git_status", + "id_map.json" + ], + "hit1": "git.git_diff_unstaged", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 113, + "k3_calls": 38, + "k3_tokens": 9484, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 23, + "k5_tokens": 9231, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9385, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "kind": "verb_product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "ranking": [ + "shopify.shopify_update_product", + "shopify.shopify_create_product", + "shopify.shopify_get_product", + "shopify.shopify_list_products", + "shopify.shopify_list_orders", + "stripe.stripe_create_product", + "stripe.stripe_create_price" + ], + "hit1": "shopify.shopify_update_product", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "kind": "verb_product", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "ranking": [ + "stripe.stripe_cancel_subscription", + "stripe.stripe_list_subscriptions", + "stripe.stripe_list_customers", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_price", + "stripe.stripe_create_invoice", + "stripe.stripe_create_product", + "id_map.json" + ], + "hit1": "stripe.stripe_cancel_subscription", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 546, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "kind": "resource", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_refund", + "id_map.json" + ], + "hit1": "stripe.stripe_create_payment_intent", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 288, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "pagerduty.pagerduty_create_incident", + "id_map.json" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 626, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "kind": "verb_product", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "ranking": [ + "datadog.datadog_create_monitor", + "datadog.datadog_list_monitors", + "datadog.datadog_get_monitor", + "datadog.datadog_query_metrics", + "datadog.datadog_list_logs", + "id_map.json" + ], + "hit1": "datadog.datadog_create_monitor", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "kind": "paraphrase", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "ranking": [ + "sentry.sentry_update_issue", + "sentry.sentry_get_issue", + "pagerduty.pagerduty_resolve_incident", + "sentry.sentry_list_projects", + "jira.jira_get_issue", + "sentry.sentry_list_issues", + "github.issue_write", + "gitlab.gitlab_get_issue", + "sentry.sentry_list_events", + "id_map.json" + ], + "hit1": "sentry.sentry_update_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_list_deployments", + "id_map.json", + "kubernetes.k8s_delete", + "vercel.vercel_get_deployment", + "kubernetes.k8s_pod_logs", + "vercel.vercel_list_deployments", + "kubernetes.k8s_list_services" + ], + "hit1": "kubernetes.k8s_scale_deployment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 174, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 622, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_apply", + "kubernetes.k8s_scale_deployment", + "terraform.tf_apply", + "kubernetes.k8s_delete", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_list_services", + "id_map.json" + ], + "hit1": "kubernetes.k8s_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 246, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "ranking": [ + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_run", + "docker.docker_build", + "docker.docker_exec", + "docker.docker_logs", + "docker.docker_ps", + "kubernetes.k8s_pod_logs", + "id_map.json" + ], + "hit1": "docker.docker_stop", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 608, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "kind": "verb_product", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "ranking": [ + "docker.docker_build", + "docker.docker_run", + "docker.docker_rm", + "docker.docker_stop", + "docker.docker_ps", + "docker.docker_images", + "figma.figma_get_images", + "id_map.json", + "filesystem.read_media_file" + ], + "hit1": "docker.docker_build", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 563, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "kind": "verb_product", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "ranking": [ + "playwright.browser_click", + "puppeteer.puppeteer_click", + "clickup.clickup_create_task", + "clickup.clickup_list_tasks", + "clickup.clickup_update_task", + "playwright.browser_type", + "asana.asana_get_task", + "playwright.browser_take_screenshot", + "jira.update_sprint", + "asana.asana_update_task" + ], + "hit1": "playwright.browser_click", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 374, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 734, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "kind": "paraphrase", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "ranking": [ + "playwright.browser_type", + "playwright.browser_wait_for", + "filesystem.read_text_file", + "playwright.browser_tab_new", + "playwright.browser_navigate", + "playwright.browser_tab_list", + "playwright.browser_tab_close", + "mongodb.mongo_insert", + "cloudflare.cf_create_dns", + "supabase.supabase_insert" + ], + "hit1": "playwright.browser_type", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_list_projects", + "git.git_show", + "vercel.vercel_list_projects", + "git.git_branch", + "sentry.sentry_list_projects", + "git.git_diff_unstaged", + "jira.jira_list_projects", + "git.git_status", + "github.projects_list", + "git.git_diff_staged" + ], + "hit1": "gitlab.gitlab_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 170, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 267, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 524, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "kind": "verb_product", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "ranking": [ + "github.pull_request_review_write", + "bitbucket.bitbucket_list_prs", + "gitlab.gitlab_get_merge_request", + "bitbucket.bitbucket_create_issue", + "github.update_pull_request", + "gitlab.gitlab_merge_merge_request", + "bitbucket.bitbucket_list_repos", + "github.pull_request_read", + "bitbucket.bitbucket_create_pr", + "github.update_pull_request_branch" + ], + "hit1": "github.pull_request_review_write", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.1111111111111111, + "ndcg@10": 0.3010299956639812, + "hit_rank": 9, + "k3_calls": 3, + "k3_tokens": 943, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 1016, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 994, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "kind": "verb_product", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "ranking": [ + "gmail.gmail_send_email", + "gmail.gmail_create_draft", + "gmail.gmail_modify_email", + "gmail.gmail_trash_email", + "gmail.gmail_get_email", + "gmail.gmail_list_labels", + "github.create_pull_request", + "id_map.json" + ], + "hit1": "gmail.gmail_send_email", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 288, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 398, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "kind": "verb_product", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_get_file", + "google_drive.gdrive_search", + "google_drive.gdrive_create_file", + "google_drive.gdrive_share_file", + "google_drive.gdrive_update_file", + "google_calendar.gcal_delete_event", + "gitlab.gitlab_get_file", + "google_calendar.gcal_update_event", + "google_drive.gdrive_list_folder", + "figma.figma_get_file" + ], + "hit1": "google_drive.gdrive_get_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 510, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 778, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "kind": "verb_product", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_update_event", + "google_drive.gdrive_search", + "kubernetes.k8s_get_events", + "google_drive.gdrive_create_file", + "google_calendar.gcal_create_event", + "sentry.sentry_list_events", + "google_drive.gdrive_get_file", + "google_calendar.gcal_list_calendars" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 376, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 770, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "id_map.json", + "kubernetes.k8s_get_events", + "github.pull_request_review_write", + "sentry.sentry_list_events" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 748, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_projects", + "vercel.vercel_get_deployment", + "sentry.sentry_list_projects", + "vercel.vercel_list_deployments", + "jira.jira_list_projects", + "vercel.vercel_create_project", + "github.projects_list", + "gitlab.gitlab_list_projects" + ], + "hit1": "vercel.vercel_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 170, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 529, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "ranking": [ + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update", + "supabase.supabase_list_tables", + "supabase.supabase_invoke_fn", + "github.delete_file", + "mongodb.mongo_delete", + "puppeteer.puppeteer_click", + "datadog.datadog_query_metrics", + "puppeteer.puppeteer_fill" + ], + "hit1": "supabase.supabase_select", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 766, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "kind": "verb_product", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "github.update_issue_comment", + "figma.figma_get_comments", + "notion.create_comment", + "figma.figma_get_file", + "figma.figma_get_images", + "linear.create_comment", + "figma.figma_get_file_nodes", + "github.add_comment_to_pending_review" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 406, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 873, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "kind": "verb_product", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "ranking": [ + "okta.okta_create_user", + "slack.slack_list_users", + "okta.okta_deactivate_user", + "linear.list_users", + "okta.okta_list_users", + "notion.list_users_tokens", + "okta.okta_get_user", + "notion.delete_users_token", + "notion.get_users_token" + ], + "hit1": "okta.okta_create_user", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.2, + "ndcg@10": 0.38685280723454163, + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 417, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 617, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query_range", + "prometheus.prom_list_metrics", + "prometheus.prom_query", + "id_map.json" + ], + "hit1": "prometheus.prom_query_range", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 275, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 681, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "kind": "verb_product", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "ranking": [ + "grafana.grafana_create_annotation", + "grafana.grafana_get_dashboard", + "grafana.grafana_search_dashboards", + "id_map.json" + ], + "hit1": "grafana.grafana_create_annotation", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 272, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 678, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "kind": "identifier", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "ranking": [ + "terraform.tf_apply", + "id_map.json", + "kubernetes.k8s_apply", + "terraform.tf_plan", + "terraform.tf_state_list" + ], + "hit1": "terraform.tf_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 146, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 238, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 595, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "kind": "verb_product", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "ranking": [ + "asana.asana_list_tasks", + "asana.asana_get_task", + "clickup.clickup_list_tasks", + "asana.asana_update_task", + "asana.asana_create_task", + "asana.asana_create_project", + "id_map.json" + ], + "hit1": "asana.asana_list_tasks", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 616, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "ranking": [ + "trello.trello_list_cards", + "trello.trello_add_comment", + "trello.trello_create_card", + "figma.figma_post_comment", + "github.update_issue_comment", + "trello.trello_create_board", + "notion.create_comment", + "linear.create_comment", + "github.add_comment_to_pending_review", + "github.get_discussion_comments" + ], + "hit1": "trello.trello_list_cards", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 438, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 934, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_update_task", + "clickup.clickup_create_task", + "clickup.clickup_get_task", + "clickup.clickup_list_tasks", + "asana.asana_get_task", + "asana.asana_update_task", + "playwright.browser_click", + "puppeteer.puppeteer_click" + ], + "hit1": "clickup.clickup_update_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "kind": "verb_product", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "ranking": [ + "confluence.confluence_get_page", + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "confluence.confluence_search" + ], + "hit1": "confluence.confluence_get_page", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 423, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 772, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_deal", + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_search_contacts", + "github.delete_file", + "github.pull_request_review_write", + "github.get_discussion_comments", + "github.get_discussion", + "github.list_discussions" + ], + "hit1": "hubspot.hubspot_create_deal", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 873, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_search", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_create_ticket", + "id_map.json" + ], + "hit1": "zendesk.zendesk_list_tickets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 617, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "kind": "verb_product", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "ranking": [ + "discord.discord_list_channels", + "discord.discord_add_reaction", + "discord.discord_get_channel", + "slack.slack_list_channels", + "discord.discord_send_message", + "discord.discord_list_members", + "id_map.json" + ], + "hit1": "discord.discord_list_channels", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 547, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "redis.redis_set", + "redis.redis_keys", + "playwright.browser_press_key", + "jira.jira_get_issue", + "redis.redis_delete", + "figma.figma_post_comment", + "figma.figma_get_comments" + ], + "hit1": "redis.redis_get", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 288, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 553, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert", + "mongodb.mongo_delete", + "mongodb.mongo_find", + "mongodb.mongo_update", + "mongodb.mongo_list_collections", + "supabase.supabase_insert", + "id_map.json" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 607, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "kind": "verb_product", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "ranking": [ + "mysql.mysql_query", + "mysql.mysql_list_tables", + "sqlite.sqlite_query", + "postgres.query", + "postgres.explain_query", + "postgres.list_schemas" + ], + "hit1": "mysql.mysql_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 173, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 568, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "elasticsearch.es_search", + "mongodb.mongo_find", + "id_map.json", + "mongodb.mongo_insert", + "jira.update_sprint", + "mongodb.mongo_delete", + "linear.list_users", + "gitlab.get_pipeline" + ], + "hit1": "elasticsearch.es_index_doc", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 318, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 661, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_delete_object", + "aws_s3.s3_get_object", + "aws_s3.s3_put_object", + "id_map.json", + "asana.asana_get_task", + "aws_s3.s3_list_objects", + "asana.asana_update_task", + "asana.asana_create_task", + "asana.asana_create_project", + "slack.slack_add_reaction" + ], + "hit1": "aws_s3.s3_delete_object", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones", + "cloudflare.cf_purge_cache", + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "time.convert_time", + "id_map.json" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 613, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_list_customers", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_price", + "stripe.stripe_create_product", + "id_map.json" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 547, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "kind": "verb_product", + "gold": [ + "github.actions_list" + ], + "alts": [], + "ranking": [ + "github.actions_list", + "git.git_show", + "github.actions_run_trigger", + "git.git_branch", + "github.actions_get_job_logs", + "git.git_diff_unstaged", + "git.git_status", + "github.actions_get", + "git.git_diff_staged", + "github.delete_file" + ], + "hit1": "github.actions_list", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 408, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "kind": "verb_product", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "ranking": [ + "github.actions_get_job_logs", + "git.git_show", + "github.actions_run_trigger", + "git.git_branch", + "gitlab.delete_job", + "git.git_diff_unstaged", + "github.actions_list", + "gitlab.update_job", + "git.git_status", + "github.actions_get" + ], + "hit1": "github.actions_get_job_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 296, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 723, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "kind": "verb_product", + "gold": [ + "github.create_repository" + ], + "alts": [], + "ranking": [ + "github.star_repository", + "git.git_show", + "git.git_branch", + "github.unstar_repository", + "git.git_diff_unstaged", + "github.create_repository", + "git.git_status", + "github.get_repository_tree", + "git.git_diff_staged", + "github.fork_repository" + ], + "hit1": "github.star_repository", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.16666666666666666, + "ndcg@10": 0.3562071871080222, + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 421, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 712, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 691, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "kind": "verb_product", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "ranking": [ + "git.git_branch", + "git.git_checkout", + "git.git_show", + "git.git_create_branch", + "github.create_branch", + "git.git_diff_unstaged", + "github.update_pull_request_branch", + "git.git_status", + "git.git_diff_staged", + "github.delete_file" + ], + "hit1": "git.git_branch", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 739, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "kind": "identifier", + "gold": [ + "git.git_log" + ], + "alts": [], + "ranking": [ + "git.git_show", + "git.git_branch", + "git.git_diff_unstaged", + "git.git_status", + "git.git_log", + "git.git_diff_staged", + "datadog.datadog_list_logs", + "docker.docker_logs", + "jira.jira_add_worklog", + "kubernetes.k8s_pod_logs" + ], + "hit1": "git.git_show", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.2, + "ndcg@10": 0.38685280723454163, + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 364, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "kind": "verb_product", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "ranking": [ + "postgres.list_schemas", + "postgres.describe_table", + "postgres.list_tables", + "postgres.query", + "postgres.explain_query", + "id_map.json" + ], + "hit1": "postgres.list_schemas", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 169, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 527, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "kind": "verb_product", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "memory.open_nodes", + "figma.figma_get_file_nodes", + "memory.delete_entities", + "memory.create_relations", + "id_map.json", + "jira.update_sprint", + "playwright.browser_navigate", + "pagerduty.pagerduty_acknowledge_incident", + "linear.list_users" + ], + "hit1": "memory.search_nodes", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 308, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 570, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "kind": "paraphrase", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_get_file", + "memory.delete_entities", + "google_drive.gdrive_update_file", + "gitlab.gitlab_get_file", + "figma.figma_get_file", + "filesystem.read_media_file", + "google_drive.gdrive_share_file", + "filesystem.get_file_info", + "github.delete_file", + "playwright.browser_navigate" + ], + "hit1": "google_drive.gdrive_get_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.125, + "ndcg@10": 0.31546487678572877, + "hit_rank": 8, + "k3_calls": 3, + "k3_tokens": 723, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 752, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 730, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "kind": "paraphrase", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "ranking": [ + "playwright.browser_tab_new", + "brave_search.brave_local_search", + "google_drive.gdrive_get_file", + "filesystem.edit_file", + "google_drive.gdrive_update_file", + "gitlab.gitlab_get_file", + "figma.figma_get_file", + "google_drive.gdrive_share_file", + "filesystem.read_file", + "filesystem.read_media_file" + ], + "hit1": "playwright.browser_tab_new", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 87, + "k3_calls": 29, + "k3_tokens": 7985, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7922, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7733, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "kind": "verb_product", + "gold": [ + "github.search_issues" + ], + "alts": [], + "ranking": [ + "git.git_show", + "jira.jira_search_issues", + "git.git_branch", + "github.list_issues", + "git.git_diff_unstaged", + "sentry.sentry_list_issues", + "git.git_status", + "github.search_issues", + "git.git_diff_staged", + "linear.linear_list_issues" + ], + "hit1": "git.git_show", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.125, + "ndcg@10": 0.31546487678572877, + "hit_rank": 8, + "k3_calls": 3, + "k3_tokens": 679, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 744, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "kind": "paraphrase", + "gold": [ + "github.get_me" + ], + "alts": [], + "ranking": [ + "git.git_show", + "git.git_branch", + "github.list_notifications", + "git.git_diff_unstaged", + "git.git_status", + "git.git_diff_staged", + "playwright.browser_navigate", + "github.get_me", + "github.delete_file", + "memory.delete_entities" + ], + "hit1": "git.git_show", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.125, + "ndcg@10": 0.31546487678572877, + "hit_rank": 8, + "k3_calls": 3, + "k3_tokens": 614, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 644, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 622, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "cost": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 4.06, + "p95_calls": 30.099999999999966, + "success_within_1": 0.8066666666666666, + "success_within_2": 0.8733333333333333, + "success_within_3": 0.9066666666666666, + "mean_tokens": 1007.1266666666667, + "p95_tokens": 8226.999999999993, + "total_tokens": 151069, + "tokens_per_successful_query": 1007.1266666666667, + "total_tokens_successful": 151069 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 2.78, + "p95_calls": 18.549999999999983, + "success_within_1": 0.8533333333333334, + "success_within_2": 0.9266666666666666, + "success_within_3": 0.9266666666666666, + "mean_tokens": 1088.7933333333333, + "p95_tokens": 8117.249999999994, + "total_tokens": 163319, + "tokens_per_successful_query": 1088.7933333333333, + "total_tokens_successful": 163319 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.84, + "p95_calls": 9.549999999999983, + "success_within_1": 0.9266666666666666, + "success_within_2": 0.9266666666666666, + "success_within_3": 0.9266666666666666, + "mean_tokens": 1371.1666666666667, + "p95_tokens": 8033.29999999999, + "total_tokens": 205675, + "tokens_per_successful_query": 1371.1666666666667, + "total_tokens_successful": 205675 + } + } +} diff --git a/experiments/mcp-tool-search/results/codedb_word.csv b/experiments/mcp-tool-search/results/codedb_word.csv new file mode 100644 index 00000000..3689a0c3 --- /dev/null +++ b/experiments/mcp-tool-search/results/codedb_word.csv @@ -0,0 +1,151 @@ +id,query,kind,hit1,recall@1,recall@3,recall@5,mrr@10,ndcg@10,hit_rank,k3_calls,k3_tokens,k3_success1,k5_calls,k5_tokens,k5_success1,k10_calls,k10_tokens,k10_success1,ranking +q001,Create a GitHub issue,verb_product,github.issue_write,1.0,1.0,1.0,1.0000,0.8597,1,1,403,1.0,1,639,1.0,1,1048,1.0,github.issue_write github.create_issue github.delete_file github.pull_request_review_write github.get_discussion_comments github.get_discussion github.list_discussions github.star_repository github.create_gist github.update_issue_comment +q002,Create a Jira ticket,verb_product,jira.jira_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,265,1.0,1,385,1.0,1,742,1.0,jira.jira_create_issue jira.jira_search_issues zendesk.zendesk_get_ticket jira.update_sprint jira.jira_get_issue zendesk.zendesk_update_ticket jira.delete_version zendesk.zendesk_create_ticket jira.list_attachments jira.get_attachment +q003,Send a Slack message,verb_product,slack.slack_add_reaction,0.0,1.0,1.0,0.5000,0.6309,2,1,249,1.0,1,416,1.0,1,813,1.0,slack.slack_add_reaction slack.slack_post_message slack.slack_search_messages discord.discord_add_reaction slack.list_bookmarks github.delete_file slack.get_canvase figma.figma_post_comment slack.delete_canvase gmail.gmail_modify_email +q004,List open pull requests,resource,bitbucket.bitbucket_list_prs,0.0,1.0,1.0,0.5000,0.6199,2,1,237,1.0,1,530,1.0,1,1113,1.0,bitbucket.bitbucket_list_prs github.search_pull_requests github.list_pull_requests github.pull_request_review_write github.create_pull_request gitlab.gitlab_list_merge_requests github.add_comment_to_pending_review github.update_pull_request bitbucket.bitbucket_create_pr github.pull_request_read +q005,Open a pull request on GitHub,verb_product,github.create_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,417,1.0,1,713,1.0,1,1200,1.0,github.create_pull_request github.request_copilot_review github.pull_request_review_write github.add_comment_to_pending_review github.update_pull_request bitbucket.bitbucket_create_pr github.pull_request_read github.update_pull_request_branch github.merge_pull_request figma.figma_post_comment +q006,Run a SQL query against postgres,verb_product,postgres.query,1.0,1.0,1.0,1.0000,1.0000,1,1,165,1.0,1,281,1.0,1,546,1.0,postgres.query postgres.list_schemas mysql.mysql_query postgres.describe_table sqlite.sqlite_query postgres.list_tables postgres.explain_query mysql.mysql_list_tables salesforce.salesforce_query +q007,Read a local file from disk,paraphrase,filesystem.read_media_file,0.0,1.0,1.0,0.5000,0.6697,2,1,223,1.0,1,444,1.0,1,848,1.0,filesystem.read_media_file filesystem.read_file filesystem.read_text_file github.delete_file github.get_file_contents gitlab.gitlab_get_file figma.figma_get_images figma.figma_get_file_nodes brave_search.brave_local_search figma.figma_post_comment +q008,What files are in this folder,paraphrase,google_drive.gdrive_search,0.0,0.0,1.0,0.2500,0.3889,4,2,427,0.0,1,350,1.0,1,664,1.0,google_drive.gdrive_search github.create_gist github.update_gist filesystem.list_directory filesystem.move_file filesystem.directory_tree cloudflare.cf_purge_cache filesystem.search_files google_drive.gdrive_list_folder git.git_add +q009,Remember this fact in the knowledge graph,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,89,30,8108,0.0,18,7832,0.0,9,7625,0.0, +q010,Search the web with Brave,verb_product,brave_search.brave_web_search,1.0,1.0,1.0,1.0000,1.0000,1,1,256,1.0,1,365,1.0,1,671,1.0,brave_search.brave_web_search github.update_pull_request_branch playwright.browser_navigate puppeteer.puppeteer_navigate memory.delete_entities jira.jira_search_issues memory.create_relations puppeteer.puppeteer_click slack.slack_set_channel_topic puppeteer.puppeteer_evaluate +q011,Fetch the contents of a URL,verb_product,filesystem.read_file,0.0,0.0,0.0,0.1429,0.3333,7,3,664,0.0,2,693,0.0,1,671,1.0,filesystem.read_file slack.slack_set_channel_topic playwright.browser_snapshot git.git_add git.git_status github.get_me fetch.fetch github.get_repository_tree playwright.browser_take_screenshot playwright.browser_navigate +q012,Show git status,identifier,git.git_status,1.0,1.0,1.0,1.0000,1.0000,1,1,263,1.0,1,482,1.0,1,838,1.0,git.git_status github.delete_file github.list_check_runs github.pull_request_review_write gitlab.list_jobs git.git_show slack.list_bookmarks github.get_discussion_comments jira.list_attachments git.git_branch +q013,Commit staged changes,paraphrase,git.git_commit,1.0,1.0,1.0,1.0000,1.0000,1,1,192,1.0,1,305,1.0,1,663,1.0,git.git_commit git.git_reset git.git_show git.git_log git.git_diff_staged github.get_commit github.push_files +q014,Create a Linear issue,verb_product,linear.linear_get_issue,0.0,1.0,1.0,0.3333,0.5000,3,1,322,1.0,1,463,1.0,1,871,1.0,linear.linear_get_issue linear.linear_update_issue linear.linear_create_issue linear.linear_create_comment linear.linear_assign_issue linear.linear_create_project jira.jira_get_issue linear.list_users github.issue_write linear.update_label +q015,Make a task in my project tracker Linear,paraphrase,asana.asana_get_task,0.0,0.0,0.0,0.0000,0.0000,146,49,12388,0.0,30,12128,0.0,15,11768,0.0,asana.asana_get_task github.list_check_runs mysql.mysql_query clickup.clickup_update_task jira.update_sprint mysql.mysql_list_tables clickup.clickup_create_task gitlab.list_jobs clickup.clickup_get_task slack.list_bookmarks +q016,Post a comment on a GitHub issue,verb_product,github.update_issue_comment,0.0,0.0,0.0,0.1111,0.3010,9,3,1027,0.0,2,1107,0.0,1,1083,1.0,github.update_issue_comment figma.figma_post_comment github.get_discussion_comments trello.trello_add_comment github.create_pull_request github.add_comment_to_pending_review github.list_commits github.request_copilot_review github.add_issue_comment github.assign_copilot_to_issue +q017,Merge the pull request,ambiguous,github.merge_pull_request,1.0,1.0,1.0,1.0000,0.7602,1,1,337,1.0,1,531,1.0,1,936,1.0,github.merge_pull_request github.update_pull_request_branch gitlab.gitlab_create_merge_request playwright.browser_navigate github.pull_request_review_write gitlab.gitlab_get_merge_request memory.delete_entities github.create_pull_request gitlab.gitlab_list_merge_requests memory.create_relations +q018,Trigger a GitHub Actions workflow,verb_product,github.actions_run_trigger,1.0,1.0,1.0,1.0000,1.0000,1,1,398,1.0,1,578,1.0,1,1011,1.0,github.actions_run_trigger github.delete_file github.pull_request_review_write github.get_discussion_comments github.get_discussion github.list_discussions github.issue_write github.star_repository github.create_gist github.update_gist +q019,Get Dependabot alerts,resource,github.list_dependabot_alerts,1.0,1.0,1.0,1.0000,0.9502,1,1,269,1.0,1,421,1.0,1,827,1.0,github.list_dependabot_alerts github.list_secret_scanning_alerts github.get_dependabot_alert github.list_code_scanning_alerts +q020,List Sentry issues,verb_product,sentry.sentry_list_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,428,1.0,1,813,1.0,sentry.sentry_list_issues sentry.sentry_list_projects jira.jira_search_issues github.list_issues sentry.sentry_update_issue sentry.sentry_get_issue github.search_issues sentry.sentry_list_events linear.linear_list_issues gitlab.gitlab_list_issues +q021,Create a Stripe customer,verb_product,stripe.stripe_create_invoice,0.0,0.0,0.0,0.1667,0.3562,6,2,407,0.0,2,678,0.0,1,658,1.0,stripe.stripe_create_invoice stripe.stripe_create_payment_intent stripe.stripe_get_customer stripe.stripe_list_subscriptions stripe.stripe_list_invoices stripe.stripe_create_customer stripe.stripe_list_customers stripe.stripe_get_payment_intent stripe.stripe_create_price stripe.stripe_create_product +q022,Refund a Stripe payment,paraphrase,stripe.stripe_create_refund,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,379,1.0,1,660,1.0,stripe.stripe_create_refund stripe.stripe_get_payment_intent stripe.stripe_create_payment_intent stripe.stripe_list_customers stripe.stripe_create_price stripe.stripe_create_invoice stripe.stripe_create_product stripe.stripe_get_customer stripe.stripe_list_subscriptions stripe.stripe_cancel_subscription +q023,Send an email with Gmail,verb_product,gmail.gmail_send_email,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,392,1.0,1,775,1.0,gmail.gmail_send_email puppeteer.puppeteer_click puppeteer.puppeteer_fill slack.slack_add_reaction stripe.stripe_list_customers puppeteer.puppeteer_navigate jira.jira_search_issues asana.asana_get_task salesforce.salesforce_create_lead gmail.gmail_modify_email +q024,Search my Gmail inbox,verb_product,mysql.mysql_query,0.0,0.0,0.0,0.1111,0.3010,9,3,642,0.0,2,676,0.0,1,656,1.0,mysql.mysql_query gmail.gmail_send_email mysql.mysql_list_tables gmail.gmail_modify_email gmail.gmail_trash_email gmail.gmail_get_email gmail.gmail_list_labels gmail.gmail_create_draft gmail.gmail_search_emails +q025,Upload a file to Google Drive,paraphrase,slack.slack_upload_file,0.0,0.0,0.0,0.0000,0.0000,257,86,20636,0.0,52,20030,0.0,26,19458,0.0,slack.slack_upload_file aws_s3.s3_put_object filesystem.edit_file git.git_add github.delete_file slack.slack_add_reaction figma.figma_post_comment puppeteer.puppeteer_navigate figma.figma_get_comments discord.discord_add_reaction +q026,Create a calendar event,resource,google_calendar.gcal_delete_event,0.0,1.0,1.0,0.3333,0.5000,3,1,300,1.0,1,481,1.0,1,820,1.0,google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event github.pull_request_review_write google_calendar.gcal_list_calendars google_calendar.gcal_list_events +q027,List Kubernetes pods,verb_product,kubernetes.k8s_list_pods,1.0,1.0,1.0,1.0000,1.0000,1,1,262,1.0,1,416,1.0,1,719,1.0,kubernetes.k8s_list_pods kubernetes.k8s_scale_deployment kubernetes.k8s_delete kubernetes.k8s_list_deployments kubernetes.k8s_pod_logs kubernetes.k8s_list_services kubernetes.k8s_apply kubernetes.k8s_get_events kubernetes.k8s_get_pod +q028,Get logs from a k8s pod,verb_product,kubernetes.k8s_pod_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,283,1.0,1,459,1.0,1,913,1.0,kubernetes.k8s_pod_logs datadog.datadog_list_logs docker.docker_logs git.git_log github.delete_file github.actions_get_job_logs mongodb.mongo_delete kubernetes.k8s_get_pod datadog.datadog_query_metrics github.get_file_contents +q029,Run a docker container,verb_product,docker.docker_run,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,345,1.0,1,641,1.0,docker.docker_run docker.docker_rm docker.docker_stop docker.docker_exec docker.docker_logs docker.docker_build kubernetes.k8s_pod_logs docker.docker_ps docker.docker_images +q030,Show docker container logs,verb_product,docker.docker_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,235,1.0,1,379,1.0,1,763,1.0,docker.docker_logs docker.docker_run docker.docker_rm kubernetes.k8s_pod_logs docker.docker_stop docker.docker_exec git.git_log github.actions_get_job_logs docker.docker_build datadog.datadog_list_logs +q031,Navigate the browser to a URL,ambiguous,playwright.browser_navigate,1.0,1.0,1.0,1.0000,0.8869,1,1,186,1.0,1,300,1.0,1,561,1.0,playwright.browser_navigate playwright.browser_snapshot playwright.browser_take_screenshot playwright.browser_click playwright.browser_evaluate playwright.browser_press_key puppeteer.puppeteer_navigate playwright.browser_tab_new memory.delete_entities memory.create_relations +q032,Take a screenshot of the page,ambiguous,playwright.browser_take_screenshot,1.0,1.0,1.0,1.0000,1.0000,1,1,227,1.0,1,442,1.0,1,730,1.0,playwright.browser_take_screenshot puppeteer.puppeteer_screenshot filesystem.list_directory slack.slack_set_channel_topic github.update_pull_request filesystem.directory_tree sentry.sentry_update_issue playwright.browser_snapshot git.git_status github.get_me +q033,Create a GitLab merge request,verb_product,gitlab.gitlab_create_merge_request,1.0,1.0,1.0,1.0000,1.0000,1,1,291,1.0,1,441,1.0,1,833,1.0,gitlab.gitlab_create_merge_request gitlab.gitlab_get_merge_request gitlab.gitlab_list_merge_requests gitlab.gitlab_merge_merge_request gitlab.gitlab_create_pipeline gitlab.gitlab_get_issue gitlab.gitlab_list_projects gitlab.gitlab_get_file gitlab.gitlab_list_issues github.merge_pull_request +q034,Search Notion pages,verb_product,notion.notion_search,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,393,1.0,1,734,1.0,notion.notion_search notion.create_comment notion.get_data_source notion.list_users_tokens notion.notion_append_block notion.delete_users_token notion.get_users_token notion.create_data_source notion.notion_get_comments notion.update_data_source +q035,Create a Notion page,verb_product,notion.notion_get_comments,0.0,1.0,1.0,0.3333,0.5000,3,1,232,1.0,1,382,1.0,1,682,1.0,notion.notion_get_comments notion.notion_update_page notion.notion_create_page notion.notion_get_page notion.create_comment github.search_code notion.get_data_source confluence.confluence_get_page notion.list_users_tokens playwright.browser_snapshot +q036,Query a Notion database,verb_product,notion.notion_create_database,0.0,1.0,1.0,0.5000,0.6309,2,1,279,1.0,1,450,1.0,1,821,1.0,notion.notion_create_database notion.notion_query_database notion.notion_create_page notion.create_comment mongodb.mongo_delete notion.get_data_source mongodb.mongo_find notion.list_users_tokens mongodb.mongo_update notion.notion_append_block +q037,Create a Zendesk ticket,verb_product,zendesk.zendesk_get_ticket,0.0,1.0,1.0,0.3333,0.5000,3,1,257,1.0,1,429,1.0,1,733,1.0,zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_create_ticket zendesk.zendesk_search jira.jira_create_issue zendesk.zendesk_list_tickets +q038,Create a HubSpot contact,verb_product,hubspot.hubspot_get_contact,0.0,1.0,1.0,0.5000,0.6309,2,1,221,1.0,1,364,1.0,1,873,1.0,hubspot.hubspot_get_contact hubspot.hubspot_create_contact hubspot.hubspot_search_contacts hubspot.hubspot_list_contacts hubspot.hubspot_create_deal github.delete_file github.pull_request_review_write github.get_discussion_comments github.get_discussion github.list_discussions +q039,Create a Salesforce lead,verb_product,salesforce.salesforce_create_lead,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,423,1.0,1,696,1.0,salesforce.salesforce_create_lead salesforce.salesforce_convert_lead salesforce.salesforce_get_record salesforce.salesforce_update_record salesforce.salesforce_delete_record salesforce.salesforce_list_opportunities salesforce.salesforce_create_record salesforce.salesforce_query +q040,Run a SOQL query,identifier,salesforce.salesforce_query,1.0,1.0,1.0,1.0000,1.0000,1,1,173,1.0,1,331,1.0,1,746,1.0,salesforce.salesforce_query +q041,List Shopify orders,verb_product,shopify.shopify_list_orders,1.0,1.0,1.0,1.0000,1.0000,1,1,230,1.0,1,376,1.0,1,645,1.0,shopify.shopify_list_orders shopify.shopify_list_products shopify.shopify_update_product shopify.shopify_create_product shopify.shopify_get_product shopify.shopify_create_order shopify.shopify_list_customers shopify.shopify_get_order +q042,Create a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,0.0,0.0,1.0,0.2500,0.4307,4,2,486,0.0,1,387,1.0,1,715,1.0,pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_list_oncalls pagerduty.pagerduty_list_incidents +q043,Who is on call in PagerDuty,paraphrase,slack.slack_create_channel,0.0,0.0,0.0,0.1667,0.3562,6,2,442,0.0,2,833,0.0,1,810,1.0,slack.slack_create_channel figma.figma_post_comment filesystem.list_allowed_directories github.get_discussion_comments figma.figma_get_comments pagerduty.pagerduty_list_oncalls github.create_pull_request gmail.gmail_modify_email trello.trello_list_cards trello.trello_add_comment +q044,Query Datadog metrics,verb_product,datadog.datadog_query_metrics,1.0,1.0,1.0,1.0000,1.0000,1,1,287,1.0,1,449,1.0,1,758,1.0,datadog.datadog_query_metrics datadog.datadog_create_monitor prometheus.prom_list_metrics datadog.datadog_list_logs datadog.datadog_list_monitors datadog.datadog_get_monitor +q045,Search Datadog logs,verb_product,datadog.datadog_list_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,283,1.0,1,492,1.0,1,828,1.0,datadog.datadog_list_logs datadog.datadog_query_metrics git.git_log datadog.datadog_create_monitor github.actions_get_job_logs kubernetes.k8s_pod_logs datadog.datadog_list_monitors datadog.datadog_get_monitor docker.docker_logs +q046,List S3 buckets,verb_product,aws_s3.s3_list_buckets,1.0,1.0,1.0,1.0000,1.0000,1,1,153,1.0,1,311,1.0,1,726,1.0,aws_s3.s3_list_buckets +q047,Upload an object to S3,paraphrase,aws_s3.s3_put_object,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,423,1.0,1,740,1.0,aws_s3.s3_put_object aws_s3.s3_get_object aws_s3.s3_delete_object slack.slack_add_reaction slack.slack_upload_file gmail.gmail_send_email asana.asana_get_task puppeteer.puppeteer_click elasticsearch.es_search okta.okta_create_user +q048,Purge Cloudflare cache,verb_product,cloudflare.cf_purge_cache,1.0,1.0,1.0,1.0000,1.0000,1,1,231,1.0,1,340,1.0,1,746,1.0,cloudflare.cf_purge_cache cloudflare.cf_list_zones cloudflare.cf_create_dns cloudflare.cf_list_dns +q049,Create a Cloudflare DNS record,verb_product,cloudflare.cf_create_dns,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,363,1.0,1,714,1.0,cloudflare.cf_create_dns cloudflare.cf_list_dns cloudflare.cf_list_zones salesforce.salesforce_get_record cloudflare.cf_purge_cache salesforce.salesforce_update_record salesforce.salesforce_delete_record salesforce.salesforce_create_record git.git_commit +q050,List Vercel deployments,verb_product,vercel.vercel_list_deployments,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,291,1.0,1,648,1.0,vercel.vercel_list_deployments vercel.vercel_list_projects vercel.vercel_get_deployment kubernetes.k8s_list_deployments vercel.vercel_create_project +q051,Insert a row into Supabase,verb_product,supabase.supabase_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,332,1.0,1,768,1.0,supabase.supabase_insert mongodb.mongo_insert playwright.browser_type +q052,Export images from Figma,paraphrase,figma.figma_get_images,1.0,1.0,1.0,1.0000,1.0000,1,1,256,1.0,1,431,1.0,1,789,1.0,figma.figma_get_images github.delete_file docker.docker_images mongodb.mongo_delete datadog.datadog_query_metrics github.get_file_contents memory.delete_entities twilio.twilio_send_sms filesystem.read_media_file memory.delete_observations +q053,Send an SMS with Twilio,verb_product,puppeteer.puppeteer_click,0.0,1.0,1.0,0.5000,0.6309,2,1,232,1.0,1,379,1.0,1,704,1.0,puppeteer.puppeteer_click twilio.twilio_send_sms puppeteer.puppeteer_fill slack.slack_add_reaction puppeteer.puppeteer_navigate gmail.gmail_send_email twilio.twilio_list_messages jira.jira_search_issues asana.asana_get_task puppeteer.puppeteer_evaluate +q054,Create an Okta user,verb_product,okta.okta_create_user,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,366,1.0,1,625,1.0,okta.okta_create_user okta.okta_deactivate_user okta.okta_get_user slack.slack_add_reaction gmail.gmail_send_email asana.asana_get_task okta.okta_list_users github.list_gists puppeteer.puppeteer_click linear.get_user +q055,Instant query Prometheus,verb_product,prometheus.prom_query,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,313,1.0,1,749,1.0,prometheus.prom_query prometheus.prom_query_range prometheus.prom_list_metrics +q056,Search Grafana dashboards,verb_product,grafana.grafana_search_dashboards,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,309,1.0,1,745,1.0,grafana.grafana_search_dashboards grafana.grafana_get_dashboard grafana.grafana_create_annotation +q057,Trigger a CircleCI pipeline,verb_product,circleci.circleci_trigger_pipeline,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,390,1.0,1,747,1.0,circleci.circleci_trigger_pipeline circleci.circleci_list_pipelines circleci.circleci_get_pipeline github.actions_run_trigger gitlab.gitlab_create_pipeline +q058,Run terraform plan,identifier,terraform.tf_plan,1.0,1.0,1.0,1.0000,1.0000,1,1,166,1.0,1,281,1.0,1,687,1.0,terraform.tf_plan terraform.tf_state_list postgres.explain_query terraform.tf_apply +q059,Create an Asana task,verb_product,asana.asana_get_task,0.0,1.0,1.0,0.3333,0.5000,3,1,267,1.0,1,428,1.0,1,824,1.0,asana.asana_get_task asana.asana_update_task asana.asana_create_task asana.asana_create_project slack.slack_add_reaction gmail.gmail_send_email asana.asana_list_tasks clickup.clickup_update_task clickup.clickup_create_task puppeteer.puppeteer_click +q060,Create a Trello card,verb_product,trello.trello_add_comment,0.0,1.0,1.0,0.5000,0.6309,2,1,240,1.0,1,370,1.0,1,776,1.0,trello.trello_add_comment trello.trello_create_card trello.trello_create_board trello.trello_list_cards +q061,Create a ClickUp task,verb_product,clickup.clickup_update_task,0.0,1.0,1.0,0.5000,0.6309,2,1,257,1.0,1,364,1.0,1,727,1.0,clickup.clickup_update_task clickup.clickup_create_task clickup.clickup_get_task clickup.clickup_list_tasks puppeteer.puppeteer_click asana.asana_get_task asana.asana_update_task playwright.browser_click asana.asana_create_task +q062,Create a Confluence page,verb_product,confluence.confluence_get_page,0.0,1.0,1.0,0.5000,0.6309,2,1,260,1.0,1,367,1.0,1,710,1.0,confluence.confluence_get_page confluence.confluence_create_page confluence.confluence_update_page github.search_code playwright.browser_snapshot confluence.confluence_search notion.notion_get_comments github.search_repositories playwright.browser_take_screenshot notion.notion_update_page +q063,Send a Discord message,verb_product,discord.discord_add_reaction,0.0,1.0,1.0,0.5000,0.6309,2,1,265,1.0,1,363,1.0,1,763,1.0,discord.discord_add_reaction discord.discord_send_message slack.slack_add_reaction discord.discord_get_channel discord.discord_list_channels github.delete_file figma.figma_post_comment discord.discord_list_members gmail.gmail_modify_email slack.slack_post_message +q064,Search Elasticsearch,verb_product,elasticsearch.es_index_doc,0.0,1.0,1.0,0.5000,0.6309,2,1,239,1.0,1,331,1.0,1,767,1.0,elasticsearch.es_index_doc elasticsearch.es_search elasticsearch.es_get_doc +q065,Get a Redis key,verb_product,redis.redis_get,1.0,1.0,1.0,1.0000,1.0000,1,1,185,1.0,1,310,1.0,1,639,1.0,redis.redis_get redis.redis_set redis.redis_keys figma.figma_post_comment jira.jira_get_issue figma.figma_get_comments redis.redis_delete jira.jira_add_worklog jira.jira_add_comment aws_s3.s3_get_object +q066,Find MongoDB documents,verb_product,mongodb.mongo_delete,0.0,1.0,1.0,0.5000,0.6309,2,1,258,1.0,1,381,1.0,1,738,1.0,mongodb.mongo_delete mongodb.mongo_find mongodb.mongo_update mongodb.mongo_insert mongodb.mongo_list_collections +q067,List SQLite tables,verb_product,sqlite.sqlite_list_tables,1.0,1.0,1.0,1.0000,1.0000,1,1,156,1.0,1,252,1.0,1,542,1.0,sqlite.sqlite_list_tables sqlite.sqlite_query sqlite.sqlite_describe_table mysql.mysql_list_tables postgres.list_tables supabase.supabase_list_tables +q068,Describe a postgres table,verb_product,postgres.describe_table,1.0,1.0,1.0,1.0000,1.0000,1,1,170,1.0,1,286,1.0,1,597,1.0,postgres.describe_table sqlite.sqlite_describe_table postgres.list_schemas supabase.supabase_insert postgres.list_tables supabase.supabase_update postgres.query postgres.explain_query supabase.supabase_select +q069,github_create_issue,identifier,github.issue_write,1.0,1.0,1.0,1.0000,0.8597,1,1,392,1.0,1,596,1.0,1,1087,1.0,github.issue_write github.create_issue github.update_issue_comment github.issue_read github.add_issue_comment github.assign_copilot_to_issue github.list_check_runs jira.jira_get_issue github.delete_file github.pull_request_review_write +q070,createIssue,identifier,jira.jira_get_issue,0.0,0.0,0.0,0.1667,0.3153,6,2,616,0.0,2,976,0.0,1,958,1.0,jira.jira_get_issue github.issue_write linear.linear_get_issue jira.jira_add_worklog linear.linear_update_issue linear.linear_create_issue linear.linear_create_comment github.create_issue gitlab.gitlab_get_issue sentry.sentry_update_issue +q071,list-directory,identifier,github.get_file_contents,0.0,1.0,1.0,0.5000,0.6309,2,1,234,1.0,1,344,1.0,1,653,1.0,github.get_file_contents filesystem.list_directory filesystem.directory_tree filesystem.create_directory filesystem.get_file_info filesystem.list_directory_with_sizes +q072,sequential thinking next thought,paraphrase,sequential_thinking.sequentialthinking,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,382,1.0,1,797,1.0,sequential_thinking.sequentialthinking +q073,What time is it in Tokyo,paraphrase,time.convert_time,1.0,1.0,1.0,1.0000,0.6509,1,1,262,1.0,1,369,1.0,1,685,1.0,time.convert_time slack.slack_create_channel jira.jira_add_worklog filesystem.list_allowed_directories playwright.browser_wait_for time.get_current_time google_calendar.gcal_list_events grafana.grafana_create_annotation +q074,Convert time between timezones,verb_product,time.convert_time,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,362,1.0,1,704,1.0,time.convert_time salesforce.salesforce_convert_lead jira.jira_add_worklog memory.create_relations playwright.browser_wait_for git.git_diff time.get_current_time google_calendar.gcal_list_events grafana.grafana_create_annotation +q075,Star a GitHub repo,verb_product,github.star_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,345,1.0,1,525,1.0,1,963,1.0,github.star_repository github.delete_file github.pull_request_review_write github.get_discussion_comments github.get_discussion github.list_discussions github.issue_write github.create_gist github.update_gist github.list_releases +q076,Fork a repository,resource,github.fork_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,287,1.0,1,426,1.0,1,845,1.0,github.fork_repository github.delete_file github.list_discussions github.star_repository github.list_releases github.list_tags github.get_file_contents github.list_labels github.list_issues github.unstar_repository +q077,Get the latest GitHub release,verb_product,github.get_latest_release,1.0,1.0,1.0,1.0000,1.0000,1,1,210,1.0,1,329,1.0,1,702,1.0,github.get_latest_release github.list_notifications git.git_status github.get_me git.git_add github.get_repository_tree github.get_teams git.git_commit playwright.browser_navigate github.delete_file +q078,Create a gist,resource,github.create_gist,1.0,1.0,1.0,1.0000,1.0000,1,1,229,1.0,1,321,1.0,1,757,1.0,github.create_gist github.update_gist github.get_gist +q079,Search GitHub code,verb_product,github.search_code,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,536,1.0,1,995,1.0,github.search_code github.list_code_scanning_alerts github.delete_file github.pull_request_review_write github.get_discussion_comments github.get_discussion github.list_discussions github.get_code_scanning_alert github.issue_write github.star_repository +q080,List GitHub notifications,verb_product,github.list_notifications,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,473,1.0,1,913,1.0,github.list_notifications github.mark_all_notifications_read github.delete_file github.pull_request_review_write github.get_discussion_comments github.get_discussion github.list_discussions github.issue_write github.star_repository github.create_gist +q081,Assign Copilot to an issue,verb_product,github.assign_copilot_to_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,301,1.0,1,448,1.0,1,887,1.0,github.assign_copilot_to_issue linear.linear_assign_issue github.request_copilot_review slack.slack_add_reaction puppeteer.puppeteer_navigate jira.jira_assign_issue discord.discord_add_reaction datadog.datadog_query_metrics gmail.gmail_send_email jira.jira_add_worklog +q082,Get secret scanning alerts,resource,github.list_secret_scanning_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,295,1.0,1,470,1.0,1,827,1.0,github.list_secret_scanning_alerts github.get_secret_scanning_alert github.list_code_scanning_alerts github.list_dependabot_alerts github.get_code_scanning_alert +q083,Create or update a file on GitHub,verb_product,figma.figma_post_comment,0.0,0.0,0.0,0.1111,0.2288,9,3,798,0.0,2,844,0.0,1,820,1.0,figma.figma_post_comment figma.figma_get_comments github.get_file_contents gmail.gmail_modify_email trello.trello_list_cards google_drive.gdrive_get_file filesystem.move_file filesystem.read_media_file github.create_or_update_file filesystem.write_file +q084,List collaborators on a repo,resource,github.list_repository_collaborators,1.0,1.0,1.0,1.0000,1.0000,1,1,342,1.0,1,550,1.0,1,1085,1.0,github.list_repository_collaborators github.get_discussion_comments github.create_pull_request github.list_commits github.update_issue_comment github.request_copilot_review github.assign_copilot_to_issue figma.figma_post_comment github.delete_file github.pull_request_review_write +q085,Reply in a Slack thread,verb_product,slack.list_bookmarks,0.0,1.0,1.0,0.5000,0.6309,2,1,247,1.0,1,375,1.0,1,769,1.0,slack.list_bookmarks slack.slack_reply_to_thread slack.get_canvase slack.delete_canvase slack.update_bookmark slack.delete_usergroup slack.create_file slack.update_usergroup slack.create_usergroup slack.update_pin +q086,Add an emoji reaction on Slack,verb_product,slack.slack_add_reaction,1.0,1.0,1.0,1.0000,1.0000,1,1,293,1.0,1,399,1.0,1,777,1.0,slack.slack_add_reaction discord.discord_add_reaction gmail.gmail_send_email asana.asana_get_task puppeteer.puppeteer_click elasticsearch.es_search okta.okta_create_user okta.okta_deactivate_user github.create_issue aws_s3.s3_list_objects +q087,Search Slack messages,verb_product,slack.slack_search_messages,1.0,1.0,1.0,1.0000,1.0000,1,1,251,1.0,1,382,1.0,1,704,1.0,slack.slack_search_messages slack.slack_get_channel_history slack.slack_add_reaction slack.list_bookmarks twilio.twilio_list_messages slack.get_canvase gmail.gmail_search_emails slack.delete_canvase slack.slack_list_users slack.update_bookmark +q088,Create a Slack channel,verb_product,slack.slack_add_reaction,0.0,0.0,1.0,0.2000,0.3869,5,2,507,0.0,1,414,1.0,1,751,1.0,slack.slack_add_reaction slack.slack_reply_to_thread slack.slack_post_message slack.slack_set_channel_topic slack.slack_create_channel slack.slack_get_channel_history slack.slack_get_thread_replies slack.slack_leave_channel slack.slack_join_channel slack.slack_upload_file +q089,Transition a Jira issue,verb_product,jira.jira_transition_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,231,1.0,1,414,1.0,1,823,1.0,jira.jira_transition_issue jira.jira_get_issue jira.jira_add_worklog jira.jira_add_comment jira.jira_create_issue jira.jira_update_issue jira.jira_assign_issue jira.jira_search_issues jira.update_sprint github.issue_write +q090,Search Jira with JQL,identifier,jira.jira_search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,324,1.0,1,636,1.0,jira.jira_search_issues puppeteer.puppeteer_navigate jira.update_sprint jira.jira_get_issue puppeteer.puppeteer_click jira.delete_version puppeteer.puppeteer_evaluate jira.list_attachments puppeteer.puppeteer_screenshot jira.get_attachment +q091,Add a worklog to Jira,verb_product,jira.jira_add_worklog,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,411,1.0,1,777,1.0,jira.jira_add_worklog jira.jira_add_comment jira.jira_transition_issue slack.slack_add_reaction jira.jira_search_issues puppeteer.puppeteer_navigate jira.update_sprint discord.discord_add_reaction jira.jira_get_issue datadog.datadog_query_metrics +q092,List Linear teams,verb_product,linear.linear_list_teams,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,302,1.0,1,773,1.0,linear.linear_list_teams linear.linear_create_project linear.list_users github.get_teams linear.linear_get_issue linear.update_label linear.linear_update_issue linear.create_comment linear.update_cycle linear.linear_create_issue +q093,Comment on a Linear issue,verb_product,figma.figma_post_comment,0.0,1.0,1.0,0.3333,0.5000,3,1,256,1.0,1,378,1.0,1,778,1.0,figma.figma_post_comment linear.create_comment linear.linear_create_comment trello.trello_add_comment linear.delete_comment github.update_issue_comment linear.update_comment linear.get_comment notion.create_comment linear.linear_create_project +q094,Move a local file,paraphrase,filesystem.move_file,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,358,1.0,1,754,1.0,filesystem.move_file filesystem.read_media_file filesystem.read_file filesystem.read_text_file brave_search.brave_local_search github.delete_file gmail.gmail_trash_email figma.figma_post_comment figma.figma_get_comments slack.create_file +q095,Recursively search files by pattern,paraphrase,filesystem.search_files,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,351,1.0,1,693,1.0,filesystem.search_files google_drive.gdrive_search salesforce.salesforce_get_record github.create_gist jira.jira_get_issue github.update_gist linear.linear_get_issue filesystem.list_directory github.search_repositories filesystem.move_file +q096,Edit a file in place with old and new string,paraphrase,filesystem.edit_file,1.0,1.0,1.0,1.0000,1.0000,1,1,323,1.0,1,565,1.0,1,950,1.0,filesystem.edit_file slack.create_file google_drive.gdrive_create_file github.create_or_update_file github.get_file_blame slack.get_file github.delete_file github.list_check_runs slack.update_file figma.figma_post_comment +q097,Read the whole knowledge graph,paraphrase,memory.delete_entities,0.0,0.0,0.0,0.1250,0.3155,8,3,511,0.0,2,563,0.0,1,542,1.0,memory.delete_entities memory.create_relations memory.search_nodes memory.add_observations memory.create_entities memory.open_nodes memory.delete_relations memory.read_graph playwright.browser_navigate slack.slack_set_channel_topic +q098,Search memory nodes,verb_product,memory.search_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,178,1.0,1,305,1.0,1,546,1.0,memory.search_nodes memory.open_nodes memory.delete_entities memory.create_relations figma.figma_get_file_nodes memory.delete_observations memory.add_observations memory.create_entities memory.delete_relations memory.read_graph +q099,Show unstaged git diff,verb_product,git.git_diff_unstaged,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,447,1.0,1,796,1.0,git.git_diff_unstaged git.git_diff_staged git.git_diff github.delete_file github.pull_request_review_write git.git_show github.get_discussion_comments git.git_branch github.get_discussion git.git_log +q100,Stage files with git add,identifier,filesystem.list_directory_with_sizes,0.0,0.0,0.0,0.0000,0.0000,115,39,9731,0.0,23,9285,0.0,12,9371,0.0,filesystem.list_directory_with_sizes salesforce.salesforce_list_opportunities google_drive.gdrive_search puppeteer.puppeteer_navigate github.create_gist jira.jira_search_issues github.update_gist puppeteer.puppeteer_click filesystem.list_directory puppeteer.puppeteer_evaluate +q101,Create a Shopify product,verb_product,shopify.shopify_update_product,0.0,1.0,1.0,0.5000,0.6309,2,1,240,1.0,1,377,1.0,1,702,1.0,shopify.shopify_update_product shopify.shopify_create_product shopify.shopify_get_product shopify.shopify_list_products shopify.shopify_list_orders stripe.stripe_create_price stripe.stripe_create_product shopify.shopify_create_order shopify.shopify_list_customers shopify.shopify_get_order +q102,Cancel a Stripe subscription,verb_product,stripe.stripe_cancel_subscription,1.0,1.0,1.0,1.0000,1.0000,1,1,193,1.0,1,321,1.0,1,660,1.0,stripe.stripe_cancel_subscription stripe.stripe_list_customers stripe.stripe_get_payment_intent stripe.stripe_create_price stripe.stripe_create_invoice stripe.stripe_create_product stripe.stripe_create_payment_intent stripe.stripe_get_customer stripe.stripe_create_refund stripe.stripe_list_subscriptions +q103,Create a payment intent,resource,stripe.stripe_get_payment_intent,0.0,1.0,1.0,0.5000,0.6309,2,1,234,1.0,1,326,1.0,1,762,1.0,stripe.stripe_get_payment_intent stripe.stripe_create_payment_intent stripe.stripe_create_refund +q104,Acknowledge a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,363,1.0,1,717,1.0,pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_list_oncalls pagerduty.pagerduty_get_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_create_incident +q105,Create a Datadog monitor,verb_product,datadog.datadog_create_monitor,1.0,1.0,1.0,1.0000,1.0000,1,1,297,1.0,1,459,1.0,1,816,1.0,datadog.datadog_create_monitor datadog.datadog_get_monitor datadog.datadog_query_metrics datadog.datadog_list_logs datadog.datadog_list_monitors +q106,Resolve a Sentry issue,paraphrase,sentry.sentry_update_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,251,1.0,1,368,1.0,1,797,1.0,sentry.sentry_update_issue sentry.sentry_get_issue sentry.sentry_list_events pagerduty.pagerduty_resolve_incident sentry.sentry_list_projects jira.jira_get_issue sentry.sentry_list_issues github.issue_write linear.linear_get_issue jira.jira_add_worklog +q107,Scale a Kubernetes deployment,verb_product,kubernetes.k8s_scale_deployment,1.0,1.0,1.0,1.0000,1.0000,1,1,266,1.0,1,420,1.0,1,726,1.0,kubernetes.k8s_scale_deployment vercel.vercel_get_deployment kubernetes.k8s_delete kubernetes.k8s_list_deployments kubernetes.k8s_pod_logs kubernetes.k8s_list_services kubernetes.k8s_apply kubernetes.k8s_list_pods kubernetes.k8s_get_events kubernetes.k8s_get_pod +q108,Apply a Kubernetes manifest,verb_product,kubernetes.k8s_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,246,1.0,1,390,1.0,1,727,1.0,kubernetes.k8s_apply kubernetes.k8s_scale_deployment terraform.tf_apply kubernetes.k8s_delete kubernetes.k8s_list_deployments kubernetes.k8s_pod_logs kubernetes.k8s_list_services kubernetes.k8s_list_pods kubernetes.k8s_get_events kubernetes.k8s_get_pod +q109,Stop a docker container,verb_product,docker.docker_stop,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,345,1.0,1,641,1.0,docker.docker_stop docker.docker_run docker.docker_rm docker.docker_exec docker.docker_logs docker.docker_build kubernetes.k8s_pod_logs docker.docker_ps docker.docker_images +q110,Build a docker image,verb_product,docker.docker_build,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,342,1.0,1,601,1.0,docker.docker_build docker.docker_run docker.docker_rm filesystem.read_media_file docker.docker_stop docker.docker_ps docker.docker_exec docker.docker_images docker.docker_logs +q111,Click an element in Playwright,verb_product,puppeteer.puppeteer_click,0.0,1.0,1.0,0.5000,0.6309,2,1,213,1.0,1,380,1.0,1,760,1.0,puppeteer.puppeteer_click playwright.browser_click playwright.browser_take_screenshot playwright.browser_type slack.slack_add_reaction clickup.clickup_update_task gmail.gmail_send_email clickup.clickup_list_tasks asana.asana_get_task clickup.clickup_create_task +q112,Type text into a browser input,paraphrase,playwright.browser_type,1.0,1.0,1.0,1.0000,0.7602,1,1,296,1.0,1,429,1.0,1,862,1.0,playwright.browser_type google_drive.gdrive_create_file slack.slack_reply_to_thread supabase.supabase_insert github.projects_list slack.slack_post_message mongodb.mongo_insert datadog.datadog_create_monitor trello.trello_add_comment cloudflare.cf_create_dns +q113,List GitLab projects,verb_product,gitlab.gitlab_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,398,1.0,1,797,1.0,gitlab.gitlab_list_projects gitlab.gitlab_create_merge_request github.projects_list gitlab.gitlab_create_pipeline gitlab.gitlab_get_issue github.projects_write gitlab.gitlab_get_file gitlab.gitlab_get_merge_request gitlab.gitlab_list_issues gitlab.gitlab_list_merge_requests +q114,Create a Bitbucket pull request,verb_product,bitbucket.bitbucket_create_pr,1.0,1.0,1.0,1.0000,1.0000,1,1,365,1.0,1,669,1.0,1,1202,1.0,bitbucket.bitbucket_create_pr github.pull_request_review_write bitbucket.bitbucket_list_prs github.create_pull_request github.add_comment_to_pending_review github.update_pull_request github.pull_request_read github.request_copilot_review github.update_pull_request_branch github.merge_pull_request +q115,Create a Gmail draft,verb_product,gmail.gmail_create_draft,1.0,1.0,1.0,1.0000,1.0000,1,1,344,1.0,1,494,1.0,1,744,1.0,gmail.gmail_create_draft gmail.gmail_send_email github.create_pull_request gmail.gmail_modify_email gmail.gmail_trash_email gmail.gmail_get_email gmail.gmail_list_labels gmail.gmail_search_emails +q116,Share a Google Drive file,verb_product,google_drive.gdrive_share_file,1.0,1.0,1.0,1.0000,1.0000,1,1,270,1.0,1,382,1.0,1,791,1.0,google_drive.gdrive_share_file google_drive.gdrive_search google_drive.gdrive_create_file google_drive.gdrive_get_file google_drive.gdrive_list_folder google_drive.gdrive_update_file google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event google_calendar.gcal_list_calendars +q117,List Google Calendar events,verb_product,google_calendar.gcal_list_events,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,427,1.0,1,770,1.0,google_calendar.gcal_list_events google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event google_calendar.gcal_list_calendars sentry.sentry_list_events google_drive.gdrive_search google_drive.gdrive_create_file kubernetes.k8s_get_events google_drive.gdrive_get_file +q118,Delete a calendar event,resource,google_calendar.gcal_delete_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,481,1.0,1,820,1.0,google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event github.pull_request_review_write google_calendar.gcal_list_calendars google_calendar.gcal_list_events +q119,List Vercel projects,verb_product,vercel.vercel_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,170,1.0,1,262,1.0,1,567,1.0,vercel.vercel_list_projects vercel.vercel_get_deployment sentry.sentry_list_projects vercel.vercel_list_deployments jira.jira_list_projects vercel.vercel_create_project github.projects_list github.projects_write gitlab.gitlab_list_projects linear.linear_list_projects +q120,Select rows from Supabase,verb_product,supabase.supabase_select,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,425,1.0,1,830,1.0,supabase.supabase_select supabase.supabase_insert github.delete_file supabase.supabase_update mongodb.mongo_delete datadog.datadog_query_metrics github.get_file_contents memory.delete_entities twilio.twilio_send_sms filesystem.read_media_file +q121,Post a Figma comment,verb_product,figma.figma_post_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,247,1.0,1,391,1.0,1,871,1.0,figma.figma_post_comment notion.create_comment figma.figma_get_comments figma.figma_get_file jira.jira_add_worklog figma.figma_get_images linear.create_comment figma.figma_get_file_nodes github.add_comment_to_pending_review linear.linear_create_comment +q122,List Okta users,verb_product,okta.okta_list_users,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,317,1.0,1,625,1.0,okta.okta_list_users okta.okta_create_user slack.slack_list_users okta.okta_deactivate_user linear.list_users pagerduty.pagerduty_list_oncalls okta.okta_get_user notion.list_users_tokens notion.delete_users_token notion.get_users_token +q123,Range query Prometheus,verb_product,prometheus.prom_query_range,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,313,1.0,1,749,1.0,prometheus.prom_query_range prometheus.prom_query prometheus.prom_list_metrics +q124,Create a Grafana annotation,verb_product,grafana.grafana_create_annotation,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,310,1.0,1,746,1.0,grafana.grafana_create_annotation grafana.grafana_search_dashboards grafana.grafana_get_dashboard +q125,Apply terraform,identifier,terraform.tf_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,194,1.0,1,292,1.0,1,698,1.0,terraform.tf_apply kubernetes.k8s_apply terraform.tf_state_list terraform.tf_plan +q126,List Asana tasks,verb_product,asana.asana_list_tasks,1.0,1.0,1.0,1.0000,1.0000,1,1,192,1.0,1,386,1.0,1,707,1.0,asana.asana_list_tasks asana.asana_get_task clickup.clickup_list_tasks asana.asana_update_task asana.asana_create_task asana.asana_create_project +q127,Comment on a Trello card,verb_product,trello.trello_add_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,444,1.0,1,835,1.0,trello.trello_add_comment figma.figma_post_comment trello.trello_list_cards github.update_issue_comment notion.create_comment trello.trello_create_board github.get_discussion_comments jira.jira_add_worklog figma.figma_get_comments linear.create_comment +q128,Update a ClickUp task,verb_product,clickup.clickup_update_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,364,1.0,1,727,1.0,clickup.clickup_update_task clickup.clickup_create_task clickup.clickup_get_task clickup.clickup_list_tasks puppeteer.puppeteer_click asana.asana_get_task asana.asana_update_task playwright.browser_click asana.asana_create_task +q129,Search Confluence,verb_product,confluence.confluence_get_page,0.0,0.0,1.0,0.2500,0.4307,4,2,423,0.0,1,366,1.0,1,772,1.0,confluence.confluence_get_page confluence.confluence_create_page confluence.confluence_update_page confluence.confluence_search +q130,Create a HubSpot deal,verb_product,hubspot.hubspot_create_deal,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,873,1.0,hubspot.hubspot_create_deal hubspot.hubspot_get_contact hubspot.hubspot_create_contact hubspot.hubspot_search_contacts hubspot.hubspot_list_contacts github.delete_file github.pull_request_review_write github.get_discussion_comments github.get_discussion github.list_discussions +q131,List Zendesk tickets,verb_product,zendesk.zendesk_search,0.0,1.0,1.0,0.5000,0.6309,2,1,189,1.0,1,367,1.0,1,724,1.0,zendesk.zendesk_search zendesk.zendesk_list_tickets zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_create_ticket +q132,List Discord channels,verb_product,discord.discord_list_channels,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,339,1.0,1,638,1.0,discord.discord_list_channels discord.discord_add_reaction slack.slack_list_channels discord.discord_get_channel discord.discord_send_message discord.discord_list_members +q133,Set a Redis key,verb_product,redis.redis_get,0.0,1.0,1.0,0.5000,0.6309,2,1,185,1.0,1,310,1.0,1,639,1.0,redis.redis_get redis.redis_set redis.redis_keys figma.figma_post_comment jira.jira_get_issue figma.figma_get_comments redis.redis_delete jira.jira_add_worklog jira.jira_add_comment aws_s3.s3_get_object +q134,Insert MongoDB documents,verb_product,mongodb.mongo_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,381,1.0,1,698,1.0,mongodb.mongo_insert mongodb.mongo_delete mongodb.mongo_find mongodb.mongo_update mongodb.mongo_list_collections supabase.supabase_insert +q135,Run a MySQL query,verb_product,mysql.mysql_query,1.0,1.0,1.0,1.0000,1.0000,1,1,158,1.0,1,271,1.0,1,523,1.0,mysql.mysql_query mysql.mysql_list_tables postgres.list_schemas postgres.describe_table postgres.list_tables sqlite.sqlite_query postgres.query postgres.explain_query +q136,Index a document in Elasticsearch,verb_product,elasticsearch.es_index_doc,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,388,1.0,1,740,1.0,elasticsearch.es_index_doc elasticsearch.es_get_doc github.list_check_runs elasticsearch.es_search jira.update_sprint gitlab.list_jobs playwright.browser_tab_close slack.list_bookmarks notion.create_comment jira.delete_version +q137,Get an S3 object,verb_product,aws_s3.s3_get_object,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,432,1.0,1,712,1.0,aws_s3.s3_get_object aws_s3.s3_put_object aws_s3.s3_delete_object slack.slack_add_reaction gmail.gmail_send_email asana.asana_get_task puppeteer.puppeteer_click elasticsearch.es_search okta.okta_create_user okta.okta_deactivate_user +q138,List Cloudflare zones,verb_product,cloudflare.cf_list_zones,1.0,1.0,1.0,1.0000,1.0000,1,1,230,1.0,1,339,1.0,1,745,1.0,cloudflare.cf_list_zones cloudflare.cf_purge_cache cloudflare.cf_create_dns cloudflare.cf_list_dns +q139,Create a Stripe invoice,verb_product,stripe.stripe_create_invoice,1.0,1.0,1.0,1.0000,1.0000,1,1,191,1.0,1,339,1.0,1,660,1.0,stripe.stripe_create_invoice stripe.stripe_list_customers stripe.stripe_get_payment_intent stripe.stripe_create_price stripe.stripe_create_product stripe.stripe_create_payment_intent stripe.stripe_get_customer stripe.stripe_create_refund stripe.stripe_list_subscriptions stripe.stripe_cancel_subscription +q140,List GitHub Actions workflows,verb_product,github.actions_run_trigger,0.0,0.0,1.0,0.2500,0.4307,4,2,672,0.0,1,558,1.0,1,1032,1.0,github.actions_run_trigger github.actions_get_job_logs github.delete_file github.actions_list github.pull_request_review_write github.get_discussion_comments github.get_discussion github.actions_get github.list_discussions github.issue_write +q141,Get GitHub Actions job logs,verb_product,github.actions_run_trigger,0.0,1.0,1.0,0.5000,0.6309,2,1,349,1.0,1,559,1.0,1,1033,1.0,github.actions_run_trigger github.actions_get_job_logs github.delete_file github.actions_list github.pull_request_review_write github.get_discussion_comments github.get_discussion github.actions_get github.list_discussions github.issue_write +q142,Create a GitHub repository,verb_product,github.delete_file,0.0,0.0,0.0,0.0000,0.0000,24,8,2112,0.0,5,2116,0.0,3,2504,0.0,github.delete_file github.list_discussions github.star_repository github.list_releases github.list_tags github.get_file_contents github.list_labels github.list_issues github.unstar_repository github.list_branches +q143,Checkout a git branch,verb_product,git.git_checkout,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,504,1.0,1,1049,1.0,git.git_checkout github.delete_file git.git_branch gitlab.gitlab_create_merge_request github.create_or_update_file git.git_create_branch github.list_commits github.push_files github.create_branch github.pull_request_review_write +q144,Show git log,identifier,git.git_log,1.0,1.0,1.0,1.0000,1.0000,1,1,338,1.0,1,492,1.0,1,900,1.0,git.git_log github.delete_file github.pull_request_review_write git.git_show github.get_discussion_comments git.git_branch github.get_discussion github.list_discussions github.issue_write github.star_repository +q145,List postgres schemas,verb_product,postgres.list_schemas,1.0,1.0,1.0,1.0000,1.0000,1,1,169,1.0,1,277,1.0,1,634,1.0,postgres.list_schemas postgres.describe_table postgres.list_tables postgres.query postgres.explain_query +q146,Open nodes in the knowledge graph,verb_product,memory.search_nodes,0.0,1.0,1.0,0.5000,0.6309,2,1,180,1.0,1,297,1.0,1,523,1.0,memory.search_nodes memory.open_nodes memory.create_relations slack.slack_list_channels time.get_current_time postgres.list_schemas memory.add_observations postgres.list_tables memory.create_entities sqlite.sqlite_list_tables +q147,Get file metadata from the filesystem,paraphrase,github.delete_file,0.0,0.0,1.0,0.2000,0.3869,5,2,525,0.0,1,412,1.0,1,810,1.0,github.delete_file google_drive.gdrive_get_file github.get_file_contents filesystem.read_media_file filesystem.get_file_info gitlab.gitlab_get_file filesystem.read_file filesystem.read_text_file figma.figma_get_images figma.figma_get_file_nodes +q148,Write a new local file,paraphrase,filesystem.read_media_file,0.0,1.0,1.0,0.5000,0.6309,2,1,206,1.0,1,384,1.0,1,847,1.0,filesystem.read_media_file filesystem.write_file filesystem.read_file filesystem.edit_file filesystem.read_text_file playwright.browser_tab_new brave_search.brave_local_search github.delete_file github.create_pull_request figma.figma_post_comment +q149,Search GitHub issues,verb_product,github.list_issues,0.0,1.0,1.0,0.5000,0.6309,2,1,322,1.0,1,558,1.0,1,998,1.0,github.list_issues github.search_issues github.delete_file github.pull_request_review_write github.get_discussion_comments github.get_discussion github.list_discussions github.issue_write github.star_repository github.create_gist +q150,Get the authenticated GitHub user,paraphrase,github.list_notifications,0.0,1.0,1.0,0.5000,0.6309,2,1,179,1.0,1,302,1.0,1,702,1.0,github.list_notifications github.get_me github.get_teams github.get_latest_release git.git_status git.git_add github.get_repository_tree git.git_commit playwright.browser_navigate github.delete_file diff --git a/experiments/mcp-tool-search/results/codedb_word.json b/experiments/mcp-tool-search/results/codedb_word.json new file mode 100644 index 00000000..6b9f1a99 --- /dev/null +++ b/experiments/mcp-tool-search/results/codedb_word.json @@ -0,0 +1,6773 @@ +{ + "summary": { + "n": 150, + "index_s": 0.44171834199994464, + "rss_mb": null, + "p50_ms": 30.814311000028738, + "p95_ms": 32.19671114995322, + "recall@1": 0.6733333333333333, + "recall@1_lo": 0.5933333333333334, + "recall@1_hi": 0.7466666666666667, + "recall@3": 0.8733333333333333, + "recall@3_lo": 0.8133333333333334, + "recall@3_hi": 0.92, + "recall@5": 0.9133333333333333, + "recall@5_lo": 0.8666666666666667, + "recall@5_hi": 0.9533333333333334, + "mrr@10": 0.7833412698412698, + "mrr@10_lo": 0.7279338624338625, + "mrr@10_hi": 0.835026455026455, + "ndcg@10": 0.8190657137179224, + "ndcg@10_lo": 0.773412050894116, + "ndcg@10_hi": 0.8605633185401583 + }, + "meta": { + "name": "codedb_word", + "needs_model": false, + "needs_api": false, + "notes": "codedb_word on distinctive query tokens, RRF" + }, + "rows": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.issue_write", + "github.create_issue", + "github.delete_file", + "github.pull_request_review_write", + "github.get_discussion_comments", + "github.get_discussion", + "github.list_discussions", + "github.star_repository", + "github.create_gist", + "github.update_issue_comment" + ], + "hit1": "github.issue_write", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8597186998521972, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 403, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 639, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1048, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "kind": "verb_product", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_create_issue", + "jira.jira_search_issues", + "zendesk.zendesk_get_ticket", + "jira.update_sprint", + "jira.jira_get_issue", + "zendesk.zendesk_update_ticket", + "jira.delete_version", + "zendesk.zendesk_create_ticket", + "jira.list_attachments", + "jira.get_attachment" + ], + "hit1": "jira.jira_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 265, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "kind": "verb_product", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "ranking": [ + "slack.slack_add_reaction", + "slack.slack_post_message", + "slack.slack_search_messages", + "discord.discord_add_reaction", + "slack.list_bookmarks", + "github.delete_file", + "slack.get_canvase", + "figma.figma_post_comment", + "slack.delete_canvase", + "gmail.gmail_modify_email" + ], + "hit1": "slack.slack_add_reaction", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 813, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "kind": "resource", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "ranking": [ + "bitbucket.bitbucket_list_prs", + "github.search_pull_requests", + "github.list_pull_requests", + "github.pull_request_review_write", + "github.create_pull_request", + "gitlab.gitlab_list_merge_requests", + "github.add_comment_to_pending_review", + "github.update_pull_request", + "bitbucket.bitbucket_create_pr", + "github.pull_request_read" + ], + "hit1": "bitbucket.bitbucket_list_prs", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6199062332840657, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 530, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1113, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "ranking": [ + "github.create_pull_request", + "github.request_copilot_review", + "github.pull_request_review_write", + "github.add_comment_to_pending_review", + "github.update_pull_request", + "bitbucket.bitbucket_create_pr", + "github.pull_request_read", + "github.update_pull_request_branch", + "github.merge_pull_request", + "figma.figma_post_comment" + ], + "hit1": "github.create_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 417, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 713, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1200, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "kind": "verb_product", + "gold": [ + "postgres.query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "postgres.list_schemas", + "mysql.mysql_query", + "postgres.describe_table", + "sqlite.sqlite_query", + "postgres.list_tables", + "postgres.explain_query", + "mysql.mysql_list_tables", + "salesforce.salesforce_query" + ], + "hit1": "postgres.query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 165, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 546, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "kind": "paraphrase", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "ranking": [ + "filesystem.read_media_file", + "filesystem.read_file", + "filesystem.read_text_file", + "github.delete_file", + "github.get_file_contents", + "gitlab.gitlab_get_file", + "figma.figma_get_images", + "figma.figma_get_file_nodes", + "brave_search.brave_local_search", + "figma.figma_post_comment" + ], + "hit1": "filesystem.read_media_file", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.66967181649423, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 444, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 848, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "kind": "paraphrase", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "ranking": [ + "google_drive.gdrive_search", + "github.create_gist", + "github.update_gist", + "filesystem.list_directory", + "filesystem.move_file", + "filesystem.directory_tree", + "cloudflare.cf_purge_cache", + "filesystem.search_files", + "google_drive.gdrive_list_folder", + "git.git_add" + ], + "hit1": "google_drive.gdrive_search", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.3888813863887987, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 427, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 664, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 89, + "k3_calls": 30, + "k3_tokens": 8108, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7832, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7625, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "kind": "verb_product", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "ranking": [ + "brave_search.brave_web_search", + "github.update_pull_request_branch", + "playwright.browser_navigate", + "puppeteer.puppeteer_navigate", + "memory.delete_entities", + "jira.jira_search_issues", + "memory.create_relations", + "puppeteer.puppeteer_click", + "slack.slack_set_channel_topic", + "puppeteer.puppeteer_evaluate" + ], + "hit1": "brave_search.brave_web_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 365, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 671, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "kind": "verb_product", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "ranking": [ + "filesystem.read_file", + "slack.slack_set_channel_topic", + "playwright.browser_snapshot", + "git.git_add", + "git.git_status", + "github.get_me", + "fetch.fetch", + "github.get_repository_tree", + "playwright.browser_take_screenshot", + "playwright.browser_navigate" + ], + "hit1": "filesystem.read_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.14285714285714285, + "ndcg@10": 0.3333333333333333, + "hit_rank": 7, + "k3_calls": 3, + "k3_tokens": 664, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 693, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 671, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "kind": "identifier", + "gold": [ + "git.git_status" + ], + "alts": [], + "ranking": [ + "git.git_status", + "github.delete_file", + "github.list_check_runs", + "github.pull_request_review_write", + "gitlab.list_jobs", + "git.git_show", + "slack.list_bookmarks", + "github.get_discussion_comments", + "jira.list_attachments", + "git.git_branch" + ], + "hit1": "git.git_status", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 263, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 482, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 838, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "kind": "paraphrase", + "gold": [ + "git.git_commit" + ], + "alts": [], + "ranking": [ + "git.git_commit", + "git.git_reset", + "git.git_show", + "git.git_log", + "git.git_diff_staged", + "github.get_commit", + "github.push_files" + ], + "hit1": "git.git_commit", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 305, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_get_issue", + "linear.linear_update_issue", + "linear.linear_create_issue", + "linear.linear_create_comment", + "linear.linear_assign_issue", + "linear.linear_create_project", + "jira.jira_get_issue", + "linear.list_users", + "github.issue_write", + "linear.update_label" + ], + "hit1": "linear.linear_get_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 322, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 871, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "kind": "paraphrase", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "asana.asana_get_task", + "github.list_check_runs", + "mysql.mysql_query", + "clickup.clickup_update_task", + "jira.update_sprint", + "mysql.mysql_list_tables", + "clickup.clickup_create_task", + "gitlab.list_jobs", + "clickup.clickup_get_task", + "slack.list_bookmarks" + ], + "hit1": "asana.asana_get_task", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 146, + "k3_calls": 49, + "k3_tokens": 12388, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 30, + "k5_tokens": 12128, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11768, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "ranking": [ + "github.update_issue_comment", + "figma.figma_post_comment", + "github.get_discussion_comments", + "trello.trello_add_comment", + "github.create_pull_request", + "github.add_comment_to_pending_review", + "github.list_commits", + "github.request_copilot_review", + "github.add_issue_comment", + "github.assign_copilot_to_issue" + ], + "hit1": "github.update_issue_comment", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.1111111111111111, + "ndcg@10": 0.3010299956639812, + "hit_rank": 9, + "k3_calls": 3, + "k3_tokens": 1027, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 1107, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1083, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "kind": "ambiguous", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "ranking": [ + "github.merge_pull_request", + "github.update_pull_request_branch", + "gitlab.gitlab_create_merge_request", + "playwright.browser_navigate", + "github.pull_request_review_write", + "gitlab.gitlab_get_merge_request", + "memory.delete_entities", + "github.create_pull_request", + "gitlab.gitlab_list_merge_requests", + "memory.create_relations" + ], + "hit1": "github.merge_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 337, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 531, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 936, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "kind": "verb_product", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "ranking": [ + "github.actions_run_trigger", + "github.delete_file", + "github.pull_request_review_write", + "github.get_discussion_comments", + "github.get_discussion", + "github.list_discussions", + "github.issue_write", + "github.star_repository", + "github.create_gist", + "github.update_gist" + ], + "hit1": "github.actions_run_trigger", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 398, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 578, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1011, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "kind": "resource", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "ranking": [ + "github.list_dependabot_alerts", + "github.list_secret_scanning_alerts", + "github.get_dependabot_alert", + "github.list_code_scanning_alerts" + ], + "hit1": "github.list_dependabot_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 421, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 827, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "kind": "verb_product", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_issues", + "sentry.sentry_list_projects", + "jira.jira_search_issues", + "github.list_issues", + "sentry.sentry_update_issue", + "sentry.sentry_get_issue", + "github.search_issues", + "sentry.sentry_list_events", + "linear.linear_list_issues", + "gitlab.gitlab_list_issues" + ], + "hit1": "sentry.sentry_list_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 813, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_customer", + "stripe.stripe_list_subscriptions", + "stripe.stripe_list_invoices", + "stripe.stripe_create_customer", + "stripe.stripe_list_customers", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_price", + "stripe.stripe_create_product" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.16666666666666666, + "ndcg@10": 0.3562071871080222, + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 407, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 678, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "kind": "paraphrase", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_refund", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_payment_intent", + "stripe.stripe_list_customers", + "stripe.stripe_create_price", + "stripe.stripe_create_invoice", + "stripe.stripe_create_product", + "stripe.stripe_get_customer", + "stripe.stripe_list_subscriptions", + "stripe.stripe_cancel_subscription" + ], + "hit1": "stripe.stripe_create_refund", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "kind": "verb_product", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "ranking": [ + "gmail.gmail_send_email", + "puppeteer.puppeteer_click", + "puppeteer.puppeteer_fill", + "slack.slack_add_reaction", + "stripe.stripe_list_customers", + "puppeteer.puppeteer_navigate", + "jira.jira_search_issues", + "asana.asana_get_task", + "salesforce.salesforce_create_lead", + "gmail.gmail_modify_email" + ], + "hit1": "gmail.gmail_send_email", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 392, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 775, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "kind": "verb_product", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "ranking": [ + "mysql.mysql_query", + "gmail.gmail_send_email", + "mysql.mysql_list_tables", + "gmail.gmail_modify_email", + "gmail.gmail_trash_email", + "gmail.gmail_get_email", + "gmail.gmail_list_labels", + "gmail.gmail_create_draft", + "gmail.gmail_search_emails" + ], + "hit1": "mysql.mysql_query", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.1111111111111111, + "ndcg@10": 0.3010299956639812, + "hit_rank": 9, + "k3_calls": 3, + "k3_tokens": 642, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 676, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 656, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "kind": "paraphrase", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "ranking": [ + "slack.slack_upload_file", + "aws_s3.s3_put_object", + "filesystem.edit_file", + "git.git_add", + "github.delete_file", + "slack.slack_add_reaction", + "figma.figma_post_comment", + "puppeteer.puppeteer_navigate", + "figma.figma_get_comments", + "discord.discord_add_reaction" + ], + "hit1": "slack.slack_upload_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 257, + "k3_calls": 86, + "k3_tokens": 20636, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 52, + "k5_tokens": 20030, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 26, + "k10_tokens": 19458, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "github.pull_request_review_write", + "google_calendar.gcal_list_calendars", + "google_calendar.gcal_list_events" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 481, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 820, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_list_pods", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_delete", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_list_services", + "kubernetes.k8s_apply", + "kubernetes.k8s_get_events", + "kubernetes.k8s_get_pod" + ], + "hit1": "kubernetes.k8s_list_pods", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 719, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_pod_logs", + "datadog.datadog_list_logs", + "docker.docker_logs", + "git.git_log", + "github.delete_file", + "github.actions_get_job_logs", + "mongodb.mongo_delete", + "kubernetes.k8s_get_pod", + "datadog.datadog_query_metrics", + "github.get_file_contents" + ], + "hit1": "kubernetes.k8s_pod_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 913, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "ranking": [ + "docker.docker_run", + "docker.docker_rm", + "docker.docker_stop", + "docker.docker_exec", + "docker.docker_logs", + "docker.docker_build", + "kubernetes.k8s_pod_logs", + "docker.docker_ps", + "docker.docker_images" + ], + "hit1": "docker.docker_run", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "kind": "verb_product", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "ranking": [ + "docker.docker_logs", + "docker.docker_run", + "docker.docker_rm", + "kubernetes.k8s_pod_logs", + "docker.docker_stop", + "docker.docker_exec", + "git.git_log", + "github.actions_get_job_logs", + "docker.docker_build", + "datadog.datadog_list_logs" + ], + "hit1": "docker.docker_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "kind": "ambiguous", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "ranking": [ + "playwright.browser_navigate", + "playwright.browser_snapshot", + "playwright.browser_take_screenshot", + "playwright.browser_click", + "playwright.browser_evaluate", + "playwright.browser_press_key", + "puppeteer.puppeteer_navigate", + "playwright.browser_tab_new", + "memory.delete_entities", + "memory.create_relations" + ], + "hit1": "playwright.browser_navigate", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8868854556705132, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 561, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "kind": "ambiguous", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "ranking": [ + "playwright.browser_take_screenshot", + "puppeteer.puppeteer_screenshot", + "filesystem.list_directory", + "slack.slack_set_channel_topic", + "github.update_pull_request", + "filesystem.directory_tree", + "sentry.sentry_update_issue", + "playwright.browser_snapshot", + "git.git_status", + "github.get_me" + ], + "hit1": "playwright.browser_take_screenshot", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 442, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 730, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_create_merge_request", + "gitlab.gitlab_get_merge_request", + "gitlab.gitlab_list_merge_requests", + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_create_pipeline", + "gitlab.gitlab_get_issue", + "gitlab.gitlab_list_projects", + "gitlab.gitlab_get_file", + "gitlab.gitlab_list_issues", + "github.merge_pull_request" + ], + "hit1": "gitlab.gitlab_create_merge_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 291, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 441, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 833, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "kind": "verb_product", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "ranking": [ + "notion.notion_search", + "notion.create_comment", + "notion.get_data_source", + "notion.list_users_tokens", + "notion.notion_append_block", + "notion.delete_users_token", + "notion.get_users_token", + "notion.create_data_source", + "notion.notion_get_comments", + "notion.update_data_source" + ], + "hit1": "notion.notion_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 393, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 734, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "kind": "verb_product", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "ranking": [ + "notion.notion_get_comments", + "notion.notion_update_page", + "notion.notion_create_page", + "notion.notion_get_page", + "notion.create_comment", + "github.search_code", + "notion.get_data_source", + "confluence.confluence_get_page", + "notion.list_users_tokens", + "playwright.browser_snapshot" + ], + "hit1": "notion.notion_get_comments", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "kind": "verb_product", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "ranking": [ + "notion.notion_create_database", + "notion.notion_query_database", + "notion.notion_create_page", + "notion.create_comment", + "mongodb.mongo_delete", + "notion.get_data_source", + "mongodb.mongo_find", + "notion.list_users_tokens", + "mongodb.mongo_update", + "notion.notion_append_block" + ], + "hit1": "notion.notion_create_database", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 279, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 450, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 821, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_search", + "jira.jira_create_issue", + "zendesk.zendesk_list_tickets" + ], + "hit1": "zendesk.zendesk_get_ticket", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_get_contact", + "hubspot.hubspot_create_contact", + "hubspot.hubspot_search_contacts", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_create_deal", + "github.delete_file", + "github.pull_request_review_write", + "github.get_discussion_comments", + "github.get_discussion", + "github.list_discussions" + ], + "hit1": "hubspot.hubspot_get_contact", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 873, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "kind": "verb_product", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_create_lead", + "salesforce.salesforce_convert_lead", + "salesforce.salesforce_get_record", + "salesforce.salesforce_update_record", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_list_opportunities", + "salesforce.salesforce_create_record", + "salesforce.salesforce_query" + ], + "hit1": "salesforce.salesforce_create_lead", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 423, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 696, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "kind": "identifier", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_query" + ], + "hit1": "salesforce.salesforce_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 173, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "kind": "verb_product", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "ranking": [ + "shopify.shopify_list_orders", + "shopify.shopify_list_products", + "shopify.shopify_update_product", + "shopify.shopify_create_product", + "shopify.shopify_get_product", + "shopify.shopify_create_order", + "shopify.shopify_list_customers", + "shopify.shopify_get_order" + ], + "hit1": "shopify.shopify_list_orders", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 376, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_list_oncalls", + "pagerduty.pagerduty_list_incidents" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 486, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 387, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 715, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "kind": "paraphrase", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "ranking": [ + "slack.slack_create_channel", + "figma.figma_post_comment", + "filesystem.list_allowed_directories", + "github.get_discussion_comments", + "figma.figma_get_comments", + "pagerduty.pagerduty_list_oncalls", + "github.create_pull_request", + "gmail.gmail_modify_email", + "trello.trello_list_cards", + "trello.trello_add_comment" + ], + "hit1": "slack.slack_create_channel", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.16666666666666666, + "ndcg@10": 0.3562071871080222, + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 442, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 833, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 810, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "kind": "verb_product", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "ranking": [ + "datadog.datadog_query_metrics", + "datadog.datadog_create_monitor", + "prometheus.prom_list_metrics", + "datadog.datadog_list_logs", + "datadog.datadog_list_monitors", + "datadog.datadog_get_monitor" + ], + "hit1": "datadog.datadog_query_metrics", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 287, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 449, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "kind": "verb_product", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "ranking": [ + "datadog.datadog_list_logs", + "datadog.datadog_query_metrics", + "git.git_log", + "datadog.datadog_create_monitor", + "github.actions_get_job_logs", + "kubernetes.k8s_pod_logs", + "datadog.datadog_list_monitors", + "datadog.datadog_get_monitor", + "docker.docker_logs" + ], + "hit1": "datadog.datadog_list_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 492, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 828, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_list_buckets" + ], + "hit1": "aws_s3.s3_list_buckets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 153, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 311, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "kind": "paraphrase", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_put_object", + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "slack.slack_add_reaction", + "slack.slack_upload_file", + "gmail.gmail_send_email", + "asana.asana_get_task", + "puppeteer.puppeteer_click", + "elasticsearch.es_search", + "okta.okta_create_user" + ], + "hit1": "aws_s3.s3_put_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 423, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns" + ], + "hit1": "cloudflare.cf_purge_cache", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "cloudflare.cf_list_zones", + "salesforce.salesforce_get_record", + "cloudflare.cf_purge_cache", + "salesforce.salesforce_update_record", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_create_record", + "git.git_commit" + ], + "hit1": "cloudflare.cf_create_dns", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_deployments", + "vercel.vercel_list_projects", + "vercel.vercel_get_deployment", + "kubernetes.k8s_list_deployments", + "vercel.vercel_create_project" + ], + "hit1": "vercel.vercel_list_deployments", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 648, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "ranking": [ + "supabase.supabase_insert", + "mongodb.mongo_insert", + "playwright.browser_type" + ], + "hit1": "supabase.supabase_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 332, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 768, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "kind": "paraphrase", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "ranking": [ + "figma.figma_get_images", + "github.delete_file", + "docker.docker_images", + "mongodb.mongo_delete", + "datadog.datadog_query_metrics", + "github.get_file_contents", + "memory.delete_entities", + "twilio.twilio_send_sms", + "filesystem.read_media_file", + "memory.delete_observations" + ], + "hit1": "figma.figma_get_images", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 431, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 789, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "kind": "verb_product", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "ranking": [ + "puppeteer.puppeteer_click", + "twilio.twilio_send_sms", + "puppeteer.puppeteer_fill", + "slack.slack_add_reaction", + "puppeteer.puppeteer_navigate", + "gmail.gmail_send_email", + "twilio.twilio_list_messages", + "jira.jira_search_issues", + "asana.asana_get_task", + "puppeteer.puppeteer_evaluate" + ], + "hit1": "puppeteer.puppeteer_click", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "kind": "verb_product", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "ranking": [ + "okta.okta_create_user", + "okta.okta_deactivate_user", + "okta.okta_get_user", + "slack.slack_add_reaction", + "gmail.gmail_send_email", + "asana.asana_get_task", + "okta.okta_list_users", + "github.list_gists", + "puppeteer.puppeteer_click", + "linear.get_user" + ], + "hit1": "okta.okta_create_user", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 625, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query", + "prometheus.prom_query_range", + "prometheus.prom_list_metrics" + ], + "hit1": "prometheus.prom_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 313, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 749, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "kind": "verb_product", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "ranking": [ + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "grafana.grafana_create_annotation" + ], + "hit1": "grafana.grafana_search_dashboards", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 309, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 745, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "kind": "verb_product", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "ranking": [ + "circleci.circleci_trigger_pipeline", + "circleci.circleci_list_pipelines", + "circleci.circleci_get_pipeline", + "github.actions_run_trigger", + "gitlab.gitlab_create_pipeline" + ], + "hit1": "circleci.circleci_trigger_pipeline", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 747, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "kind": "identifier", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "ranking": [ + "terraform.tf_plan", + "terraform.tf_state_list", + "postgres.explain_query", + "terraform.tf_apply" + ], + "hit1": "terraform.tf_plan", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 166, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "kind": "verb_product", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "ranking": [ + "asana.asana_get_task", + "asana.asana_update_task", + "asana.asana_create_task", + "asana.asana_create_project", + "slack.slack_add_reaction", + "gmail.gmail_send_email", + "asana.asana_list_tasks", + "clickup.clickup_update_task", + "clickup.clickup_create_task", + "puppeteer.puppeteer_click" + ], + "hit1": "asana.asana_get_task", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 824, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "ranking": [ + "trello.trello_add_comment", + "trello.trello_create_card", + "trello.trello_create_board", + "trello.trello_list_cards" + ], + "hit1": "trello.trello_add_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 776, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_update_task", + "clickup.clickup_create_task", + "clickup.clickup_get_task", + "clickup.clickup_list_tasks", + "puppeteer.puppeteer_click", + "asana.asana_get_task", + "asana.asana_update_task", + "playwright.browser_click", + "asana.asana_create_task" + ], + "hit1": "clickup.clickup_update_task", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "kind": "verb_product", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "ranking": [ + "confluence.confluence_get_page", + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "github.search_code", + "playwright.browser_snapshot", + "confluence.confluence_search", + "notion.notion_get_comments", + "github.search_repositories", + "playwright.browser_take_screenshot", + "notion.notion_update_page" + ], + "hit1": "confluence.confluence_get_page", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 710, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "kind": "verb_product", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "ranking": [ + "discord.discord_add_reaction", + "discord.discord_send_message", + "slack.slack_add_reaction", + "discord.discord_get_channel", + "discord.discord_list_channels", + "github.delete_file", + "figma.figma_post_comment", + "discord.discord_list_members", + "gmail.gmail_modify_email", + "slack.slack_post_message" + ], + "hit1": "discord.discord_add_reaction", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 265, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_index_doc", + "elasticsearch.es_search", + "elasticsearch.es_get_doc" + ], + "hit1": "elasticsearch.es_index_doc", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 767, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "redis.redis_set", + "redis.redis_keys", + "figma.figma_post_comment", + "jira.jira_get_issue", + "figma.figma_get_comments", + "redis.redis_delete", + "jira.jira_add_worklog", + "jira.jira_add_comment", + "aws_s3.s3_get_object" + ], + "hit1": "redis.redis_get", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 310, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 639, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_delete", + "mongodb.mongo_find", + "mongodb.mongo_update", + "mongodb.mongo_insert", + "mongodb.mongo_list_collections" + ], + "hit1": "mongodb.mongo_delete", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "kind": "verb_product", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "ranking": [ + "sqlite.sqlite_list_tables", + "sqlite.sqlite_query", + "sqlite.sqlite_describe_table", + "mysql.mysql_list_tables", + "postgres.list_tables", + "supabase.supabase_list_tables" + ], + "hit1": "sqlite.sqlite_list_tables", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 156, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 252, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 542, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "kind": "verb_product", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "ranking": [ + "postgres.describe_table", + "sqlite.sqlite_describe_table", + "postgres.list_schemas", + "supabase.supabase_insert", + "postgres.list_tables", + "supabase.supabase_update", + "postgres.query", + "postgres.explain_query", + "supabase.supabase_select" + ], + "hit1": "postgres.describe_table", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 170, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 286, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.issue_write", + "github.create_issue", + "github.update_issue_comment", + "github.issue_read", + "github.add_issue_comment", + "github.assign_copilot_to_issue", + "github.list_check_runs", + "jira.jira_get_issue", + "github.delete_file", + "github.pull_request_review_write" + ], + "hit1": "github.issue_write", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8597186998521972, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 392, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 596, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1087, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "ranking": [ + "jira.jira_get_issue", + "github.issue_write", + "linear.linear_get_issue", + "jira.jira_add_worklog", + "linear.linear_update_issue", + "linear.linear_create_issue", + "linear.linear_create_comment", + "github.create_issue", + "gitlab.gitlab_get_issue", + "sentry.sentry_update_issue" + ], + "hit1": "jira.jira_get_issue", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.16666666666666666, + "ndcg@10": 0.31528556000129054, + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 616, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 976, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 958, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "kind": "identifier", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "ranking": [ + "github.get_file_contents", + "filesystem.list_directory", + "filesystem.directory_tree", + "filesystem.create_directory", + "filesystem.get_file_info", + "filesystem.list_directory_with_sizes" + ], + "hit1": "github.get_file_contents", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 653, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "kind": "paraphrase", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "ranking": [ + "sequential_thinking.sequentialthinking" + ], + "hit1": "sequential_thinking.sequentialthinking", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "kind": "paraphrase", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "ranking": [ + "time.convert_time", + "slack.slack_create_channel", + "jira.jira_add_worklog", + "filesystem.list_allowed_directories", + "playwright.browser_wait_for", + "time.get_current_time", + "google_calendar.gcal_list_events", + "grafana.grafana_create_annotation" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.6508780296742857, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 369, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 685, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "kind": "verb_product", + "gold": [ + "time.convert_time" + ], + "alts": [], + "ranking": [ + "time.convert_time", + "salesforce.salesforce_convert_lead", + "jira.jira_add_worklog", + "memory.create_relations", + "playwright.browser_wait_for", + "git.git_diff", + "time.get_current_time", + "google_calendar.gcal_list_events", + "grafana.grafana_create_annotation" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "kind": "verb_product", + "gold": [ + "github.star_repository" + ], + "alts": [], + "ranking": [ + "github.star_repository", + "github.delete_file", + "github.pull_request_review_write", + "github.get_discussion_comments", + "github.get_discussion", + "github.list_discussions", + "github.issue_write", + "github.create_gist", + "github.update_gist", + "github.list_releases" + ], + "hit1": "github.star_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 345, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 525, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 963, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "kind": "resource", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "ranking": [ + "github.fork_repository", + "github.delete_file", + "github.list_discussions", + "github.star_repository", + "github.list_releases", + "github.list_tags", + "github.get_file_contents", + "github.list_labels", + "github.list_issues", + "github.unstar_repository" + ], + "hit1": "github.fork_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 287, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 426, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 845, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "kind": "verb_product", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "ranking": [ + "github.get_latest_release", + "github.list_notifications", + "git.git_status", + "github.get_me", + "git.git_add", + "github.get_repository_tree", + "github.get_teams", + "git.git_commit", + "playwright.browser_navigate", + "github.delete_file" + ], + "hit1": "github.get_latest_release", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 210, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "kind": "resource", + "gold": [ + "github.create_gist" + ], + "alts": [], + "ranking": [ + "github.create_gist", + "github.update_gist", + "github.get_gist" + ], + "hit1": "github.create_gist", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "kind": "verb_product", + "gold": [ + "github.search_code" + ], + "alts": [], + "ranking": [ + "github.search_code", + "github.list_code_scanning_alerts", + "github.delete_file", + "github.pull_request_review_write", + "github.get_discussion_comments", + "github.get_discussion", + "github.list_discussions", + "github.get_code_scanning_alert", + "github.issue_write", + "github.star_repository" + ], + "hit1": "github.search_code", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 536, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 995, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "kind": "verb_product", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "ranking": [ + "github.list_notifications", + "github.mark_all_notifications_read", + "github.delete_file", + "github.pull_request_review_write", + "github.get_discussion_comments", + "github.get_discussion", + "github.list_discussions", + "github.issue_write", + "github.star_repository", + "github.create_gist" + ], + "hit1": "github.list_notifications", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 473, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 913, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "kind": "verb_product", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "ranking": [ + "github.assign_copilot_to_issue", + "linear.linear_assign_issue", + "github.request_copilot_review", + "slack.slack_add_reaction", + "puppeteer.puppeteer_navigate", + "jira.jira_assign_issue", + "discord.discord_add_reaction", + "datadog.datadog_query_metrics", + "gmail.gmail_send_email", + "jira.jira_add_worklog" + ], + "hit1": "github.assign_copilot_to_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 301, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 448, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 887, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "kind": "resource", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "ranking": [ + "github.list_secret_scanning_alerts", + "github.get_secret_scanning_alert", + "github.list_code_scanning_alerts", + "github.list_dependabot_alerts", + "github.get_code_scanning_alert" + ], + "hit1": "github.list_secret_scanning_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 827, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "ranking": [ + "figma.figma_post_comment", + "figma.figma_get_comments", + "github.get_file_contents", + "gmail.gmail_modify_email", + "trello.trello_list_cards", + "google_drive.gdrive_get_file", + "filesystem.move_file", + "filesystem.read_media_file", + "github.create_or_update_file", + "filesystem.write_file" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.1111111111111111, + "ndcg@10": 0.22883924989280793, + "hit_rank": 9, + "k3_calls": 3, + "k3_tokens": 798, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 844, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 820, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "kind": "resource", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "ranking": [ + "github.list_repository_collaborators", + "github.get_discussion_comments", + "github.create_pull_request", + "github.list_commits", + "github.update_issue_comment", + "github.request_copilot_review", + "github.assign_copilot_to_issue", + "figma.figma_post_comment", + "github.delete_file", + "github.pull_request_review_write" + ], + "hit1": "github.list_repository_collaborators", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 342, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 550, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1085, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "kind": "verb_product", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "ranking": [ + "slack.list_bookmarks", + "slack.slack_reply_to_thread", + "slack.get_canvase", + "slack.delete_canvase", + "slack.update_bookmark", + "slack.delete_usergroup", + "slack.create_file", + "slack.update_usergroup", + "slack.create_usergroup", + "slack.update_pin" + ], + "hit1": "slack.list_bookmarks", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 769, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "kind": "verb_product", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "ranking": [ + "slack.slack_add_reaction", + "discord.discord_add_reaction", + "gmail.gmail_send_email", + "asana.asana_get_task", + "puppeteer.puppeteer_click", + "elasticsearch.es_search", + "okta.okta_create_user", + "okta.okta_deactivate_user", + "github.create_issue", + "aws_s3.s3_list_objects" + ], + "hit1": "slack.slack_add_reaction", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "kind": "verb_product", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "ranking": [ + "slack.slack_search_messages", + "slack.slack_get_channel_history", + "slack.slack_add_reaction", + "slack.list_bookmarks", + "twilio.twilio_list_messages", + "slack.get_canvase", + "gmail.gmail_search_emails", + "slack.delete_canvase", + "slack.slack_list_users", + "slack.update_bookmark" + ], + "hit1": "slack.slack_search_messages", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "kind": "verb_product", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "ranking": [ + "slack.slack_add_reaction", + "slack.slack_reply_to_thread", + "slack.slack_post_message", + "slack.slack_set_channel_topic", + "slack.slack_create_channel", + "slack.slack_get_channel_history", + "slack.slack_get_thread_replies", + "slack.slack_leave_channel", + "slack.slack_join_channel", + "slack.slack_upload_file" + ], + "hit1": "slack.slack_add_reaction", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.2, + "ndcg@10": 0.38685280723454163, + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 507, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 414, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "kind": "verb_product", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_transition_issue", + "jira.jira_get_issue", + "jira.jira_add_worklog", + "jira.jira_add_comment", + "jira.jira_create_issue", + "jira.jira_update_issue", + "jira.jira_assign_issue", + "jira.jira_search_issues", + "jira.update_sprint", + "github.issue_write" + ], + "hit1": "jira.jira_transition_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 414, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 823, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "kind": "identifier", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "ranking": [ + "jira.jira_search_issues", + "puppeteer.puppeteer_navigate", + "jira.update_sprint", + "jira.jira_get_issue", + "puppeteer.puppeteer_click", + "jira.delete_version", + "puppeteer.puppeteer_evaluate", + "jira.list_attachments", + "puppeteer.puppeteer_screenshot", + "jira.get_attachment" + ], + "hit1": "jira.jira_search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 636, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "kind": "verb_product", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "ranking": [ + "jira.jira_add_worklog", + "jira.jira_add_comment", + "jira.jira_transition_issue", + "slack.slack_add_reaction", + "jira.jira_search_issues", + "puppeteer.puppeteer_navigate", + "jira.update_sprint", + "discord.discord_add_reaction", + "jira.jira_get_issue", + "datadog.datadog_query_metrics" + ], + "hit1": "jira.jira_add_worklog", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 411, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "kind": "verb_product", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "ranking": [ + "linear.linear_list_teams", + "linear.linear_create_project", + "linear.list_users", + "github.get_teams", + "linear.linear_get_issue", + "linear.update_label", + "linear.linear_update_issue", + "linear.create_comment", + "linear.update_cycle", + "linear.linear_create_issue" + ], + "hit1": "linear.linear_list_teams", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 773, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "linear.create_comment", + "linear.linear_create_comment", + "trello.trello_add_comment", + "linear.delete_comment", + "github.update_issue_comment", + "linear.update_comment", + "linear.get_comment", + "notion.create_comment", + "linear.linear_create_project" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 778, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "kind": "paraphrase", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "ranking": [ + "filesystem.move_file", + "filesystem.read_media_file", + "filesystem.read_file", + "filesystem.read_text_file", + "brave_search.brave_local_search", + "github.delete_file", + "gmail.gmail_trash_email", + "figma.figma_post_comment", + "figma.figma_get_comments", + "slack.create_file" + ], + "hit1": "filesystem.move_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 754, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "kind": "paraphrase", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "ranking": [ + "filesystem.search_files", + "google_drive.gdrive_search", + "salesforce.salesforce_get_record", + "github.create_gist", + "jira.jira_get_issue", + "github.update_gist", + "linear.linear_get_issue", + "filesystem.list_directory", + "github.search_repositories", + "filesystem.move_file" + ], + "hit1": "filesystem.search_files", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 351, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 693, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "kind": "paraphrase", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "ranking": [ + "filesystem.edit_file", + "slack.create_file", + "google_drive.gdrive_create_file", + "github.create_or_update_file", + "github.get_file_blame", + "slack.get_file", + "github.delete_file", + "github.list_check_runs", + "slack.update_file", + "figma.figma_post_comment" + ], + "hit1": "filesystem.edit_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 323, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 565, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 950, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "ranking": [ + "memory.delete_entities", + "memory.create_relations", + "memory.search_nodes", + "memory.add_observations", + "memory.create_entities", + "memory.open_nodes", + "memory.delete_relations", + "memory.read_graph", + "playwright.browser_navigate", + "slack.slack_set_channel_topic" + ], + "hit1": "memory.delete_entities", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.125, + "ndcg@10": 0.31546487678572877, + "hit_rank": 8, + "k3_calls": 3, + "k3_tokens": 511, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 563, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 542, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "kind": "verb_product", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "memory.open_nodes", + "memory.delete_entities", + "memory.create_relations", + "figma.figma_get_file_nodes", + "memory.delete_observations", + "memory.add_observations", + "memory.create_entities", + "memory.delete_relations", + "memory.read_graph" + ], + "hit1": "memory.search_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 305, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 546, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "kind": "verb_product", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "ranking": [ + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_diff", + "github.delete_file", + "github.pull_request_review_write", + "git.git_show", + "github.get_discussion_comments", + "git.git_branch", + "github.get_discussion", + "git.git_log" + ], + "hit1": "git.git_diff_unstaged", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 447, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "kind": "identifier", + "gold": [ + "git.git_add" + ], + "alts": [], + "ranking": [ + "filesystem.list_directory_with_sizes", + "salesforce.salesforce_list_opportunities", + "google_drive.gdrive_search", + "puppeteer.puppeteer_navigate", + "github.create_gist", + "jira.jira_search_issues", + "github.update_gist", + "puppeteer.puppeteer_click", + "filesystem.list_directory", + "puppeteer.puppeteer_evaluate" + ], + "hit1": "filesystem.list_directory_with_sizes", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 115, + "k3_calls": 39, + "k3_tokens": 9731, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 23, + "k5_tokens": 9285, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9371, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "kind": "verb_product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "ranking": [ + "shopify.shopify_update_product", + "shopify.shopify_create_product", + "shopify.shopify_get_product", + "shopify.shopify_list_products", + "shopify.shopify_list_orders", + "stripe.stripe_create_price", + "stripe.stripe_create_product", + "shopify.shopify_create_order", + "shopify.shopify_list_customers", + "shopify.shopify_get_order" + ], + "hit1": "shopify.shopify_update_product", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "kind": "verb_product", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "ranking": [ + "stripe.stripe_cancel_subscription", + "stripe.stripe_list_customers", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_price", + "stripe.stripe_create_invoice", + "stripe.stripe_create_product", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_customer", + "stripe.stripe_create_refund", + "stripe.stripe_list_subscriptions" + ], + "hit1": "stripe.stripe_cancel_subscription", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "kind": "resource", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "ranking": [ + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_refund" + ], + "hit1": "stripe.stripe_get_payment_intent", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 326, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 762, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_list_oncalls", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_create_incident" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "kind": "verb_product", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "ranking": [ + "datadog.datadog_create_monitor", + "datadog.datadog_get_monitor", + "datadog.datadog_query_metrics", + "datadog.datadog_list_logs", + "datadog.datadog_list_monitors" + ], + "hit1": "datadog.datadog_create_monitor", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 816, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "kind": "paraphrase", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "ranking": [ + "sentry.sentry_update_issue", + "sentry.sentry_get_issue", + "sentry.sentry_list_events", + "pagerduty.pagerduty_resolve_incident", + "sentry.sentry_list_projects", + "jira.jira_get_issue", + "sentry.sentry_list_issues", + "github.issue_write", + "linear.linear_get_issue", + "jira.jira_add_worklog" + ], + "hit1": "sentry.sentry_update_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_scale_deployment", + "vercel.vercel_get_deployment", + "kubernetes.k8s_delete", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_list_services", + "kubernetes.k8s_apply", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_get_events", + "kubernetes.k8s_get_pod" + ], + "hit1": "kubernetes.k8s_scale_deployment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 266, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 420, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_apply", + "kubernetes.k8s_scale_deployment", + "terraform.tf_apply", + "kubernetes.k8s_delete", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_list_services", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_get_events", + "kubernetes.k8s_get_pod" + ], + "hit1": "kubernetes.k8s_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 246, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "ranking": [ + "docker.docker_stop", + "docker.docker_run", + "docker.docker_rm", + "docker.docker_exec", + "docker.docker_logs", + "docker.docker_build", + "kubernetes.k8s_pod_logs", + "docker.docker_ps", + "docker.docker_images" + ], + "hit1": "docker.docker_stop", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "kind": "verb_product", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "ranking": [ + "docker.docker_build", + "docker.docker_run", + "docker.docker_rm", + "filesystem.read_media_file", + "docker.docker_stop", + "docker.docker_ps", + "docker.docker_exec", + "docker.docker_images", + "docker.docker_logs" + ], + "hit1": "docker.docker_build", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "kind": "verb_product", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "ranking": [ + "puppeteer.puppeteer_click", + "playwright.browser_click", + "playwright.browser_take_screenshot", + "playwright.browser_type", + "slack.slack_add_reaction", + "clickup.clickup_update_task", + "gmail.gmail_send_email", + "clickup.clickup_list_tasks", + "asana.asana_get_task", + "clickup.clickup_create_task" + ], + "hit1": "puppeteer.puppeteer_click", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 760, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "kind": "paraphrase", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "ranking": [ + "playwright.browser_type", + "google_drive.gdrive_create_file", + "slack.slack_reply_to_thread", + "supabase.supabase_insert", + "github.projects_list", + "slack.slack_post_message", + "mongodb.mongo_insert", + "datadog.datadog_create_monitor", + "trello.trello_add_comment", + "cloudflare.cf_create_dns" + ], + "hit1": "playwright.browser_type", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 296, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 862, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_list_projects", + "gitlab.gitlab_create_merge_request", + "github.projects_list", + "gitlab.gitlab_create_pipeline", + "gitlab.gitlab_get_issue", + "github.projects_write", + "gitlab.gitlab_get_file", + "gitlab.gitlab_get_merge_request", + "gitlab.gitlab_list_issues", + "gitlab.gitlab_list_merge_requests" + ], + "hit1": "gitlab.gitlab_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 398, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "kind": "verb_product", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "ranking": [ + "bitbucket.bitbucket_create_pr", + "github.pull_request_review_write", + "bitbucket.bitbucket_list_prs", + "github.create_pull_request", + "github.add_comment_to_pending_review", + "github.update_pull_request", + "github.pull_request_read", + "github.request_copilot_review", + "github.update_pull_request_branch", + "github.merge_pull_request" + ], + "hit1": "bitbucket.bitbucket_create_pr", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 365, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 669, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1202, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "kind": "verb_product", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "ranking": [ + "gmail.gmail_create_draft", + "gmail.gmail_send_email", + "github.create_pull_request", + "gmail.gmail_modify_email", + "gmail.gmail_trash_email", + "gmail.gmail_get_email", + "gmail.gmail_list_labels", + "gmail.gmail_search_emails" + ], + "hit1": "gmail.gmail_create_draft", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 344, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 494, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 744, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "kind": "verb_product", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_share_file", + "google_drive.gdrive_search", + "google_drive.gdrive_create_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_list_folder", + "google_drive.gdrive_update_file", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_list_calendars" + ], + "hit1": "google_drive.gdrive_share_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 270, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 791, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "kind": "verb_product", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_list_events", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_list_calendars", + "sentry.sentry_list_events", + "google_drive.gdrive_search", + "google_drive.gdrive_create_file", + "kubernetes.k8s_get_events", + "google_drive.gdrive_get_file" + ], + "hit1": "google_calendar.gcal_list_events", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 770, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "github.pull_request_review_write", + "google_calendar.gcal_list_calendars", + "google_calendar.gcal_list_events" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 481, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 820, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_projects", + "vercel.vercel_get_deployment", + "sentry.sentry_list_projects", + "vercel.vercel_list_deployments", + "jira.jira_list_projects", + "vercel.vercel_create_project", + "github.projects_list", + "github.projects_write", + "gitlab.gitlab_list_projects", + "linear.linear_list_projects" + ], + "hit1": "vercel.vercel_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 170, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 567, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "ranking": [ + "supabase.supabase_select", + "supabase.supabase_insert", + "github.delete_file", + "supabase.supabase_update", + "mongodb.mongo_delete", + "datadog.datadog_query_metrics", + "github.get_file_contents", + "memory.delete_entities", + "twilio.twilio_send_sms", + "filesystem.read_media_file" + ], + "hit1": "supabase.supabase_select", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 425, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 830, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "kind": "verb_product", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "notion.create_comment", + "figma.figma_get_comments", + "figma.figma_get_file", + "jira.jira_add_worklog", + "figma.figma_get_images", + "linear.create_comment", + "figma.figma_get_file_nodes", + "github.add_comment_to_pending_review", + "linear.linear_create_comment" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 871, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "kind": "verb_product", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "ranking": [ + "okta.okta_list_users", + "okta.okta_create_user", + "slack.slack_list_users", + "okta.okta_deactivate_user", + "linear.list_users", + "pagerduty.pagerduty_list_oncalls", + "okta.okta_get_user", + "notion.list_users_tokens", + "notion.delete_users_token", + "notion.get_users_token" + ], + "hit1": "okta.okta_list_users", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 625, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query_range", + "prometheus.prom_query", + "prometheus.prom_list_metrics" + ], + "hit1": "prometheus.prom_query_range", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 313, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 749, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "kind": "verb_product", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "ranking": [ + "grafana.grafana_create_annotation", + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard" + ], + "hit1": "grafana.grafana_create_annotation", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 310, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "kind": "identifier", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "ranking": [ + "terraform.tf_apply", + "kubernetes.k8s_apply", + "terraform.tf_state_list", + "terraform.tf_plan" + ], + "hit1": "terraform.tf_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "kind": "verb_product", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "ranking": [ + "asana.asana_list_tasks", + "asana.asana_get_task", + "clickup.clickup_list_tasks", + "asana.asana_update_task", + "asana.asana_create_task", + "asana.asana_create_project" + ], + "hit1": "asana.asana_list_tasks", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "ranking": [ + "trello.trello_add_comment", + "figma.figma_post_comment", + "trello.trello_list_cards", + "github.update_issue_comment", + "notion.create_comment", + "trello.trello_create_board", + "github.get_discussion_comments", + "jira.jira_add_worklog", + "figma.figma_get_comments", + "linear.create_comment" + ], + "hit1": "trello.trello_add_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 444, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 835, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_update_task", + "clickup.clickup_create_task", + "clickup.clickup_get_task", + "clickup.clickup_list_tasks", + "puppeteer.puppeteer_click", + "asana.asana_get_task", + "asana.asana_update_task", + "playwright.browser_click", + "asana.asana_create_task" + ], + "hit1": "clickup.clickup_update_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "kind": "verb_product", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "ranking": [ + "confluence.confluence_get_page", + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "confluence.confluence_search" + ], + "hit1": "confluence.confluence_get_page", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 423, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 772, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_deal", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_create_contact", + "hubspot.hubspot_search_contacts", + "hubspot.hubspot_list_contacts", + "github.delete_file", + "github.pull_request_review_write", + "github.get_discussion_comments", + "github.get_discussion", + "github.list_discussions" + ], + "hit1": "hubspot.hubspot_create_deal", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 873, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_search", + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_create_ticket" + ], + "hit1": "zendesk.zendesk_search", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "kind": "verb_product", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "ranking": [ + "discord.discord_list_channels", + "discord.discord_add_reaction", + "slack.slack_list_channels", + "discord.discord_get_channel", + "discord.discord_send_message", + "discord.discord_list_members" + ], + "hit1": "discord.discord_list_channels", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 638, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "redis.redis_set", + "redis.redis_keys", + "figma.figma_post_comment", + "jira.jira_get_issue", + "figma.figma_get_comments", + "redis.redis_delete", + "jira.jira_add_worklog", + "jira.jira_add_comment", + "aws_s3.s3_get_object" + ], + "hit1": "redis.redis_get", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 310, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 639, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert", + "mongodb.mongo_delete", + "mongodb.mongo_find", + "mongodb.mongo_update", + "mongodb.mongo_list_collections", + "supabase.supabase_insert" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "kind": "verb_product", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "ranking": [ + "mysql.mysql_query", + "mysql.mysql_list_tables", + "postgres.list_schemas", + "postgres.describe_table", + "postgres.list_tables", + "sqlite.sqlite_query", + "postgres.query", + "postgres.explain_query" + ], + "hit1": "mysql.mysql_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 158, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 271, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 523, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "github.list_check_runs", + "elasticsearch.es_search", + "jira.update_sprint", + "gitlab.list_jobs", + "playwright.browser_tab_close", + "slack.list_bookmarks", + "notion.create_comment", + "jira.delete_version" + ], + "hit1": "elasticsearch.es_index_doc", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_get_object", + "aws_s3.s3_put_object", + "aws_s3.s3_delete_object", + "slack.slack_add_reaction", + "gmail.gmail_send_email", + "asana.asana_get_task", + "puppeteer.puppeteer_click", + "elasticsearch.es_search", + "okta.okta_create_user", + "okta.okta_deactivate_user" + ], + "hit1": "aws_s3.s3_get_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 432, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 712, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones", + "cloudflare.cf_purge_cache", + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 745, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_list_customers", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_price", + "stripe.stripe_create_product", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_customer", + "stripe.stripe_create_refund", + "stripe.stripe_list_subscriptions", + "stripe.stripe_cancel_subscription" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 191, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "kind": "verb_product", + "gold": [ + "github.actions_list" + ], + "alts": [], + "ranking": [ + "github.actions_run_trigger", + "github.actions_get_job_logs", + "github.delete_file", + "github.actions_list", + "github.pull_request_review_write", + "github.get_discussion_comments", + "github.get_discussion", + "github.actions_get", + "github.list_discussions", + "github.issue_write" + ], + "hit1": "github.actions_run_trigger", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 672, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 558, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1032, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "kind": "verb_product", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "ranking": [ + "github.actions_run_trigger", + "github.actions_get_job_logs", + "github.delete_file", + "github.actions_list", + "github.pull_request_review_write", + "github.get_discussion_comments", + "github.get_discussion", + "github.actions_get", + "github.list_discussions", + "github.issue_write" + ], + "hit1": "github.actions_run_trigger", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 349, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 559, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1033, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "kind": "verb_product", + "gold": [ + "github.create_repository" + ], + "alts": [], + "ranking": [ + "github.delete_file", + "github.list_discussions", + "github.star_repository", + "github.list_releases", + "github.list_tags", + "github.get_file_contents", + "github.list_labels", + "github.list_issues", + "github.unstar_repository", + "github.list_branches" + ], + "hit1": "github.delete_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 24, + "k3_calls": 8, + "k3_tokens": 2112, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 5, + "k5_tokens": 2116, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 3, + "k10_tokens": 2504, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "kind": "verb_product", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "ranking": [ + "git.git_checkout", + "github.delete_file", + "git.git_branch", + "gitlab.gitlab_create_merge_request", + "github.create_or_update_file", + "git.git_create_branch", + "github.list_commits", + "github.push_files", + "github.create_branch", + "github.pull_request_review_write" + ], + "hit1": "git.git_checkout", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 504, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1049, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "kind": "identifier", + "gold": [ + "git.git_log" + ], + "alts": [], + "ranking": [ + "git.git_log", + "github.delete_file", + "github.pull_request_review_write", + "git.git_show", + "github.get_discussion_comments", + "git.git_branch", + "github.get_discussion", + "github.list_discussions", + "github.issue_write", + "github.star_repository" + ], + "hit1": "git.git_log", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 338, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 492, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 900, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "kind": "verb_product", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "ranking": [ + "postgres.list_schemas", + "postgres.describe_table", + "postgres.list_tables", + "postgres.query", + "postgres.explain_query" + ], + "hit1": "postgres.list_schemas", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 169, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 634, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "kind": "verb_product", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "memory.open_nodes", + "memory.create_relations", + "slack.slack_list_channels", + "time.get_current_time", + "postgres.list_schemas", + "memory.add_observations", + "postgres.list_tables", + "memory.create_entities", + "sqlite.sqlite_list_tables" + ], + "hit1": "memory.search_nodes", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 297, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 523, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "kind": "paraphrase", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "ranking": [ + "github.delete_file", + "google_drive.gdrive_get_file", + "github.get_file_contents", + "filesystem.read_media_file", + "filesystem.get_file_info", + "gitlab.gitlab_get_file", + "filesystem.read_file", + "filesystem.read_text_file", + "figma.figma_get_images", + "figma.figma_get_file_nodes" + ], + "hit1": "github.delete_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.2, + "ndcg@10": 0.38685280723454163, + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 525, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 810, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "kind": "paraphrase", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "ranking": [ + "filesystem.read_media_file", + "filesystem.write_file", + "filesystem.read_file", + "filesystem.edit_file", + "filesystem.read_text_file", + "playwright.browser_tab_new", + "brave_search.brave_local_search", + "github.delete_file", + "github.create_pull_request", + "figma.figma_post_comment" + ], + "hit1": "filesystem.read_media_file", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 847, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "kind": "verb_product", + "gold": [ + "github.search_issues" + ], + "alts": [], + "ranking": [ + "github.list_issues", + "github.search_issues", + "github.delete_file", + "github.pull_request_review_write", + "github.get_discussion_comments", + "github.get_discussion", + "github.list_discussions", + "github.issue_write", + "github.star_repository", + "github.create_gist" + ], + "hit1": "github.list_issues", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 322, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 558, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 998, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "kind": "paraphrase", + "gold": [ + "github.get_me" + ], + "alts": [], + "ranking": [ + "github.list_notifications", + "github.get_me", + "github.get_teams", + "github.get_latest_release", + "git.git_status", + "git.git_add", + "github.get_repository_tree", + "git.git_commit", + "playwright.browser_navigate", + "github.delete_file" + ], + "hit1": "github.list_notifications", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "cost": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 2.506666666666667, + "p95_calls": 3.0, + "success_within_1": 0.8733333333333333, + "success_within_2": 0.9333333333333333, + "success_within_3": 0.9666666666666667, + "mean_tokens": 622.4733333333334, + "p95_tokens": 668.3999999999999, + "total_tokens": 93371, + "tokens_per_successful_query": 622.4733333333334, + "total_tokens_successful": 93371 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.8733333333333333, + "p95_calls": 2.0, + "success_within_1": 0.9133333333333333, + "success_within_2": 0.9666666666666667, + "success_within_3": 0.9666666666666667, + "mean_tokens": 748.3533333333334, + "p95_tokens": 839.0499999999998, + "total_tokens": 112253, + "tokens_per_successful_query": 748.3533333333334, + "total_tokens_successful": 112253 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.4, + "p95_calls": 1.0, + "success_within_1": 0.9666666666666667, + "success_within_2": 0.9666666666666667, + "success_within_3": 0.9733333333333334, + "mean_tokens": 1083.8266666666666, + "p95_tokens": 1101.2999999999995, + "total_tokens": 162574, + "tokens_per_successful_query": 1083.8266666666666, + "total_tokens_successful": 162574 + } + } +} diff --git a/experiments/mcp-tool-search/results/cost.csv b/experiments/mcp-tool-search/results/cost.csv new file mode 100644 index 00000000..b8fc8cc3 --- /dev/null +++ b/experiments/mcp-tool-search/results/cost.csv @@ -0,0 +1,91 @@ +name,recall@1,k,mean_calls,p95_calls,success_within_1,success_within_2,success_within_3,mean_tokens,p95_tokens,total_tokens,tokens_per_successful_query +bm25_desc_only,0.8267,3,1.9000,2.0000,0.9267,0.9667,0.9733,466.9,441.1,70036,466.9 +bm25_desc_only,0.8267,5,1.5200,1.0000,0.9600,0.9733,0.9733,597.3,598.3,89594,597.3 +bm25_desc_only,0.8267,10,1.2400,1.0000,0.9733,0.9733,0.9733,932.5,1045.0,139870,932.5 +bm25_ident_split,0.9067,3,1.3400,1.0000,0.9800,0.9867,0.9933,315.9,332.5,47382,315.9 +bm25_ident_split,0.9067,5,1.2000,1.0000,0.9867,0.9933,0.9933,452.4,515.4,67853,452.4 +bm25_ident_split,0.9067,10,1.0933,1.0000,0.9933,0.9933,0.9933,795.4,1001.0,119308,795.4 +bm25_name_only,0.8200,3,3.7667,3.5500,0.9000,0.9400,0.9467,924.1,975.8,138610,924.1 +bm25_name_only,0.8200,5,2.6333,2.5500,0.9267,0.9467,0.9533,1036.0,1114.9,155394,1036.0 +bm25_name_only,0.8200,10,1.7933,1.5500,0.9467,0.9533,0.9533,1365.7,1301.8,204854,1365.7 +bm25_plain,0.9067,3,1.4000,1.0000,0.9733,0.9800,0.9867,331.6,324.0,49733,331.6 +bm25_plain,0.9067,5,1.2333,1.0000,0.9800,0.9867,0.9867,465.3,478.3,69795,465.3 +bm25_plain,0.9067,10,1.1067,1.0000,0.9867,0.9867,0.9933,803.8,989.3,120565,803.8 +bm25_tfidf_hybrid,0.8200,3,1.3467,1.0000,0.9733,0.9867,0.9933,318.6,336.9,47789,318.6 +bm25_tfidf_hybrid,0.8200,5,1.1933,1.0000,0.9867,0.9933,0.9933,450.0,486.0,67506,450.0 +bm25_tfidf_hybrid,0.8200,10,1.0933,1.0000,0.9933,0.9933,0.9933,812.5,962.0,121875,812.5 +bm25_weighted,0.8533,3,1.0400,1.0000,0.9733,0.9933,0.9933,242.9,349.7,36435,242.9 +bm25_weighted,0.8533,5,1.0200,1.0000,0.9867,0.9933,1.0000,382.8,521.2,57427,382.8 +bm25_weighted,0.8533,10,1.0067,1.0000,0.9933,1.0000,1.0000,733.5,952.8,110019,733.5 +boolean_and,0.7667,3,10.7333,59.7500,0.7733,0.7733,0.7733,2656.5,14505.5,398471,2656.5 +boolean_and,0.7667,5,6.7933,35.6500,0.7733,0.7733,0.7733,2707.6,13944.2,406143,2707.6 +boolean_and,0.7667,10,3.8400,18.1000,0.7733,0.7800,0.7800,3003.6,13725.3,450537,3003.6 +boolean_or,0.9467,3,1.2000,1.0000,0.9800,0.9933,0.9933,309.1,392.8,46369,309.1 +boolean_or,0.9467,5,1.1067,1.0000,0.9933,0.9933,0.9933,464.9,600.5,69734,464.9 +boolean_or,0.9467,10,1.0533,1.0000,0.9933,0.9933,0.9933,875.5,1188.2,131331,875.5 +char_trigram,0.8533,3,1.5400,1.0000,0.9667,0.9867,0.9867,368.2,329.7,55230,368.2 +char_trigram,0.8533,5,1.3200,1.0000,0.9800,0.9867,0.9867,503.1,500.9,75462,503.1 +char_trigram,0.8533,10,1.1533,1.0000,0.9867,0.9867,0.9867,850.8,994.1,127618,850.8 +codedb_find,0.4867,3,28.3133,101.2000,0.5067,0.5067,0.5133,6934.5,24073.0,1040178,6934.5 +codedb_find,0.4867,5,17.2600,61.1000,0.5067,0.5200,0.5200,6801.5,23343.4,1020221,6801.5 +codedb_find,0.4867,10,9.0133,30.5500,0.5200,0.5267,0.5333,6907.2,22689.3,1036085,6907.2 +codedb_search,0.9133,3,1.6333,1.0000,0.9600,0.9733,0.9733,399.6,353.1,59945,399.6 +codedb_search,0.9133,5,1.3733,1.0000,0.9667,0.9733,0.9733,529.3,573.0,79401,529.3 +codedb_search,0.9133,10,1.1733,1.0000,0.9733,0.9733,0.9867,844.0,1107.4,126604,844.0 +codedb_symbol,0.3067,3,15.0067,85.6500,0.5333,0.6200,0.7000,3683.1,20395.5,552471,3683.1 +codedb_symbol,0.3067,5,9.2933,52.1000,0.6133,0.7267,0.7333,3668.0,20001.0,550193,3668.0 +codedb_symbol,0.3067,10,5.0267,26.5500,0.7267,0.7333,0.7400,3851.0,19869.4,577646,3851.0 +codedb_trigram_tokens,0.6067,3,4.0600,30.1000,0.8067,0.8733,0.9067,1007.1,8227.0,151069,1007.1 +codedb_trigram_tokens,0.6067,5,2.7800,18.5500,0.8533,0.9267,0.9267,1088.8,8117.2,163319,1088.8 +codedb_trigram_tokens,0.6067,10,1.8400,9.5500,0.9267,0.9267,0.9267,1371.2,8033.3,205675,1371.2 +codedb_word,0.6733,3,2.5067,3.0000,0.8733,0.9333,0.9667,622.5,668.4,93371,622.5 +codedb_word,0.6733,5,1.8733,2.0000,0.9133,0.9667,0.9667,748.4,839.0,112253,748.4 +codedb_word,0.6733,10,1.4000,1.0000,0.9667,0.9667,0.9733,1083.8,1101.3,162574,1083.8 +decision_heuristic,0.8533,3,1.0400,1.0000,0.9733,0.9933,0.9933,242.9,349.7,36435,242.9 +decision_heuristic,0.8533,5,1.0200,1.0000,0.9867,0.9933,1.0000,382.8,521.2,57427,382.8 +decision_heuristic,0.8533,10,1.0067,1.0000,0.9933,1.0000,1.0000,733.5,952.8,110019,733.5 +embed_bge_small,0.9267,3,1.0800,1.0000,0.9933,0.9933,0.9933,253.5,315.6,38022,253.5 +embed_bge_small,0.9267,5,1.0467,1.0000,0.9933,0.9933,0.9933,393.7,507.3,59055,393.7 +embed_bge_small,0.9267,10,1.0200,1.0000,0.9933,0.9933,0.9933,751.1,979.1,112672,751.1 +hashed_ngram_embed,0.8400,3,1.0333,1.0000,0.9667,1.0000,1.0000,240.9,340.3,36135,240.9 +hashed_ngram_embed,0.8400,5,1.0067,1.0000,0.9933,1.0000,1.0000,380.9,509.0,57132,380.9 +hashed_ngram_embed,0.8400,10,1.0000,1.0000,1.0000,1.0000,1.0000,756.1,941.6,113413,756.1 +hybrid_codedb_embed,0.9333,3,1.0867,1.0000,0.9867,0.9933,0.9933,253.0,345.5,37948,253.0 +hybrid_codedb_embed,0.9333,5,1.0467,1.0000,0.9933,0.9933,0.9933,393.0,538.2,58957,393.0 +hybrid_codedb_embed,0.9333,10,1.0200,1.0000,0.9933,0.9933,0.9933,729.5,979.4,109423,729.5 +hybrid_codedb_trigram_bm25,0.7067,3,1.1600,2.0000,0.9000,0.9533,0.9867,268.3,577.3,40240,268.3 +hybrid_codedb_trigram_bm25,0.7067,5,1.0533,1.0000,0.9533,0.9933,1.0000,384.9,587.2,57741,384.9 +hybrid_codedb_trigram_bm25,0.7067,10,1.0067,1.0000,0.9933,1.0000,1.0000,715.5,946.7,107331,715.5 +hybrid_codedb_weighted_bm25,0.9333,3,1.0133,1.0000,0.9867,1.0000,1.0000,236.3,336.8,35445,236.3 +hybrid_codedb_weighted_bm25,0.9333,5,1.0067,1.0000,0.9933,1.0000,1.0000,372.8,522.6,55917,372.8 +hybrid_codedb_weighted_bm25,0.9333,10,1.0000,1.0000,1.0000,1.0000,1.0000,705.2,945.7,105777,705.2 +hybrid_rrf_bm25_embed,0.9333,3,1.0867,1.0000,0.9867,0.9933,0.9933,250.8,322.3,37621,250.8 +hybrid_rrf_bm25_embed,0.9333,5,1.0467,1.0000,0.9933,0.9933,0.9933,389.6,535.5,58441,389.6 +hybrid_rrf_bm25_embed,0.9333,10,1.0200,1.0000,0.9933,0.9933,0.9933,736.7,995.8,110507,736.7 +hybrid_rrf_bm25_trigram,0.9200,3,1.3400,1.0000,0.9800,0.9867,0.9933,314.4,332.5,47157,314.4 +hybrid_rrf_bm25_trigram,0.9200,5,1.2000,1.0000,0.9867,0.9933,0.9933,446.9,517.9,67039,446.9 +hybrid_rrf_bm25_trigram,0.9200,10,1.0933,1.0000,0.9933,0.9933,0.9933,784.1,985.0,117622,784.1 +hybrid_rrf_trigram_embed,0.9000,3,1.1000,1.0000,0.9733,0.9933,0.9933,253.9,328.6,38082,253.9 +hybrid_rrf_trigram_embed,0.9000,5,1.0467,1.0000,0.9933,0.9933,0.9933,388.1,489.0,58209,388.1 +hybrid_rrf_trigram_embed,0.9000,10,1.0200,1.0000,0.9933,0.9933,0.9933,745.5,970.1,111823,745.5 +hybrid_rrf_weighted_embed,0.8867,3,1.0800,1.0000,0.9933,0.9933,0.9933,255.4,317.9,38304,255.4 +hybrid_rrf_weighted_embed,0.8867,5,1.0467,1.0000,0.9933,0.9933,0.9933,397.3,492.3,59589,397.3 +hybrid_rrf_weighted_embed,0.8867,10,1.0200,1.0000,0.9933,0.9933,0.9933,758.6,1008.0,113793,758.6 +hybrid_rrf_weighted_hashed,0.8600,3,1.0133,1.0000,0.9867,1.0000,1.0000,239.2,324.8,35885,239.2 +hybrid_rrf_weighted_hashed,0.8600,5,1.0067,1.0000,0.9933,1.0000,1.0000,379.0,511.5,56851,379.0 +hybrid_rrf_weighted_hashed,0.8600,10,1.0000,1.0000,1.0000,1.0000,1.0000,754.9,1007.8,113236,754.9 +hybrid_rrf_weighted_trigram,0.8600,3,1.0200,1.0000,0.9867,0.9933,1.0000,236.7,333.0,35500,236.7 +hybrid_rrf_weighted_trigram,0.8600,5,1.0067,1.0000,0.9933,1.0000,1.0000,376.5,516.6,56478,376.5 +hybrid_rrf_weighted_trigram,0.8600,10,1.0000,1.0000,1.0000,1.0000,1.0000,740.6,1001.2,111088,740.6 +lexical_rerank_weighted_top20,0.8800,3,1.0267,1.0000,0.9733,1.0000,1.0000,241.0,372.9,36149,241.0 +lexical_rerank_weighted_top20,0.8800,5,1.0133,1.0000,0.9867,1.0000,1.0000,380.4,510.9,57062,380.4 +lexical_rerank_weighted_top20,0.8800,10,1.0000,1.0000,1.0000,1.0000,1.0000,731.0,959.9,109649,731.0 +name_exact_substr,0.0000,3,63.7467,116.5500,0.0067,0.0067,0.0133,15418.6,27301.7,2312788,15418.6 +name_exact_substr,0.0000,5,38.4800,70.0000,0.0067,0.0133,0.0200,14978.9,26505.0,2246838,14978.9 +name_exact_substr,0.0000,10,19.4867,35.0000,0.0133,0.0400,0.0733,14759.1,25789.2,2213858,14759.1 +query_rewrite_weighted_bm25,0.8333,3,1.1067,1.0000,0.9533,0.9867,0.9933,261.1,389.1,39171,261.1 +query_rewrite_weighted_bm25,0.8333,5,1.0533,1.0000,0.9733,0.9933,0.9933,398.8,565.8,59822,398.8 +query_rewrite_weighted_bm25,0.8333,10,1.0133,1.0000,0.9933,0.9933,1.0000,749.8,987.9,112464,749.8 +tfidf,0.8000,3,1.3467,1.0000,0.9733,0.9867,0.9933,318.5,314.5,47779,318.5 +tfidf,0.8000,5,1.1933,1.0000,0.9867,0.9933,0.9933,449.4,471.1,67404,449.4 +tfidf,0.8000,10,1.0933,1.0000,0.9933,0.9933,0.9933,813.0,954.3,121949,813.0 diff --git a/experiments/mcp-tool-search/results/cost.json b/experiments/mcp-tool-search/results/cost.json new file mode 100644 index 00000000..f30999af --- /dev/null +++ b/experiments/mcp-tool-search/results/cost.json @@ -0,0 +1,1851 @@ +{ + "encoding": "o200k_base", + "page_ks": [ + 3, + 5, + 10 + ], + "n_queries": 150, + "n_tools": 536, + "all_schemas_baseline": { + "n_tools": 536, + "tokens_per_turn": 38003, + "encoding": "o200k_base", + "payload_bytes": 170223 + }, + "methods": [ + { + "name": "bm25_desc_only", + "recall@1": 0.8266666666666667, + "by_k": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.9, + "p95_calls": 2.0, + "success_within_1": 0.9266666666666666, + "success_within_2": 0.9666666666666667, + "success_within_3": 0.9733333333333334, + "mean_tokens": 466.9066666666667, + "p95_tokens": 441.14999999999975, + "total_tokens": 70036, + "tokens_per_successful_query": 466.9066666666667, + "total_tokens_successful": 70036 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.52, + "p95_calls": 1.0, + "success_within_1": 0.96, + "success_within_2": 0.9733333333333334, + "success_within_3": 0.9733333333333334, + "mean_tokens": 597.2933333333333, + "p95_tokens": 598.2999999999995, + "total_tokens": 89594, + "tokens_per_successful_query": 597.2933333333333, + "total_tokens_successful": 89594 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.24, + "p95_calls": 1.0, + "success_within_1": 0.9733333333333334, + "success_within_2": 0.9733333333333334, + "success_within_3": 0.9733333333333334, + "mean_tokens": 932.4666666666667, + "p95_tokens": 1045.0499999999997, + "total_tokens": 139870, + "tokens_per_successful_query": 932.4666666666667, + "total_tokens_successful": 139870 + } + } + }, + { + "name": "bm25_ident_split", + "recall@1": 0.9066666666666666, + "by_k": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.34, + "p95_calls": 1.0, + "success_within_1": 0.98, + "success_within_2": 0.9866666666666667, + "success_within_3": 0.9933333333333333, + "mean_tokens": 315.88, + "p95_tokens": 332.54999999999995, + "total_tokens": 47382, + "tokens_per_successful_query": 315.88, + "total_tokens_successful": 47382 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.2, + "p95_calls": 1.0, + "success_within_1": 0.9866666666666667, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 452.35333333333335, + "p95_tokens": 515.3999999999988, + "total_tokens": 67853, + "tokens_per_successful_query": 452.35333333333335, + "total_tokens_successful": 67853 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.0933333333333333, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 795.3866666666667, + "p95_tokens": 1001.0, + "total_tokens": 119308, + "tokens_per_successful_query": 795.3866666666667, + "total_tokens_successful": 119308 + } + } + }, + { + "name": "bm25_name_only", + "recall@1": 0.82, + "by_k": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 3.7666666666666666, + "p95_calls": 3.549999999999983, + "success_within_1": 0.9, + "success_within_2": 0.94, + "success_within_3": 0.9466666666666667, + "mean_tokens": 924.0666666666667, + "p95_tokens": 975.8499999999945, + "total_tokens": 138610, + "tokens_per_successful_query": 924.0666666666667, + "total_tokens_successful": 138610 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 2.6333333333333333, + "p95_calls": 2.549999999999983, + "success_within_1": 0.9266666666666666, + "success_within_2": 0.9466666666666667, + "success_within_3": 0.9533333333333334, + "mean_tokens": 1035.96, + "p95_tokens": 1114.8999999999976, + "total_tokens": 155394, + "tokens_per_successful_query": 1035.96, + "total_tokens_successful": 155394 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.7933333333333332, + "p95_calls": 1.549999999999983, + "success_within_1": 0.9466666666666667, + "success_within_2": 0.9533333333333334, + "success_within_3": 0.9533333333333334, + "mean_tokens": 1365.6933333333334, + "p95_tokens": 1301.8499999999954, + "total_tokens": 204854, + "tokens_per_successful_query": 1365.6933333333334, + "total_tokens_successful": 204854 + } + } + }, + { + "name": "bm25_plain", + "recall@1": 0.9066666666666666, + "by_k": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.4, + "p95_calls": 1.0, + "success_within_1": 0.9733333333333334, + "success_within_2": 0.98, + "success_within_3": 0.9866666666666667, + "mean_tokens": 331.55333333333334, + "p95_tokens": 323.99999999999966, + "total_tokens": 49733, + "tokens_per_successful_query": 331.55333333333334, + "total_tokens_successful": 49733 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.2333333333333334, + "p95_calls": 1.0, + "success_within_1": 0.98, + "success_within_2": 0.9866666666666667, + "success_within_3": 0.9866666666666667, + "mean_tokens": 465.3, + "p95_tokens": 478.29999999999984, + "total_tokens": 69795, + "tokens_per_successful_query": 465.3, + "total_tokens_successful": 69795 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.1066666666666667, + "p95_calls": 1.0, + "success_within_1": 0.9866666666666667, + "success_within_2": 0.9866666666666667, + "success_within_3": 0.9933333333333333, + "mean_tokens": 803.7666666666667, + "p95_tokens": 989.2999999999995, + "total_tokens": 120565, + "tokens_per_successful_query": 803.7666666666667, + "total_tokens_successful": 120565 + } + } + }, + { + "name": "bm25_tfidf_hybrid", + "recall@1": 0.82, + "by_k": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.3466666666666667, + "p95_calls": 1.0, + "success_within_1": 0.9733333333333334, + "success_within_2": 0.9866666666666667, + "success_within_3": 0.9933333333333333, + "mean_tokens": 318.5933333333333, + "p95_tokens": 336.9499999999998, + "total_tokens": 47789, + "tokens_per_successful_query": 318.5933333333333, + "total_tokens_successful": 47789 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.1933333333333334, + "p95_calls": 1.0, + "success_within_1": 0.9866666666666667, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 450.04, + "p95_tokens": 485.99999999999966, + "total_tokens": 67506, + "tokens_per_successful_query": 450.04, + "total_tokens_successful": 67506 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.0933333333333333, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 812.5, + "p95_tokens": 962.0, + "total_tokens": 121875, + "tokens_per_successful_query": 812.5, + "total_tokens_successful": 121875 + } + } + }, + { + "name": "bm25_weighted", + "recall@1": 0.8533333333333334, + "by_k": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.04, + "p95_calls": 1.0, + "success_within_1": 0.9733333333333334, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 242.9, + "p95_tokens": 349.7499999999999, + "total_tokens": 36435, + "tokens_per_successful_query": 242.9, + "total_tokens_successful": 36435 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.02, + "p95_calls": 1.0, + "success_within_1": 0.9866666666666667, + "success_within_2": 0.9933333333333333, + "success_within_3": 1.0, + "mean_tokens": 382.8466666666667, + "p95_tokens": 521.1999999999999, + "total_tokens": 57427, + "tokens_per_successful_query": 382.8466666666667, + "total_tokens_successful": 57427 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.0066666666666666, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 733.46, + "p95_tokens": 952.75, + "total_tokens": 110019, + "tokens_per_successful_query": 733.46, + "total_tokens_successful": 110019 + } + } + }, + { + "name": "boolean_and", + "recall@1": 0.7666666666666667, + "by_k": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 10.733333333333333, + "p95_calls": 59.749999999999915, + "success_within_1": 0.7733333333333333, + "success_within_2": 0.7733333333333333, + "success_within_3": 0.7733333333333333, + "mean_tokens": 2656.4733333333334, + "p95_tokens": 14505.499999999985, + "total_tokens": 398471, + "tokens_per_successful_query": 2656.4733333333334, + "total_tokens_successful": 398471 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 6.793333333333333, + "p95_calls": 35.64999999999995, + "success_within_1": 0.7733333333333333, + "success_within_2": 0.7733333333333333, + "success_within_3": 0.7733333333333333, + "mean_tokens": 2707.62, + "p95_tokens": 13944.199999999986, + "total_tokens": 406143, + "tokens_per_successful_query": 2707.62, + "total_tokens_successful": 406143 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 3.84, + "p95_calls": 18.099999999999966, + "success_within_1": 0.7733333333333333, + "success_within_2": 0.78, + "success_within_3": 0.78, + "mean_tokens": 3003.58, + "p95_tokens": 13725.299999999981, + "total_tokens": 450537, + "tokens_per_successful_query": 3003.58, + "total_tokens_successful": 450537 + } + } + }, + { + "name": "boolean_or", + "recall@1": 0.9466666666666667, + "by_k": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.2, + "p95_calls": 1.0, + "success_within_1": 0.98, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 309.12666666666667, + "p95_tokens": 392.8499999999999, + "total_tokens": 46369, + "tokens_per_successful_query": 309.12666666666667, + "total_tokens_successful": 46369 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.1066666666666667, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 464.8933333333333, + "p95_tokens": 600.55, + "total_tokens": 69734, + "tokens_per_successful_query": 464.8933333333333, + "total_tokens_successful": 69734 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.0533333333333332, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 875.54, + "p95_tokens": 1188.2, + "total_tokens": 131331, + "tokens_per_successful_query": 875.54, + "total_tokens_successful": 131331 + } + } + }, + { + "name": "char_trigram", + "recall@1": 0.8533333333333334, + "by_k": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.54, + "p95_calls": 1.0, + "success_within_1": 0.9666666666666667, + "success_within_2": 0.9866666666666667, + "success_within_3": 0.9866666666666667, + "mean_tokens": 368.2, + "p95_tokens": 329.7499999999999, + "total_tokens": 55230, + "tokens_per_successful_query": 368.2, + "total_tokens_successful": 55230 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.32, + "p95_calls": 1.0, + "success_within_1": 0.98, + "success_within_2": 0.9866666666666667, + "success_within_3": 0.9866666666666667, + "mean_tokens": 503.08, + "p95_tokens": 500.9499999999998, + "total_tokens": 75462, + "tokens_per_successful_query": 503.08, + "total_tokens_successful": 75462 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.1533333333333333, + "p95_calls": 1.0, + "success_within_1": 0.9866666666666667, + "success_within_2": 0.9866666666666667, + "success_within_3": 0.9866666666666667, + "mean_tokens": 850.7866666666666, + "p95_tokens": 994.0999999999997, + "total_tokens": 127618, + "tokens_per_successful_query": 850.7866666666666, + "total_tokens_successful": 127618 + } + } + }, + { + "name": "codedb_find", + "recall@1": 0.4866666666666667, + "by_k": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 28.313333333333333, + "p95_calls": 101.19999999999993, + "success_within_1": 0.5066666666666667, + "success_within_2": 0.5066666666666667, + "success_within_3": 0.5133333333333333, + "mean_tokens": 6934.52, + "p95_tokens": 24072.999999999985, + "total_tokens": 1040178, + "tokens_per_successful_query": 6934.52, + "total_tokens_successful": 1040178 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 17.26, + "p95_calls": 61.099999999999966, + "success_within_1": 0.5066666666666667, + "success_within_2": 0.52, + "success_within_3": 0.52, + "mean_tokens": 6801.473333333333, + "p95_tokens": 23343.44999999999, + "total_tokens": 1020221, + "tokens_per_successful_query": 6801.473333333333, + "total_tokens_successful": 1020221 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 9.013333333333334, + "p95_calls": 30.549999999999983, + "success_within_1": 0.52, + "success_within_2": 0.5266666666666666, + "success_within_3": 0.5333333333333333, + "mean_tokens": 6907.233333333334, + "p95_tokens": 22689.29999999999, + "total_tokens": 1036085, + "tokens_per_successful_query": 6907.233333333334, + "total_tokens_successful": 1036085 + } + } + }, + { + "name": "codedb_search", + "recall@1": 0.9133333333333333, + "by_k": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.6333333333333333, + "p95_calls": 1.0, + "success_within_1": 0.96, + "success_within_2": 0.9733333333333334, + "success_within_3": 0.9733333333333334, + "mean_tokens": 399.6333333333333, + "p95_tokens": 353.1499999999994, + "total_tokens": 59945, + "tokens_per_successful_query": 399.6333333333333, + "total_tokens_successful": 59945 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.3733333333333333, + "p95_calls": 1.0, + "success_within_1": 0.9666666666666667, + "success_within_2": 0.9733333333333334, + "success_within_3": 0.9733333333333334, + "mean_tokens": 529.34, + "p95_tokens": 572.9999999999997, + "total_tokens": 79401, + "tokens_per_successful_query": 529.34, + "total_tokens_successful": 79401 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.1733333333333333, + "p95_calls": 1.0, + "success_within_1": 0.9733333333333334, + "success_within_2": 0.9733333333333334, + "success_within_3": 0.9866666666666667, + "mean_tokens": 844.0266666666666, + "p95_tokens": 1107.3999999999999, + "total_tokens": 126604, + "tokens_per_successful_query": 844.0266666666666, + "total_tokens_successful": 126604 + } + } + }, + { + "name": "codedb_symbol", + "recall@1": 0.30666666666666664, + "by_k": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 15.006666666666666, + "p95_calls": 85.64999999999995, + "success_within_1": 0.5333333333333333, + "success_within_2": 0.62, + "success_within_3": 0.7, + "mean_tokens": 3683.14, + "p95_tokens": 20395.54999999999, + "total_tokens": 552471, + "tokens_per_successful_query": 3683.14, + "total_tokens_successful": 552471 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 9.293333333333333, + "p95_calls": 52.099999999999966, + "success_within_1": 0.6133333333333333, + "success_within_2": 0.7266666666666667, + "success_within_3": 0.7333333333333333, + "mean_tokens": 3667.9533333333334, + "p95_tokens": 20001.049999999985, + "total_tokens": 550193, + "tokens_per_successful_query": 3667.9533333333334, + "total_tokens_successful": 550193 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 5.026666666666666, + "p95_calls": 26.549999999999983, + "success_within_1": 0.7266666666666667, + "success_within_2": 0.7333333333333333, + "success_within_3": 0.74, + "mean_tokens": 3850.9733333333334, + "p95_tokens": 19869.449999999983, + "total_tokens": 577646, + "tokens_per_successful_query": 3850.9733333333334, + "total_tokens_successful": 577646 + } + } + }, + { + "name": "codedb_trigram_tokens", + "recall@1": 0.6066666666666667, + "by_k": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 4.06, + "p95_calls": 30.099999999999966, + "success_within_1": 0.8066666666666666, + "success_within_2": 0.8733333333333333, + "success_within_3": 0.9066666666666666, + "mean_tokens": 1007.1266666666667, + "p95_tokens": 8226.999999999993, + "total_tokens": 151069, + "tokens_per_successful_query": 1007.1266666666667, + "total_tokens_successful": 151069 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 2.78, + "p95_calls": 18.549999999999983, + "success_within_1": 0.8533333333333334, + "success_within_2": 0.9266666666666666, + "success_within_3": 0.9266666666666666, + "mean_tokens": 1088.7933333333333, + "p95_tokens": 8117.249999999994, + "total_tokens": 163319, + "tokens_per_successful_query": 1088.7933333333333, + "total_tokens_successful": 163319 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.84, + "p95_calls": 9.549999999999983, + "success_within_1": 0.9266666666666666, + "success_within_2": 0.9266666666666666, + "success_within_3": 0.9266666666666666, + "mean_tokens": 1371.1666666666667, + "p95_tokens": 8033.29999999999, + "total_tokens": 205675, + "tokens_per_successful_query": 1371.1666666666667, + "total_tokens_successful": 205675 + } + } + }, + { + "name": "codedb_word", + "recall@1": 0.6733333333333333, + "by_k": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 2.506666666666667, + "p95_calls": 3.0, + "success_within_1": 0.8733333333333333, + "success_within_2": 0.9333333333333333, + "success_within_3": 0.9666666666666667, + "mean_tokens": 622.4733333333334, + "p95_tokens": 668.3999999999999, + "total_tokens": 93371, + "tokens_per_successful_query": 622.4733333333334, + "total_tokens_successful": 93371 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.8733333333333333, + "p95_calls": 2.0, + "success_within_1": 0.9133333333333333, + "success_within_2": 0.9666666666666667, + "success_within_3": 0.9666666666666667, + "mean_tokens": 748.3533333333334, + "p95_tokens": 839.0499999999998, + "total_tokens": 112253, + "tokens_per_successful_query": 748.3533333333334, + "total_tokens_successful": 112253 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.4, + "p95_calls": 1.0, + "success_within_1": 0.9666666666666667, + "success_within_2": 0.9666666666666667, + "success_within_3": 0.9733333333333334, + "mean_tokens": 1083.8266666666666, + "p95_tokens": 1101.2999999999995, + "total_tokens": 162574, + "tokens_per_successful_query": 1083.8266666666666, + "total_tokens_successful": 162574 + } + } + }, + { + "name": "decision_heuristic", + "recall@1": 0.8533333333333334, + "by_k": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.04, + "p95_calls": 1.0, + "success_within_1": 0.9733333333333334, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 242.9, + "p95_tokens": 349.7499999999999, + "total_tokens": 36435, + "tokens_per_successful_query": 242.9, + "total_tokens_successful": 36435 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.02, + "p95_calls": 1.0, + "success_within_1": 0.9866666666666667, + "success_within_2": 0.9933333333333333, + "success_within_3": 1.0, + "mean_tokens": 382.8466666666667, + "p95_tokens": 521.1999999999999, + "total_tokens": 57427, + "tokens_per_successful_query": 382.8466666666667, + "total_tokens_successful": 57427 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.0066666666666666, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 733.46, + "p95_tokens": 952.75, + "total_tokens": 110019, + "tokens_per_successful_query": 733.46, + "total_tokens_successful": 110019 + } + } + }, + { + "name": "embed_bge_small", + "recall@1": 0.9266666666666666, + "by_k": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.08, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 253.48, + "p95_tokens": 315.65, + "total_tokens": 38022, + "tokens_per_successful_query": 253.48, + "total_tokens_successful": 38022 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.0466666666666666, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 393.7, + "p95_tokens": 507.2999999999996, + "total_tokens": 59055, + "tokens_per_successful_query": 393.7, + "total_tokens_successful": 59055 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.02, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 751.1466666666666, + "p95_tokens": 979.1499999999992, + "total_tokens": 112672, + "tokens_per_successful_query": 751.1466666666666, + "total_tokens_successful": 112672 + } + } + }, + { + "name": "hashed_ngram_embed", + "recall@1": 0.84, + "by_k": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.0333333333333334, + "p95_calls": 1.0, + "success_within_1": 0.9666666666666667, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 240.9, + "p95_tokens": 340.2999999999999, + "total_tokens": 36135, + "tokens_per_successful_query": 240.9, + "total_tokens_successful": 36135 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.0066666666666666, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 380.88, + "p95_tokens": 509.0, + "total_tokens": 57132, + "tokens_per_successful_query": 380.88, + "total_tokens_successful": 57132 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.0, + "p95_calls": 1.0, + "success_within_1": 1.0, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 756.0866666666667, + "p95_tokens": 941.65, + "total_tokens": 113413, + "tokens_per_successful_query": 756.0866666666667, + "total_tokens_successful": 113413 + } + } + }, + { + "name": "hybrid_codedb_embed", + "recall@1": 0.9333333333333333, + "by_k": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.0866666666666667, + "p95_calls": 1.0, + "success_within_1": 0.9866666666666667, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 252.98666666666668, + "p95_tokens": 345.54999999999995, + "total_tokens": 37948, + "tokens_per_successful_query": 252.98666666666668, + "total_tokens_successful": 37948 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.0466666666666666, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 393.0466666666667, + "p95_tokens": 538.1999999999999, + "total_tokens": 58957, + "tokens_per_successful_query": 393.0466666666667, + "total_tokens_successful": 58957 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.02, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 729.4866666666667, + "p95_tokens": 979.3999999999999, + "total_tokens": 109423, + "tokens_per_successful_query": 729.4866666666667, + "total_tokens_successful": 109423 + } + } + }, + { + "name": "hybrid_codedb_trigram_bm25", + "recall@1": 0.7066666666666667, + "by_k": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.16, + "p95_calls": 2.0, + "success_within_1": 0.9, + "success_within_2": 0.9533333333333334, + "success_within_3": 0.9866666666666667, + "mean_tokens": 268.26666666666665, + "p95_tokens": 577.2999999999993, + "total_tokens": 40240, + "tokens_per_successful_query": 268.26666666666665, + "total_tokens_successful": 40240 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.0533333333333332, + "p95_calls": 1.0, + "success_within_1": 0.9533333333333334, + "success_within_2": 0.9933333333333333, + "success_within_3": 1.0, + "mean_tokens": 384.94, + "p95_tokens": 587.1999999999979, + "total_tokens": 57741, + "tokens_per_successful_query": 384.94, + "total_tokens_successful": 57741 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.0066666666666666, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 715.54, + "p95_tokens": 946.6999999999991, + "total_tokens": 107331, + "tokens_per_successful_query": 715.54, + "total_tokens_successful": 107331 + } + } + }, + { + "name": "hybrid_codedb_weighted_bm25", + "recall@1": 0.9333333333333333, + "by_k": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.0133333333333334, + "p95_calls": 1.0, + "success_within_1": 0.9866666666666667, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 236.3, + "p95_tokens": 336.8499999999999, + "total_tokens": 35445, + "tokens_per_successful_query": 236.3, + "total_tokens_successful": 35445 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.0066666666666666, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 372.78, + "p95_tokens": 522.5999999999991, + "total_tokens": 55917, + "tokens_per_successful_query": 372.78, + "total_tokens_successful": 55917 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.0, + "p95_calls": 1.0, + "success_within_1": 1.0, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 705.18, + "p95_tokens": 945.6999999999994, + "total_tokens": 105777, + "tokens_per_successful_query": 705.18, + "total_tokens_successful": 105777 + } + } + }, + { + "name": "hybrid_rrf_bm25_embed", + "recall@1": 0.9333333333333333, + "by_k": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.0866666666666667, + "p95_calls": 1.0, + "success_within_1": 0.9866666666666667, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 250.80666666666667, + "p95_tokens": 322.3499999999997, + "total_tokens": 37621, + "tokens_per_successful_query": 250.80666666666667, + "total_tokens_successful": 37621 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.0466666666666666, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 389.6066666666667, + "p95_tokens": 535.55, + "total_tokens": 58441, + "tokens_per_successful_query": 389.6066666666667, + "total_tokens_successful": 58441 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.02, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 736.7133333333334, + "p95_tokens": 995.7999999999994, + "total_tokens": 110507, + "tokens_per_successful_query": 736.7133333333334, + "total_tokens_successful": 110507 + } + } + }, + { + "name": "hybrid_rrf_bm25_trigram", + "recall@1": 0.92, + "by_k": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.34, + "p95_calls": 1.0, + "success_within_1": 0.98, + "success_within_2": 0.9866666666666667, + "success_within_3": 0.9933333333333333, + "mean_tokens": 314.38, + "p95_tokens": 332.54999999999995, + "total_tokens": 47157, + "tokens_per_successful_query": 314.38, + "total_tokens_successful": 47157 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.2, + "p95_calls": 1.0, + "success_within_1": 0.9866666666666667, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 446.9266666666667, + "p95_tokens": 517.9499999999998, + "total_tokens": 67039, + "tokens_per_successful_query": 446.9266666666667, + "total_tokens_successful": 67039 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.0933333333333333, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 784.1466666666666, + "p95_tokens": 984.9999999999993, + "total_tokens": 117622, + "tokens_per_successful_query": 784.1466666666666, + "total_tokens_successful": 117622 + } + } + }, + { + "name": "hybrid_rrf_trigram_embed", + "recall@1": 0.9, + "by_k": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.1, + "p95_calls": 1.0, + "success_within_1": 0.9733333333333334, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 253.88, + "p95_tokens": 328.65, + "total_tokens": 38082, + "tokens_per_successful_query": 253.88, + "total_tokens_successful": 38082 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.0466666666666666, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 388.06, + "p95_tokens": 489.04999999999984, + "total_tokens": 58209, + "tokens_per_successful_query": 388.06, + "total_tokens_successful": 58209 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.02, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 745.4866666666667, + "p95_tokens": 970.1499999999997, + "total_tokens": 111823, + "tokens_per_successful_query": 745.4866666666667, + "total_tokens_successful": 111823 + } + } + }, + { + "name": "hybrid_rrf_weighted_embed", + "recall@1": 0.8866666666666667, + "by_k": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.08, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 255.36, + "p95_tokens": 317.9499999999998, + "total_tokens": 38304, + "tokens_per_successful_query": 255.36, + "total_tokens_successful": 38304 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.0466666666666666, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 397.26, + "p95_tokens": 492.3499999999997, + "total_tokens": 59589, + "tokens_per_successful_query": 397.26, + "total_tokens_successful": 59589 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.02, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 758.62, + "p95_tokens": 1008.0499999999997, + "total_tokens": 113793, + "tokens_per_successful_query": 758.62, + "total_tokens_successful": 113793 + } + } + }, + { + "name": "hybrid_rrf_weighted_hashed", + "recall@1": 0.86, + "by_k": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.0133333333333334, + "p95_calls": 1.0, + "success_within_1": 0.9866666666666667, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 239.23333333333332, + "p95_tokens": 324.7999999999997, + "total_tokens": 35885, + "tokens_per_successful_query": 239.23333333333332, + "total_tokens_successful": 35885 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.0066666666666666, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 379.00666666666666, + "p95_tokens": 511.49999999999983, + "total_tokens": 56851, + "tokens_per_successful_query": 379.00666666666666, + "total_tokens_successful": 56851 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.0, + "p95_calls": 1.0, + "success_within_1": 1.0, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 754.9066666666666, + "p95_tokens": 1007.7999999999994, + "total_tokens": 113236, + "tokens_per_successful_query": 754.9066666666666, + "total_tokens_successful": 113236 + } + } + }, + { + "name": "hybrid_rrf_weighted_trigram", + "recall@1": 0.86, + "by_k": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.02, + "p95_calls": 1.0, + "success_within_1": 0.9866666666666667, + "success_within_2": 0.9933333333333333, + "success_within_3": 1.0, + "mean_tokens": 236.66666666666666, + "p95_tokens": 333.0, + "total_tokens": 35500, + "tokens_per_successful_query": 236.66666666666666, + "total_tokens_successful": 35500 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.0066666666666666, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 376.52, + "p95_tokens": 516.5999999999998, + "total_tokens": 56478, + "tokens_per_successful_query": 376.52, + "total_tokens_successful": 56478 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.0, + "p95_calls": 1.0, + "success_within_1": 1.0, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 740.5866666666667, + "p95_tokens": 1001.1999999999999, + "total_tokens": 111088, + "tokens_per_successful_query": 740.5866666666667, + "total_tokens_successful": 111088 + } + } + }, + { + "name": "lexical_rerank_weighted_top20", + "recall@1": 0.88, + "by_k": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.0266666666666666, + "p95_calls": 1.0, + "success_within_1": 0.9733333333333334, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 240.99333333333334, + "p95_tokens": 372.9499999999998, + "total_tokens": 36149, + "tokens_per_successful_query": 240.99333333333334, + "total_tokens_successful": 36149 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.0133333333333334, + "p95_calls": 1.0, + "success_within_1": 0.9866666666666667, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 380.41333333333336, + "p95_tokens": 510.89999999999975, + "total_tokens": 57062, + "tokens_per_successful_query": 380.41333333333336, + "total_tokens_successful": 57062 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.0, + "p95_calls": 1.0, + "success_within_1": 1.0, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 730.9933333333333, + "p95_tokens": 959.9499999999998, + "total_tokens": 109649, + "tokens_per_successful_query": 730.9933333333333, + "total_tokens_successful": 109649 + } + } + }, + { + "name": "name_exact_substr", + "recall@1": 0.0, + "by_k": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 63.74666666666667, + "p95_calls": 116.54999999999998, + "success_within_1": 0.006666666666666667, + "success_within_2": 0.006666666666666667, + "success_within_3": 0.013333333333333334, + "mean_tokens": 15418.586666666666, + "p95_tokens": 27301.65, + "total_tokens": 2312788, + "tokens_per_successful_query": 15418.586666666666, + "total_tokens_successful": 2312788 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 38.48, + "p95_calls": 70.0, + "success_within_1": 0.006666666666666667, + "success_within_2": 0.013333333333333334, + "success_within_3": 0.02, + "mean_tokens": 14978.92, + "p95_tokens": 26505.0, + "total_tokens": 2246838, + "tokens_per_successful_query": 14978.92, + "total_tokens_successful": 2246838 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 19.486666666666668, + "p95_calls": 35.0, + "success_within_1": 0.013333333333333334, + "success_within_2": 0.04, + "success_within_3": 0.07333333333333333, + "mean_tokens": 14759.053333333333, + "p95_tokens": 25789.25, + "total_tokens": 2213858, + "tokens_per_successful_query": 14759.053333333333, + "total_tokens_successful": 2213858 + } + } + }, + { + "name": "query_rewrite_weighted_bm25", + "recall@1": 0.8333333333333334, + "by_k": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.1066666666666667, + "p95_calls": 1.0, + "success_within_1": 0.9533333333333334, + "success_within_2": 0.9866666666666667, + "success_within_3": 0.9933333333333333, + "mean_tokens": 261.14, + "p95_tokens": 389.09999999999997, + "total_tokens": 39171, + "tokens_per_successful_query": 261.14, + "total_tokens_successful": 39171 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.0533333333333332, + "p95_calls": 1.0, + "success_within_1": 0.9733333333333334, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 398.81333333333333, + "p95_tokens": 565.7999999999997, + "total_tokens": 59822, + "tokens_per_successful_query": 398.81333333333333, + "total_tokens_successful": 59822 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.0133333333333334, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 1.0, + "mean_tokens": 749.76, + "p95_tokens": 987.9499999999989, + "total_tokens": 112464, + "tokens_per_successful_query": 749.76, + "total_tokens_successful": 112464 + } + } + }, + { + "name": "tfidf", + "recall@1": 0.8, + "by_k": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.3466666666666667, + "p95_calls": 1.0, + "success_within_1": 0.9733333333333334, + "success_within_2": 0.9866666666666667, + "success_within_3": 0.9933333333333333, + "mean_tokens": 318.52666666666664, + "p95_tokens": 314.54999999999995, + "total_tokens": 47779, + "tokens_per_successful_query": 318.52666666666664, + "total_tokens_successful": 47779 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.1933333333333334, + "p95_calls": 1.0, + "success_within_1": 0.9866666666666667, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 449.36, + "p95_tokens": 471.09999999999997, + "total_tokens": 67404, + "tokens_per_successful_query": 449.36, + "total_tokens_successful": 67404 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.0933333333333333, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 812.9933333333333, + "p95_tokens": 954.2999999999998, + "total_tokens": 121949, + "tokens_per_successful_query": 812.9933333333333, + "total_tokens_successful": 121949 + } + } + } + ], + "ranking_by_tokens_per_success_k5": [ + { + "cost_rank": 1, + "recall_rank": 3, + "name": "hybrid_codedb_weighted_bm25", + "recall@1": 0.9333333333333333, + "k5_tokens_per_success": 372.78, + "k5_mean_calls": 1.0066666666666666, + "k5_success_within_1": 0.9933333333333333, + "rank_diff": 2 + }, + { + "cost_rank": 2, + "recall_rank": 14, + "name": "hybrid_rrf_weighted_trigram", + "recall@1": 0.86, + "k5_tokens_per_success": 376.52, + "k5_mean_calls": 1.0066666666666666, + "k5_success_within_1": 0.9933333333333333, + "rank_diff": 12 + }, + { + "cost_rank": 3, + "recall_rank": 13, + "name": "hybrid_rrf_weighted_hashed", + "recall@1": 0.86, + "k5_tokens_per_success": 379.00666666666666, + "k5_mean_calls": 1.0066666666666666, + "k5_success_within_1": 0.9933333333333333, + "rank_diff": 10 + }, + { + "cost_rank": 4, + "recall_rank": 12, + "name": "lexical_rerank_weighted_top20", + "recall@1": 0.88, + "k5_tokens_per_success": 380.41333333333336, + "k5_mean_calls": 1.0133333333333334, + "k5_success_within_1": 0.9866666666666667, + "rank_diff": 8 + }, + { + "cost_rank": 5, + "recall_rank": 18, + "name": "hashed_ngram_embed", + "recall@1": 0.84, + "k5_tokens_per_success": 380.88, + "k5_mean_calls": 1.0066666666666666, + "k5_success_within_1": 0.9933333333333333, + "rank_diff": 13 + }, + { + "cost_rank": 6, + "recall_rank": 15, + "name": "bm25_weighted", + "recall@1": 0.8533333333333334, + "k5_tokens_per_success": 382.8466666666667, + "k5_mean_calls": 1.02, + "k5_success_within_1": 0.9866666666666667, + "rank_diff": 9 + }, + { + "cost_rank": 7, + "recall_rank": 17, + "name": "decision_heuristic", + "recall@1": 0.8533333333333334, + "k5_tokens_per_success": 382.8466666666667, + "k5_mean_calls": 1.02, + "k5_success_within_1": 0.9866666666666667, + "rank_diff": 10 + }, + { + "cost_rank": 8, + "recall_rank": 25, + "name": "hybrid_codedb_trigram_bm25", + "recall@1": 0.7066666666666667, + "k5_tokens_per_success": 384.94, + "k5_mean_calls": 1.0533333333333332, + "k5_success_within_1": 0.9533333333333334, + "rank_diff": 17 + }, + { + "cost_rank": 9, + "recall_rank": 10, + "name": "hybrid_rrf_trigram_embed", + "recall@1": 0.9, + "k5_tokens_per_success": 388.06, + "k5_mean_calls": 1.0466666666666666, + "k5_success_within_1": 0.9933333333333333, + "rank_diff": 1 + }, + { + "cost_rank": 10, + "recall_rank": 4, + "name": "hybrid_rrf_bm25_embed", + "recall@1": 0.9333333333333333, + "k5_tokens_per_success": 389.6066666666667, + "k5_mean_calls": 1.0466666666666666, + "k5_success_within_1": 0.9933333333333333, + "rank_diff": -6 + }, + { + "cost_rank": 11, + "recall_rank": 2, + "name": "hybrid_codedb_embed", + "recall@1": 0.9333333333333333, + "k5_tokens_per_success": 393.0466666666667, + "k5_mean_calls": 1.0466666666666666, + "k5_success_within_1": 0.9933333333333333, + "rank_diff": -9 + }, + { + "cost_rank": 12, + "recall_rank": 5, + "name": "embed_bge_small", + "recall@1": 0.9266666666666666, + "k5_tokens_per_success": 393.7, + "k5_mean_calls": 1.0466666666666666, + "k5_success_within_1": 0.9933333333333333, + "rank_diff": -7 + }, + { + "cost_rank": 13, + "recall_rank": 11, + "name": "hybrid_rrf_weighted_embed", + "recall@1": 0.8866666666666667, + "k5_tokens_per_success": 397.26, + "k5_mean_calls": 1.0466666666666666, + "k5_success_within_1": 0.9933333333333333, + "rank_diff": -2 + }, + { + "cost_rank": 14, + "recall_rank": 19, + "name": "query_rewrite_weighted_bm25", + "recall@1": 0.8333333333333334, + "k5_tokens_per_success": 398.81333333333333, + "k5_mean_calls": 1.0533333333333332, + "k5_success_within_1": 0.9733333333333334, + "rank_diff": 5 + }, + { + "cost_rank": 15, + "recall_rank": 6, + "name": "hybrid_rrf_bm25_trigram", + "recall@1": 0.92, + "k5_tokens_per_success": 446.9266666666667, + "k5_mean_calls": 1.2, + "k5_success_within_1": 0.9866666666666667, + "rank_diff": -9 + }, + { + "cost_rank": 16, + "recall_rank": 23, + "name": "tfidf", + "recall@1": 0.8, + "k5_tokens_per_success": 449.36, + "k5_mean_calls": 1.1933333333333334, + "k5_success_within_1": 0.9866666666666667, + "rank_diff": 7 + }, + { + "cost_rank": 17, + "recall_rank": 22, + "name": "bm25_tfidf_hybrid", + "recall@1": 0.82, + "k5_tokens_per_success": 450.04, + "k5_mean_calls": 1.1933333333333334, + "k5_success_within_1": 0.9866666666666667, + "rank_diff": 5 + }, + { + "cost_rank": 18, + "recall_rank": 8, + "name": "bm25_ident_split", + "recall@1": 0.9066666666666666, + "k5_tokens_per_success": 452.35333333333335, + "k5_mean_calls": 1.2, + "k5_success_within_1": 0.9866666666666667, + "rank_diff": -10 + }, + { + "cost_rank": 19, + "recall_rank": 1, + "name": "boolean_or", + "recall@1": 0.9466666666666667, + "k5_tokens_per_success": 464.8933333333333, + "k5_mean_calls": 1.1066666666666667, + "k5_success_within_1": 0.9933333333333333, + "rank_diff": -18 + }, + { + "cost_rank": 20, + "recall_rank": 9, + "name": "bm25_plain", + "recall@1": 0.9066666666666666, + "k5_tokens_per_success": 465.3, + "k5_mean_calls": 1.2333333333333334, + "k5_success_within_1": 0.98, + "rank_diff": -11 + }, + { + "cost_rank": 21, + "recall_rank": 16, + "name": "char_trigram", + "recall@1": 0.8533333333333334, + "k5_tokens_per_success": 503.08, + "k5_mean_calls": 1.32, + "k5_success_within_1": 0.98, + "rank_diff": -5 + }, + { + "cost_rank": 22, + "recall_rank": 7, + "name": "codedb_search", + "recall@1": 0.9133333333333333, + "k5_tokens_per_success": 529.34, + "k5_mean_calls": 1.3733333333333333, + "k5_success_within_1": 0.9666666666666667, + "rank_diff": -15 + }, + { + "cost_rank": 23, + "recall_rank": 20, + "name": "bm25_desc_only", + "recall@1": 0.8266666666666667, + "k5_tokens_per_success": 597.2933333333333, + "k5_mean_calls": 1.52, + "k5_success_within_1": 0.96, + "rank_diff": -3 + }, + { + "cost_rank": 24, + "recall_rank": 26, + "name": "codedb_word", + "recall@1": 0.6733333333333333, + "k5_tokens_per_success": 748.3533333333334, + "k5_mean_calls": 1.8733333333333333, + "k5_success_within_1": 0.9133333333333333, + "rank_diff": 2 + }, + { + "cost_rank": 25, + "recall_rank": 21, + "name": "bm25_name_only", + "recall@1": 0.82, + "k5_tokens_per_success": 1035.96, + "k5_mean_calls": 2.6333333333333333, + "k5_success_within_1": 0.9266666666666666, + "rank_diff": -4 + }, + { + "cost_rank": 26, + "recall_rank": 27, + "name": "codedb_trigram_tokens", + "recall@1": 0.6066666666666667, + "k5_tokens_per_success": 1088.7933333333333, + "k5_mean_calls": 2.78, + "k5_success_within_1": 0.8533333333333334, + "rank_diff": 1 + }, + { + "cost_rank": 27, + "recall_rank": 24, + "name": "boolean_and", + "recall@1": 0.7666666666666667, + "k5_tokens_per_success": 2707.62, + "k5_mean_calls": 6.793333333333333, + "k5_success_within_1": 0.7733333333333333, + "rank_diff": -3 + }, + { + "cost_rank": 28, + "recall_rank": 29, + "name": "codedb_symbol", + "recall@1": 0.30666666666666664, + "k5_tokens_per_success": 3667.9533333333334, + "k5_mean_calls": 9.293333333333333, + "k5_success_within_1": 0.6133333333333333, + "rank_diff": 1 + }, + { + "cost_rank": 29, + "recall_rank": 28, + "name": "codedb_find", + "recall@1": 0.4866666666666667, + "k5_tokens_per_success": 6801.473333333333, + "k5_mean_calls": 17.26, + "k5_success_within_1": 0.5066666666666667, + "rank_diff": -1 + }, + { + "cost_rank": 30, + "recall_rank": 30, + "name": "name_exact_substr", + "recall@1": 0.0, + "k5_tokens_per_success": 14978.92, + "k5_mean_calls": 38.48, + "k5_success_within_1": 0.006666666666666667, + "rank_diff": 0 + } + ], + "model": "Deterministic offset paging of the saved ranking. Call n returns ranks[(n-1)k : n k] as full MCP tool schemas. Stops when any gold/alt appears. Tools not in the saved top-10 are appended in catalog order (pessimistic tail for misses)." +} diff --git a/experiments/mcp-tool-search/results/cost_per_query.csv b/experiments/mcp-tool-search/results/cost_per_query.csv new file mode 100644 index 00000000..38f13b7e --- /dev/null +++ b/experiments/mcp-tool-search/results/cost_per_query.csv @@ -0,0 +1,4501 @@ +method,id,query,hit_rank,k3_calls,k3_tokens,k3_success1,k3_success2,k3_success3,k5_calls,k5_tokens,k5_success1,k5_success2,k5_success3,k10_calls,k10_tokens,k10_success1,k10_success2,k10_success3 +bm25_desc_only,q001,Create a GitHub issue,2,1,380,1.0,1.0,1.0,1,584,1.0,1.0,1.0,1,1011,1.0,1.0,1.0 +bm25_desc_only,q002,Create a Jira ticket,1,1,281,1.0,1.0,1.0,1,481,1.0,1.0,1.0,1,966,1.0,1.0,1.0 +bm25_desc_only,q003,Send a Slack message,2,1,247,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,771,1.0,1.0,1.0 +bm25_desc_only,q004,List open pull requests,2,1,237,1.0,1.0,1.0,1,423,1.0,1.0,1.0,1,825,1.0,1.0,1.0 +bm25_desc_only,q005,Open a pull request on GitHub,1,1,379,1.0,1.0,1.0,1,612,1.0,1.0,1.0,1,1163,1.0,1.0,1.0 +bm25_desc_only,q006,Run a SQL query against postgres,3,1,180,1.0,1.0,1.0,1,295,1.0,1.0,1.0,1,671,1.0,1.0,1.0 +bm25_desc_only,q007,Read a local file from disk,1,1,223,1.0,1.0,1.0,1,425,1.0,1.0,1.0,1,832,1.0,1.0,1.0 +bm25_desc_only,q008,What files are in this folder,6,2,413,0.0,1.0,1.0,2,727,0.0,1.0,1.0,1,705,1.0,1.0,1.0 +bm25_desc_only,q009,Remember this fact in the knowledge graph,2,1,154,1.0,1.0,1.0,1,261,1.0,1.0,1.0,1,537,1.0,1.0,1.0 +bm25_desc_only,q010,Search the web with Brave,1,1,258,1.0,1.0,1.0,1,430,1.0,1.0,1.0,1,723,1.0,1.0,1.0 +bm25_desc_only,q011,Fetch the contents of a URL,1,1,208,1.0,1.0,1.0,1,346,1.0,1.0,1.0,1,662,1.0,1.0,1.0 +bm25_desc_only,q012,Show git status,1,1,186,1.0,1.0,1.0,1,299,1.0,1.0,1.0,1,675,1.0,1.0,1.0 +bm25_desc_only,q013,Commit staged changes,2,1,193,1.0,1.0,1.0,1,305,1.0,1.0,1.0,1,663,1.0,1.0,1.0 +bm25_desc_only,q014,Create a Linear issue,1,1,357,1.0,1.0,1.0,1,498,1.0,1.0,1.0,1,956,1.0,1.0,1.0 +bm25_desc_only,q015,Make a task in my project tracker Linear,146,49,12370,0.0,0.0,0.0,30,12110,0.0,0.0,0.0,15,11750,0.0,0.0,0.0 +bm25_desc_only,q016,Post a comment on a GitHub issue,3,1,307,1.0,1.0,1.0,1,482,1.0,1.0,1.0,1,887,1.0,1.0,1.0 +bm25_desc_only,q017,Merge the pull request,1,1,321,1.0,1.0,1.0,1,524,1.0,1.0,1.0,1,1103,1.0,1.0,1.0 +bm25_desc_only,q018,Trigger a GitHub Actions workflow,1,1,294,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,831,1.0,1.0,1.0 +bm25_desc_only,q019,Get Dependabot alerts,1,1,287,1.0,1.0,1.0,1,420,1.0,1.0,1.0,1,739,1.0,1.0,1.0 +bm25_desc_only,q020,List Sentry issues,1,1,238,1.0,1.0,1.0,1,395,1.0,1.0,1.0,1,796,1.0,1.0,1.0 +bm25_desc_only,q021,Create a Stripe customer,1,1,186,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +bm25_desc_only,q022,Refund a Stripe payment,1,1,234,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,662,1.0,1.0,1.0 +bm25_desc_only,q023,Send an email with Gmail,1,1,255,1.0,1.0,1.0,1,395,1.0,1.0,1.0,1,707,1.0,1.0,1.0 +bm25_desc_only,q024,Search my Gmail inbox,1,1,219,1.0,1.0,1.0,1,305,1.0,1.0,1.0,1,666,1.0,1.0,1.0 +bm25_desc_only,q025,Upload a file to Google Drive,4,2,555,0.0,1.0,1.0,1,446,1.0,1.0,1.0,1,747,1.0,1.0,1.0 +bm25_desc_only,q026,Create a calendar event,1,1,300,1.0,1.0,1.0,1,461,1.0,1.0,1.0,1,813,1.0,1.0,1.0 +bm25_desc_only,q027,List Kubernetes pods,1,1,184,1.0,1.0,1.0,1,314,1.0,1.0,1.0,1,741,1.0,1.0,1.0 +bm25_desc_only,q028,Get logs from a k8s pod,1,1,262,1.0,1.0,1.0,1,423,1.0,1.0,1.0,1,822,1.0,1.0,1.0 +bm25_desc_only,q029,Run a docker container,1,1,223,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,594,1.0,1.0,1.0 +bm25_desc_only,q030,Show docker container logs,1,1,237,1.0,1.0,1.0,1,343,1.0,1.0,1.0,1,665,1.0,1.0,1.0 +bm25_desc_only,q031,Navigate the browser to a URL,1,1,179,1.0,1.0,1.0,1,293,1.0,1.0,1.0,1,568,1.0,1.0,1.0 +bm25_desc_only,q032,Take a screenshot of the page,1,1,212,1.0,1.0,1.0,1,326,1.0,1.0,1.0,1,665,1.0,1.0,1.0 +bm25_desc_only,q033,Create a GitLab merge request,2,1,297,1.0,1.0,1.0,1,475,1.0,1.0,1.0,1,833,1.0,1.0,1.0 +bm25_desc_only,q034,Search Notion pages,1,1,247,1.0,1.0,1.0,1,417,1.0,1.0,1.0,1,721,1.0,1.0,1.0 +bm25_desc_only,q035,Create a Notion page,2,1,230,1.0,1.0,1.0,1,376,1.0,1.0,1.0,1,822,1.0,1.0,1.0 +bm25_desc_only,q036,Query a Notion database,1,1,279,1.0,1.0,1.0,1,430,1.0,1.0,1.0,1,752,1.0,1.0,1.0 +bm25_desc_only,q037,Create a Zendesk ticket,1,1,257,1.0,1.0,1.0,1,426,1.0,1.0,1.0,1,791,1.0,1.0,1.0 +bm25_desc_only,q038,Create a HubSpot contact,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,838,1.0,1.0,1.0 +bm25_desc_only,q039,Create a Salesforce lead,1,1,252,1.0,1.0,1.0,1,416,1.0,1.0,1.0,1,758,1.0,1.0,1.0 +bm25_desc_only,q040,Run a SOQL query,1,1,186,1.0,1.0,1.0,1,324,1.0,1.0,1.0,1,673,1.0,1.0,1.0 +bm25_desc_only,q041,List Shopify orders,1,1,208,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,675,1.0,1.0,1.0 +bm25_desc_only,q042,Create a PagerDuty incident,1,1,259,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,773,1.0,1.0,1.0 +bm25_desc_only,q043,Who is on call in PagerDuty,1,1,227,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,721,1.0,1.0,1.0 +bm25_desc_only,q044,Query Datadog metrics,1,1,244,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,809,1.0,1.0,1.0 +bm25_desc_only,q045,Search Datadog logs,1,1,245,1.0,1.0,1.0,1,419,1.0,1.0,1.0,1,863,1.0,1.0,1.0 +bm25_desc_only,q046,List S3 buckets,1,1,183,1.0,1.0,1.0,1,263,1.0,1.0,1.0,1,490,1.0,1.0,1.0 +bm25_desc_only,q047,Upload an object to S3,1,1,262,1.0,1.0,1.0,1,423,1.0,1.0,1.0,1,788,1.0,1.0,1.0 +bm25_desc_only,q048,Purge Cloudflare cache,1,1,231,1.0,1.0,1.0,1,340,1.0,1.0,1.0,1,746,1.0,1.0,1.0 +bm25_desc_only,q049,Create a Cloudflare DNS record,1,1,249,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,732,1.0,1.0,1.0 +bm25_desc_only,q050,List Vercel deployments,1,1,173,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,527,1.0,1.0,1.0 +bm25_desc_only,q051,Insert a row into Supabase,1,1,240,1.0,1.0,1.0,1,357,1.0,1.0,1.0,1,665,1.0,1.0,1.0 +bm25_desc_only,q052,Export images from Figma,1,1,218,1.0,1.0,1.0,1,328,1.0,1.0,1.0,1,658,1.0,1.0,1.0 +bm25_desc_only,q053,Send an SMS with Twilio,1,1,254,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,738,1.0,1.0,1.0 +bm25_desc_only,q054,Create an Okta user,1,1,183,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,743,1.0,1.0,1.0 +bm25_desc_only,q055,Instant query Prometheus,1,1,221,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,680,1.0,1.0,1.0 +bm25_desc_only,q056,Search Grafana dashboards,1,1,217,1.0,1.0,1.0,1,375,1.0,1.0,1.0,1,735,1.0,1.0,1.0 +bm25_desc_only,q057,Trigger a CircleCI pipeline,1,1,200,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,718,1.0,1.0,1.0 +bm25_desc_only,q058,Run terraform plan,1,1,179,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,658,1.0,1.0,1.0 +bm25_desc_only,q059,Create an Asana task,1,1,267,1.0,1.0,1.0,1,429,1.0,1.0,1.0,1,825,1.0,1.0,1.0 +bm25_desc_only,q060,Create a Trello card,1,1,240,1.0,1.0,1.0,1,388,1.0,1.0,1.0,1,740,1.0,1.0,1.0 +bm25_desc_only,q061,Create a ClickUp task,1,1,257,1.0,1.0,1.0,1,415,1.0,1.0,1.0,1,745,1.0,1.0,1.0 +bm25_desc_only,q062,Create a Confluence page,1,1,260,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,692,1.0,1.0,1.0 +bm25_desc_only,q063,Send a Discord message,1,1,224,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,682,1.0,1.0,1.0 +bm25_desc_only,q064,Search Elasticsearch,1,1,239,1.0,1.0,1.0,1,397,1.0,1.0,1.0,1,766,1.0,1.0,1.0 +bm25_desc_only,q065,Get a Redis key,1,1,190,1.0,1.0,1.0,1,285,1.0,1.0,1.0,1,610,1.0,1.0,1.0 +bm25_desc_only,q066,Find MongoDB documents,1,1,258,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,738,1.0,1.0,1.0 +bm25_desc_only,q067,List SQLite tables,1,1,156,1.0,1.0,1.0,1,243,1.0,1.0,1.0,1,494,1.0,1.0,1.0 +bm25_desc_only,q068,Describe a postgres table,2,1,200,1.0,1.0,1.0,1,349,1.0,1.0,1.0,1,706,1.0,1.0,1.0 +bm25_desc_only,q069,github_create_issue,39,13,3683,0.0,0.0,0.0,8,3724,0.0,0.0,0.0,4,3648,0.0,0.0,0.0 +bm25_desc_only,q070,createIssue,3,1,332,1.0,1.0,1.0,1,579,1.0,1.0,1.0,1,1039,1.0,1.0,1.0 +bm25_desc_only,q071,list-directory,90,30,7901,0.0,0.0,0.0,18,7685,0.0,0.0,0.0,9,7523,0.0,0.0,0.0 +bm25_desc_only,q072,sequential thinking next thought,116,39,9799,0.0,0.0,0.0,24,9727,0.0,0.0,0.0,12,9475,0.0,0.0,0.0 +bm25_desc_only,q073,What time is it in Tokyo,1,1,190,1.0,1.0,1.0,1,342,1.0,1.0,1.0,1,709,1.0,1.0,1.0 +bm25_desc_only,q074,Convert time between timezones,1,1,204,1.0,1.0,1.0,1,335,1.0,1.0,1.0,1,636,1.0,1.0,1.0 +bm25_desc_only,q075,Star a GitHub repo,1,1,237,1.0,1.0,1.0,1,394,1.0,1.0,1.0,1,772,1.0,1.0,1.0 +bm25_desc_only,q076,Fork a repository,1,1,235,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,822,1.0,1.0,1.0 +bm25_desc_only,q077,Get the latest GitHub release,1,1,291,1.0,1.0,1.0,1,383,1.0,1.0,1.0,1,759,1.0,1.0,1.0 +bm25_desc_only,q078,Create a gist,1,1,229,1.0,1.0,1.0,1,383,1.0,1.0,1.0,1,733,1.0,1.0,1.0 +bm25_desc_only,q079,Search GitHub code,1,1,283,1.0,1.0,1.0,1,419,1.0,1.0,1.0,1,725,1.0,1.0,1.0 +bm25_desc_only,q080,List GitHub notifications,1,1,189,1.0,1.0,1.0,1,326,1.0,1.0,1.0,1,717,1.0,1.0,1.0 +bm25_desc_only,q081,Assign Copilot to an issue,1,1,267,1.0,1.0,1.0,1,421,1.0,1.0,1.0,1,819,1.0,1.0,1.0 +bm25_desc_only,q082,Get secret scanning alerts,1,1,295,1.0,1.0,1.0,1,470,1.0,1.0,1.0,1,804,1.0,1.0,1.0 +bm25_desc_only,q083,Create or update a file on GitHub,1,1,369,1.0,1.0,1.0,1,610,1.0,1.0,1.0,1,1088,1.0,1.0,1.0 +bm25_desc_only,q084,List collaborators on a repo,1,1,237,1.0,1.0,1.0,1,411,1.0,1.0,1.0,1,751,1.0,1.0,1.0 +bm25_desc_only,q085,Reply in a Slack thread,1,1,240,1.0,1.0,1.0,1,397,1.0,1.0,1.0,1,767,1.0,1.0,1.0 +bm25_desc_only,q086,Add an emoji reaction on Slack,1,1,294,1.0,1.0,1.0,1,461,1.0,1.0,1.0,1,827,1.0,1.0,1.0 +bm25_desc_only,q087,Search Slack messages,1,1,232,1.0,1.0,1.0,1,373,1.0,1.0,1.0,1,721,1.0,1.0,1.0 +bm25_desc_only,q088,Create a Slack channel,1,1,196,1.0,1.0,1.0,1,339,1.0,1.0,1.0,1,773,1.0,1.0,1.0 +bm25_desc_only,q089,Transition a Jira issue,1,1,222,1.0,1.0,1.0,1,342,1.0,1.0,1.0,1,822,1.0,1.0,1.0 +bm25_desc_only,q090,Search Jira with JQL,1,1,259,1.0,1.0,1.0,1,370,1.0,1.0,1.0,1,734,1.0,1.0,1.0 +bm25_desc_only,q091,Add a worklog to Jira,1,1,247,1.0,1.0,1.0,1,440,1.0,1.0,1.0,1,874,1.0,1.0,1.0 +bm25_desc_only,q092,List Linear teams,1,1,172,1.0,1.0,1.0,1,318,1.0,1.0,1.0,1,705,1.0,1.0,1.0 +bm25_desc_only,q093,Comment on a Linear issue,1,1,264,1.0,1.0,1.0,1,451,1.0,1.0,1.0,1,781,1.0,1.0,1.0 +bm25_desc_only,q094,Move a local file,5,2,434,0.0,1.0,1.0,1,338,1.0,1.0,1.0,1,712,1.0,1.0,1.0 +bm25_desc_only,q095,Recursively search files by pattern,1,1,200,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,741,1.0,1.0,1.0 +bm25_desc_only,q096,Edit a file in place with old and new string,1,1,247,1.0,1.0,1.0,1,419,1.0,1.0,1.0,1,763,1.0,1.0,1.0 +bm25_desc_only,q097,Read the whole knowledge graph,1,1,152,1.0,1.0,1.0,1,259,1.0,1.0,1.0,1,561,1.0,1.0,1.0 +bm25_desc_only,q098,Search memory nodes,1,1,241,1.0,1.0,1.0,1,409,1.0,1.0,1.0,1,729,1.0,1.0,1.0 +bm25_desc_only,q099,Show unstaged git diff,1,1,200,1.0,1.0,1.0,1,313,1.0,1.0,1.0,1,628,1.0,1.0,1.0 +bm25_desc_only,q100,Stage files with git add,5,2,495,0.0,1.0,1.0,1,402,1.0,1.0,1.0,1,710,1.0,1.0,1.0 +bm25_desc_only,q101,Create a Shopify product,1,1,240,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,694,1.0,1.0,1.0 +bm25_desc_only,q102,Cancel a Stripe subscription,1,1,188,1.0,1.0,1.0,1,301,1.0,1.0,1.0,1,632,1.0,1.0,1.0 +bm25_desc_only,q103,Create a payment intent,1,1,237,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,738,1.0,1.0,1.0 +bm25_desc_only,q104,Acknowledge a PagerDuty incident,1,1,261,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,717,1.0,1.0,1.0 +bm25_desc_only,q105,Create a Datadog monitor,1,1,259,1.0,1.0,1.0,1,459,1.0,1.0,1.0,1,815,1.0,1.0,1.0 +bm25_desc_only,q106,Resolve a Sentry issue,3,1,251,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,826,1.0,1.0,1.0 +bm25_desc_only,q107,Scale a Kubernetes deployment,1,1,256,1.0,1.0,1.0,1,411,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +bm25_desc_only,q108,Apply a Kubernetes manifest,1,1,226,1.0,1.0,1.0,1,370,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +bm25_desc_only,q109,Stop a docker container,1,1,219,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,581,1.0,1.0,1.0 +bm25_desc_only,q110,Build a docker image,1,1,224,1.0,1.0,1.0,1,317,1.0,1.0,1.0,1,601,1.0,1.0,1.0 +bm25_desc_only,q111,Click an element in Playwright,2,1,216,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,721,1.0,1.0,1.0 +bm25_desc_only,q112,Type text into a browser input,1,1,217,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,636,1.0,1.0,1.0 +bm25_desc_only,q113,List GitLab projects,1,1,209,1.0,1.0,1.0,1,289,1.0,1.0,1.0,1,615,1.0,1.0,1.0 +bm25_desc_only,q114,Create a Bitbucket pull request,1,1,313,1.0,1.0,1.0,1,519,1.0,1.0,1.0,1,1050,1.0,1.0,1.0 +bm25_desc_only,q115,Create a Gmail draft,1,1,205,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,742,1.0,1.0,1.0 +bm25_desc_only,q116,Share a Google Drive file,1,1,306,1.0,1.0,1.0,1,417,1.0,1.0,1.0,1,808,1.0,1.0,1.0 +bm25_desc_only,q117,List Google Calendar events,1,1,317,1.0,1.0,1.0,1,441,1.0,1.0,1.0,1,784,1.0,1.0,1.0 +bm25_desc_only,q118,Delete a calendar event,1,1,300,1.0,1.0,1.0,1,477,1.0,1.0,1.0,1,787,1.0,1.0,1.0 +bm25_desc_only,q119,List Vercel projects,1,1,163,1.0,1.0,1.0,1,248,1.0,1.0,1.0,1,559,1.0,1.0,1.0 +bm25_desc_only,q120,Select rows from Supabase,1,1,237,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,709,1.0,1.0,1.0 +bm25_desc_only,q121,Post a Figma comment,1,1,218,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,720,1.0,1.0,1.0 +bm25_desc_only,q122,List Okta users,1,1,174,1.0,1.0,1.0,1,301,1.0,1.0,1.0,1,618,1.0,1.0,1.0 +bm25_desc_only,q123,Range query Prometheus,1,1,221,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,680,1.0,1.0,1.0 +bm25_desc_only,q124,Create a Grafana annotation,1,1,218,1.0,1.0,1.0,1,372,1.0,1.0,1.0,1,722,1.0,1.0,1.0 +bm25_desc_only,q125,Apply terraform,1,1,190,1.0,1.0,1.0,1,292,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +bm25_desc_only,q126,List Asana tasks,1,1,245,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,659,1.0,1.0,1.0 +bm25_desc_only,q127,Comment on a Trello card,1,1,257,1.0,1.0,1.0,1,438,1.0,1.0,1.0,1,757,1.0,1.0,1.0 +bm25_desc_only,q128,Update a ClickUp task,1,1,257,1.0,1.0,1.0,1,399,1.0,1.0,1.0,1,787,1.0,1.0,1.0 +bm25_desc_only,q129,Search Confluence,1,1,221,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,785,1.0,1.0,1.0 +bm25_desc_only,q130,Create a HubSpot deal,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,838,1.0,1.0,1.0 +bm25_desc_only,q131,List Zendesk tickets,1,1,189,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,603,1.0,1.0,1.0 +bm25_desc_only,q132,List Discord channels,1,1,184,1.0,1.0,1.0,1,300,1.0,1.0,1.0,1,590,1.0,1.0,1.0 +bm25_desc_only,q133,Set a Redis key,1,1,182,1.0,1.0,1.0,1,307,1.0,1.0,1.0,1,564,1.0,1.0,1.0 +bm25_desc_only,q134,Insert MongoDB documents,1,1,256,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +bm25_desc_only,q135,Run a MySQL query,1,1,179,1.0,1.0,1.0,1,281,1.0,1.0,1.0,1,670,1.0,1.0,1.0 +bm25_desc_only,q136,Index a document in Elasticsearch,1,1,242,1.0,1.0,1.0,1,402,1.0,1.0,1.0,1,755,1.0,1.0,1.0 +bm25_desc_only,q137,Get an S3 object,1,1,249,1.0,1.0,1.0,1,355,1.0,1.0,1.0,1,723,1.0,1.0,1.0 +bm25_desc_only,q138,List Cloudflare zones,1,1,182,1.0,1.0,1.0,1,361,1.0,1.0,1.0,1,554,1.0,1.0,1.0 +bm25_desc_only,q139,Create a Stripe invoice,1,1,204,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +bm25_desc_only,q140,List GitHub Actions workflows,1,1,293,1.0,1.0,1.0,1,453,1.0,1.0,1.0,1,812,1.0,1.0,1.0 +bm25_desc_only,q141,Get GitHub Actions job logs,1,1,269,1.0,1.0,1.0,1,442,1.0,1.0,1.0,1,839,1.0,1.0,1.0 +bm25_desc_only,q142,Create a GitHub repository,1,1,268,1.0,1.0,1.0,1,417,1.0,1.0,1.0,1,920,1.0,1.0,1.0 +bm25_desc_only,q143,Checkout a git branch,7,3,766,0.0,0.0,1.0,2,852,0.0,1.0,1.0,1,832,1.0,1.0,1.0 +bm25_desc_only,q144,Show git log,4,2,382,0.0,1.0,1.0,1,311,1.0,1.0,1.0,1,626,1.0,1.0,1.0 +bm25_desc_only,q145,List postgres schemas,1,1,178,1.0,1.0,1.0,1,258,1.0,1.0,1.0,1,485,1.0,1.0,1.0 +bm25_desc_only,q146,Open nodes in the knowledge graph,1,1,163,1.0,1.0,1.0,1,260,1.0,1.0,1.0,1,505,1.0,1.0,1.0 +bm25_desc_only,q147,Get file metadata from the filesystem,5,2,447,0.0,1.0,1.0,1,334,1.0,1.0,1.0,1,750,1.0,1.0,1.0 +bm25_desc_only,q148,Write a new local file,2,1,234,1.0,1.0,1.0,1,376,1.0,1.0,1.0,1,803,1.0,1.0,1.0 +bm25_desc_only,q149,Search GitHub issues,1,1,208,1.0,1.0,1.0,1,335,1.0,1.0,1.0,1,683,1.0,1.0,1.0 +bm25_desc_only,q150,Get the authenticated GitHub user,3,1,179,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,699,1.0,1.0,1.0 +bm25_ident_split,q001,Create a GitHub issue,1,1,380,1.0,1.0,1.0,1,584,1.0,1.0,1.0,1,1035,1.0,1.0,1.0 +bm25_ident_split,q002,Create a Jira ticket,1,1,281,1.0,1.0,1.0,1,466,1.0,1.0,1.0,1,951,1.0,1.0,1.0 +bm25_ident_split,q003,Send a Slack message,2,1,244,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,760,1.0,1.0,1.0 +bm25_ident_split,q004,List open pull requests,1,1,237,1.0,1.0,1.0,1,460,1.0,1.0,1.0,1,836,1.0,1.0,1.0 +bm25_ident_split,q005,Open a pull request on GitHub,1,1,364,1.0,1.0,1.0,1,612,1.0,1.0,1.0,1,1144,1.0,1.0,1.0 +bm25_ident_split,q006,Run a SQL query against postgres,1,1,180,1.0,1.0,1.0,1,295,1.0,1.0,1.0,1,592,1.0,1.0,1.0 +bm25_ident_split,q007,Read a local file from disk,1,1,223,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,809,1.0,1.0,1.0 +bm25_ident_split,q008,What files are in this folder,9,3,717,0.0,0.0,1.0,2,756,0.0,1.0,1.0,1,734,1.0,1.0,1.0 +bm25_ident_split,q009,Remember this fact in the knowledge graph,2,1,154,1.0,1.0,1.0,1,261,1.0,1.0,1.0,1,507,1.0,1.0,1.0 +bm25_ident_split,q010,Search the web with Brave,1,1,258,1.0,1.0,1.0,1,410,1.0,1.0,1.0,1,684,1.0,1.0,1.0 +bm25_ident_split,q011,Fetch the contents of a URL,1,1,208,1.0,1.0,1.0,1,330,1.0,1.0,1.0,1,663,1.0,1.0,1.0 +bm25_ident_split,q012,Show git status,1,1,186,1.0,1.0,1.0,1,299,1.0,1.0,1.0,1,613,1.0,1.0,1.0 +bm25_ident_split,q013,Commit staged changes,1,1,182,1.0,1.0,1.0,1,327,1.0,1.0,1.0,1,663,1.0,1.0,1.0 +bm25_ident_split,q014,Create a Linear issue,2,1,333,1.0,1.0,1.0,1,463,1.0,1.0,1.0,1,917,1.0,1.0,1.0 +bm25_ident_split,q015,Make a task in my project tracker Linear,146,49,12414,0.0,0.0,0.0,30,12154,0.0,0.0,0.0,15,11794,0.0,0.0,0.0 +bm25_ident_split,q016,Post a comment on a GitHub issue,3,1,307,1.0,1.0,1.0,1,478,1.0,1.0,1.0,1,922,1.0,1.0,1.0 +bm25_ident_split,q017,Merge the pull request,1,1,283,1.0,1.0,1.0,1,468,1.0,1.0,1.0,1,1121,1.0,1.0,1.0 +bm25_ident_split,q018,Trigger a GitHub Actions workflow,1,1,294,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,768,1.0,1.0,1.0 +bm25_ident_split,q019,Get Dependabot alerts,1,1,269,1.0,1.0,1.0,1,412,1.0,1.0,1.0,1,715,1.0,1.0,1.0 +bm25_ident_split,q020,List Sentry issues,1,1,238,1.0,1.0,1.0,1,395,1.0,1.0,1.0,1,796,1.0,1.0,1.0 +bm25_ident_split,q021,Create a Stripe customer,1,1,186,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +bm25_ident_split,q022,Refund a Stripe payment,1,1,234,1.0,1.0,1.0,1,331,1.0,1.0,1.0,1,647,1.0,1.0,1.0 +bm25_ident_split,q023,Send an email with Gmail,1,1,224,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,713,1.0,1.0,1.0 +bm25_ident_split,q024,Search my Gmail inbox,1,1,178,1.0,1.0,1.0,1,315,1.0,1.0,1.0,1,691,1.0,1.0,1.0 +bm25_ident_split,q025,Upload a file to Google Drive,4,2,523,0.0,1.0,1.0,1,446,1.0,1.0,1.0,1,774,1.0,1.0,1.0 +bm25_ident_split,q026,Create a calendar event,1,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,801,1.0,1.0,1.0 +bm25_ident_split,q027,List Kubernetes pods,1,1,184,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,718,1.0,1.0,1.0 +bm25_ident_split,q028,Get logs from a k8s pod,1,1,262,1.0,1.0,1.0,1,455,1.0,1.0,1.0,1,830,1.0,1.0,1.0 +bm25_ident_split,q029,Run a docker container,1,1,223,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,597,1.0,1.0,1.0 +bm25_ident_split,q030,Show docker container logs,1,1,188,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,649,1.0,1.0,1.0 +bm25_ident_split,q031,Navigate the browser to a URL,1,1,180,1.0,1.0,1.0,1,346,1.0,1.0,1.0,1,617,1.0,1.0,1.0 +bm25_ident_split,q032,Take a screenshot of the page,1,1,212,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,608,1.0,1.0,1.0 +bm25_ident_split,q033,Create a GitLab merge request,3,1,297,1.0,1.0,1.0,1,475,1.0,1.0,1.0,1,1031,1.0,1.0,1.0 +bm25_ident_split,q034,Search Notion pages,1,1,180,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,671,1.0,1.0,1.0 +bm25_ident_split,q035,Create a Notion page,1,1,230,1.0,1.0,1.0,1,371,1.0,1.0,1.0,1,822,1.0,1.0,1.0 +bm25_ident_split,q036,Query a Notion database,1,1,261,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,668,1.0,1.0,1.0 +bm25_ident_split,q037,Create a Zendesk ticket,1,1,257,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,714,1.0,1.0,1.0 +bm25_ident_split,q038,Create a HubSpot contact,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,899,1.0,1.0,1.0 +bm25_ident_split,q039,Create a Salesforce lead,2,1,252,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,710,1.0,1.0,1.0 +bm25_ident_split,q040,Run a SOQL query,1,1,186,1.0,1.0,1.0,1,332,1.0,1.0,1.0,1,662,1.0,1.0,1.0 +bm25_ident_split,q041,List Shopify orders,1,1,208,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,682,1.0,1.0,1.0 +bm25_ident_split,q042,Create a PagerDuty incident,1,1,259,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,696,1.0,1.0,1.0 +bm25_ident_split,q043,Who is on call in PagerDuty,1,1,219,1.0,1.0,1.0,1,363,1.0,1.0,1.0,1,724,1.0,1.0,1.0 +bm25_ident_split,q044,Query Datadog metrics,1,1,300,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,774,1.0,1.0,1.0 +bm25_ident_split,q045,Search Datadog logs,1,1,242,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,855,1.0,1.0,1.0 +bm25_ident_split,q046,List S3 buckets,1,1,190,1.0,1.0,1.0,1,258,1.0,1.0,1.0,1,474,1.0,1.0,1.0 +bm25_ident_split,q047,Upload an object to S3,1,1,250,1.0,1.0,1.0,1,437,1.0,1.0,1.0,1,850,1.0,1.0,1.0 +bm25_ident_split,q048,Purge Cloudflare cache,1,1,183,1.0,1.0,1.0,1,340,1.0,1.0,1.0,1,746,1.0,1.0,1.0 +bm25_ident_split,q049,Create a Cloudflare DNS record,1,1,249,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,708,1.0,1.0,1.0 +bm25_ident_split,q050,List Vercel deployments,1,1,173,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,529,1.0,1.0,1.0 +bm25_ident_split,q051,Insert a row into Supabase,1,1,203,1.0,1.0,1.0,1,347,1.0,1.0,1.0,1,665,1.0,1.0,1.0 +bm25_ident_split,q052,Export images from Figma,1,1,218,1.0,1.0,1.0,1,328,1.0,1.0,1.0,1,637,1.0,1.0,1.0 +bm25_ident_split,q053,Send an SMS with Twilio,1,1,254,1.0,1.0,1.0,1,396,1.0,1.0,1.0,1,725,1.0,1.0,1.0 +bm25_ident_split,q054,Create an Okta user,1,1,183,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,622,1.0,1.0,1.0 +bm25_ident_split,q055,Instant query Prometheus,1,1,221,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,680,1.0,1.0,1.0 +bm25_ident_split,q056,Search Grafana dashboards,1,1,217,1.0,1.0,1.0,1,360,1.0,1.0,1.0,1,660,1.0,1.0,1.0 +bm25_ident_split,q057,Trigger a CircleCI pipeline,1,1,216,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,755,1.0,1.0,1.0 +bm25_ident_split,q058,Run terraform plan,1,1,172,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,658,1.0,1.0,1.0 +bm25_ident_split,q059,Create an Asana task,1,1,267,1.0,1.0,1.0,1,429,1.0,1.0,1.0,1,825,1.0,1.0,1.0 +bm25_ident_split,q060,Create a Trello card,1,1,240,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,679,1.0,1.0,1.0 +bm25_ident_split,q061,Create a ClickUp task,1,1,257,1.0,1.0,1.0,1,415,1.0,1.0,1.0,1,721,1.0,1.0,1.0 +bm25_ident_split,q062,Create a Confluence page,1,1,260,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,699,1.0,1.0,1.0 +bm25_ident_split,q063,Send a Discord message,1,1,224,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,682,1.0,1.0,1.0 +bm25_ident_split,q064,Search Elasticsearch,1,1,239,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,672,1.0,1.0,1.0 +bm25_ident_split,q065,Get a Redis key,1,1,182,1.0,1.0,1.0,1,281,1.0,1.0,1.0,1,589,1.0,1.0,1.0 +bm25_ident_split,q066,Find MongoDB documents,1,1,244,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,738,1.0,1.0,1.0 +bm25_ident_split,q067,List SQLite tables,1,1,156,1.0,1.0,1.0,1,243,1.0,1.0,1.0,1,496,1.0,1.0,1.0 +bm25_ident_split,q068,Describe a postgres table,1,1,200,1.0,1.0,1.0,1,349,1.0,1.0,1.0,1,597,1.0,1.0,1.0 +bm25_ident_split,q069,github_create_issue,1,1,378,1.0,1.0,1.0,1,568,1.0,1.0,1.0,1,1091,1.0,1.0,1.0 +bm25_ident_split,q070,createIssue,3,1,332,1.0,1.0,1.0,1,579,1.0,1.0,1.0,1,971,1.0,1.0,1.0 +bm25_ident_split,q071,list-directory,2,1,186,1.0,1.0,1.0,1,296,1.0,1.0,1.0,1,607,1.0,1.0,1.0 +bm25_ident_split,q072,sequential thinking next thought,1,1,224,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,797,1.0,1.0,1.0 +bm25_ident_split,q073,What time is it in Tokyo,1,1,217,1.0,1.0,1.0,1,378,1.0,1.0,1.0,1,703,1.0,1.0,1.0 +bm25_ident_split,q074,Convert time between timezones,1,1,204,1.0,1.0,1.0,1,360,1.0,1.0,1.0,1,704,1.0,1.0,1.0 +bm25_ident_split,q075,Star a GitHub repo,1,1,225,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,771,1.0,1.0,1.0 +bm25_ident_split,q076,Fork a repository,1,1,235,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,792,1.0,1.0,1.0 +bm25_ident_split,q077,Get the latest GitHub release,1,1,239,1.0,1.0,1.0,1,383,1.0,1.0,1.0,1,742,1.0,1.0,1.0 +bm25_ident_split,q078,Create a gist,1,1,229,1.0,1.0,1.0,1,335,1.0,1.0,1.0,1,649,1.0,1.0,1.0 +bm25_ident_split,q079,Search GitHub code,1,1,283,1.0,1.0,1.0,1,385,1.0,1.0,1.0,1,725,1.0,1.0,1.0 +bm25_ident_split,q080,List GitHub notifications,1,1,193,1.0,1.0,1.0,1,311,1.0,1.0,1.0,1,662,1.0,1.0,1.0 +bm25_ident_split,q081,Assign Copilot to an issue,1,1,267,1.0,1.0,1.0,1,402,1.0,1.0,1.0,1,833,1.0,1.0,1.0 +bm25_ident_split,q082,Get secret scanning alerts,1,1,295,1.0,1.0,1.0,1,470,1.0,1.0,1.0,1,765,1.0,1.0,1.0 +bm25_ident_split,q083,Create or update a file on GitHub,1,1,369,1.0,1.0,1.0,1,546,1.0,1.0,1.0,1,1001,1.0,1.0,1.0 +bm25_ident_split,q084,List collaborators on a repo,1,1,278,1.0,1.0,1.0,1,433,1.0,1.0,1.0,1,932,1.0,1.0,1.0 +bm25_ident_split,q085,Reply in a Slack thread,1,1,240,1.0,1.0,1.0,1,350,1.0,1.0,1.0,1,632,1.0,1.0,1.0 +bm25_ident_split,q086,Add an emoji reaction on Slack,1,1,271,1.0,1.0,1.0,1,428,1.0,1.0,1.0,1,811,1.0,1.0,1.0 +bm25_ident_split,q087,Search Slack messages,1,1,232,1.0,1.0,1.0,1,373,1.0,1.0,1.0,1,664,1.0,1.0,1.0 +bm25_ident_split,q088,Create a Slack channel,1,1,196,1.0,1.0,1.0,1,342,1.0,1.0,1.0,1,751,1.0,1.0,1.0 +bm25_ident_split,q089,Transition a Jira issue,1,1,208,1.0,1.0,1.0,1,342,1.0,1.0,1.0,1,778,1.0,1.0,1.0 +bm25_ident_split,q090,Search Jira with JQL,1,1,241,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,619,1.0,1.0,1.0 +bm25_ident_split,q091,Add a worklog to Jira,1,1,236,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,874,1.0,1.0,1.0 +bm25_ident_split,q092,List Linear teams,1,1,137,1.0,1.0,1.0,1,295,1.0,1.0,1.0,1,646,1.0,1.0,1.0 +bm25_ident_split,q093,Comment on a Linear issue,1,1,244,1.0,1.0,1.0,1,360,1.0,1.0,1.0,1,777,1.0,1.0,1.0 +bm25_ident_split,q094,Move a local file,1,1,224,1.0,1.0,1.0,1,358,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +bm25_ident_split,q095,Recursively search files by pattern,1,1,200,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,721,1.0,1.0,1.0 +bm25_ident_split,q096,Edit a file in place with old and new string,1,1,247,1.0,1.0,1.0,1,358,1.0,1.0,1.0,1,673,1.0,1.0,1.0 +bm25_ident_split,q097,Read the whole knowledge graph,1,1,152,1.0,1.0,1.0,1,259,1.0,1.0,1.0,1,561,1.0,1.0,1.0 +bm25_ident_split,q098,Search memory nodes,1,1,202,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,635,1.0,1.0,1.0 +bm25_ident_split,q099,Show unstaged git diff,1,1,190,1.0,1.0,1.0,1,313,1.0,1.0,1.0,1,602,1.0,1.0,1.0 +bm25_ident_split,q100,Stage files with git add,1,1,197,1.0,1.0,1.0,1,417,1.0,1.0,1.0,1,872,1.0,1.0,1.0 +bm25_ident_split,q101,Create a Shopify product,1,1,240,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +bm25_ident_split,q102,Cancel a Stripe subscription,1,1,167,1.0,1.0,1.0,1,285,1.0,1.0,1.0,1,610,1.0,1.0,1.0 +bm25_ident_split,q103,Create a payment intent,1,1,234,1.0,1.0,1.0,1,340,1.0,1.0,1.0,1,654,1.0,1.0,1.0 +bm25_ident_split,q104,Acknowledge a PagerDuty incident,1,1,223,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,717,1.0,1.0,1.0 +bm25_ident_split,q105,Create a Datadog monitor,1,1,259,1.0,1.0,1.0,1,459,1.0,1.0,1.0,1,758,1.0,1.0,1.0 +bm25_ident_split,q106,Resolve a Sentry issue,2,1,251,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,721,1.0,1.0,1.0 +bm25_ident_split,q107,Scale a Kubernetes deployment,1,1,245,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +bm25_ident_split,q108,Apply a Kubernetes manifest,1,1,204,1.0,1.0,1.0,1,308,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +bm25_ident_split,q109,Stop a docker container,1,1,188,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,641,1.0,1.0,1.0 +bm25_ident_split,q110,Build a docker image,1,1,211,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,601,1.0,1.0,1.0 +bm25_ident_split,q111,Click an element in Playwright,1,1,216,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,612,1.0,1.0,1.0 +bm25_ident_split,q112,Type text into a browser input,1,1,235,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,648,1.0,1.0,1.0 +bm25_ident_split,q113,List GitLab projects,1,1,171,1.0,1.0,1.0,1,255,1.0,1.0,1.0,1,609,1.0,1.0,1.0 +bm25_ident_split,q114,Create a Bitbucket pull request,1,1,313,1.0,1.0,1.0,1,546,1.0,1.0,1.0,1,1139,1.0,1.0,1.0 +bm25_ident_split,q115,Create a Gmail draft,1,1,205,1.0,1.0,1.0,1,315,1.0,1.0,1.0,1,758,1.0,1.0,1.0 +bm25_ident_split,q116,Share a Google Drive file,1,1,254,1.0,1.0,1.0,1,417,1.0,1.0,1.0,1,725,1.0,1.0,1.0 +bm25_ident_split,q117,List Google Calendar events,1,1,219,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,753,1.0,1.0,1.0 +bm25_ident_split,q118,Delete a calendar event,1,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,735,1.0,1.0,1.0 +bm25_ident_split,q119,List Vercel projects,1,1,151,1.0,1.0,1.0,1,248,1.0,1.0,1.0,1,550,1.0,1.0,1.0 +bm25_ident_split,q120,Select rows from Supabase,1,1,237,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,660,1.0,1.0,1.0 +bm25_ident_split,q121,Post a Figma comment,1,1,204,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,646,1.0,1.0,1.0 +bm25_ident_split,q122,List Okta users,1,1,177,1.0,1.0,1.0,1,271,1.0,1.0,1.0,1,606,1.0,1.0,1.0 +bm25_ident_split,q123,Range query Prometheus,1,1,221,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,680,1.0,1.0,1.0 +bm25_ident_split,q124,Create a Grafana annotation,1,1,218,1.0,1.0,1.0,1,324,1.0,1.0,1.0,1,638,1.0,1.0,1.0 +bm25_ident_split,q125,Apply terraform,1,1,190,1.0,1.0,1.0,1,292,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +bm25_ident_split,q126,List Asana tasks,1,1,192,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,642,1.0,1.0,1.0 +bm25_ident_split,q127,Comment on a Trello card,1,1,260,1.0,1.0,1.0,1,422,1.0,1.0,1.0,1,703,1.0,1.0,1.0 +bm25_ident_split,q128,Update a ClickUp task,1,1,257,1.0,1.0,1.0,1,399,1.0,1.0,1.0,1,761,1.0,1.0,1.0 +bm25_ident_split,q129,Search Confluence,1,1,221,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,691,1.0,1.0,1.0 +bm25_ident_split,q130,Create a HubSpot deal,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,899,1.0,1.0,1.0 +bm25_ident_split,q131,List Zendesk tickets,1,1,189,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,605,1.0,1.0,1.0 +bm25_ident_split,q132,List Discord channels,1,1,184,1.0,1.0,1.0,1,300,1.0,1.0,1.0,1,592,1.0,1.0,1.0 +bm25_ident_split,q133,Set a Redis key,1,1,182,1.0,1.0,1.0,1,302,1.0,1.0,1.0,1,600,1.0,1.0,1.0 +bm25_ident_split,q134,Insert MongoDB documents,1,1,244,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +bm25_ident_split,q135,Run a MySQL query,1,1,179,1.0,1.0,1.0,1,281,1.0,1.0,1.0,1,611,1.0,1.0,1.0 +bm25_ident_split,q136,Index a document in Elasticsearch,1,1,242,1.0,1.0,1.0,1,346,1.0,1.0,1.0,1,605,1.0,1.0,1.0 +bm25_ident_split,q137,Get an S3 object,1,1,249,1.0,1.0,1.0,1,355,1.0,1.0,1.0,1,668,1.0,1.0,1.0 +bm25_ident_split,q138,List Cloudflare zones,1,1,182,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,579,1.0,1.0,1.0 +bm25_ident_split,q139,Create a Stripe invoice,1,1,204,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +bm25_ident_split,q140,List GitHub Actions workflows,1,1,293,1.0,1.0,1.0,1,457,1.0,1.0,1.0,1,790,1.0,1.0,1.0 +bm25_ident_split,q141,Get GitHub Actions job logs,1,1,269,1.0,1.0,1.0,1,442,1.0,1.0,1.0,1,786,1.0,1.0,1.0 +bm25_ident_split,q142,Create a GitHub repository,1,1,235,1.0,1.0,1.0,1,406,1.0,1.0,1.0,1,874,1.0,1.0,1.0 +bm25_ident_split,q143,Checkout a git branch,1,1,266,1.0,1.0,1.0,1,429,1.0,1.0,1.0,1,1001,1.0,1.0,1.0 +bm25_ident_split,q144,Show git log,1,1,196,1.0,1.0,1.0,1,299,1.0,1.0,1.0,1,600,1.0,1.0,1.0 +bm25_ident_split,q145,List postgres schemas,1,1,160,1.0,1.0,1.0,1,277,1.0,1.0,1.0,1,515,1.0,1.0,1.0 +bm25_ident_split,q146,Open nodes in the knowledge graph,1,1,163,1.0,1.0,1.0,1,260,1.0,1.0,1.0,1,531,1.0,1.0,1.0 +bm25_ident_split,q147,Get file metadata from the filesystem,2,1,220,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,766,1.0,1.0,1.0 +bm25_ident_split,q148,Write a new local file,1,1,234,1.0,1.0,1.0,1,384,1.0,1.0,1.0,1,791,1.0,1.0,1.0 +bm25_ident_split,q149,Search GitHub issues,1,1,233,1.0,1.0,1.0,1,385,1.0,1.0,1.0,1,683,1.0,1.0,1.0 +bm25_ident_split,q150,Get the authenticated GitHub user,2,1,179,1.0,1.0,1.0,1,304,1.0,1.0,1.0,1,622,1.0,1.0,1.0 +bm25_name_only,q001,Create a GitHub issue,1,1,369,1.0,1.0,1.0,1,575,1.0,1.0,1.0,1,911,1.0,1.0,1.0 +bm25_name_only,q002,Create a Jira ticket,2,1,333,1.0,1.0,1.0,1,481,1.0,1.0,1.0,1,808,1.0,1.0,1.0 +bm25_name_only,q003,Send a Slack message,2,1,250,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,737,1.0,1.0,1.0 +bm25_name_only,q004,List open pull requests,1,1,211,1.0,1.0,1.0,1,434,1.0,1.0,1.0,1,1009,1.0,1.0,1.0 +bm25_name_only,q005,Open a pull request on GitHub,1,1,405,1.0,1.0,1.0,1,560,1.0,1.0,1.0,1,978,1.0,1.0,1.0 +bm25_name_only,q006,Run a SQL query against postgres,1,1,223,1.0,1.0,1.0,1,335,1.0,1.0,1.0,1,714,1.0,1.0,1.0 +bm25_name_only,q007,Read a local file from disk,1,1,223,1.0,1.0,1.0,1,398,1.0,1.0,1.0,1,798,1.0,1.0,1.0 +bm25_name_only,q008,What files are in this folder,85,29,7932,0.0,0.0,0.0,17,7554,0.0,0.0,0.0,9,7648,0.0,0.0,0.0 +bm25_name_only,q009,Remember this fact in the knowledge graph,90,30,8091,0.0,0.0,0.0,18,7815,0.0,0.0,0.0,9,7608,0.0,0.0,0.0 +bm25_name_only,q010,Search the web with Brave,1,1,245,1.0,1.0,1.0,1,404,1.0,1.0,1.0,1,713,1.0,1.0,1.0 +bm25_name_only,q011,Fetch the contents of a URL,1,1,261,1.0,1.0,1.0,1,366,1.0,1.0,1.0,1,765,1.0,1.0,1.0 +bm25_name_only,q012,Show git status,1,1,196,1.0,1.0,1.0,1,325,1.0,1.0,1.0,1,597,1.0,1.0,1.0 +bm25_name_only,q013,Commit staged changes,2,1,218,1.0,1.0,1.0,1,310,1.0,1.0,1.0,1,746,1.0,1.0,1.0 +bm25_name_only,q014,Create a Linear issue,1,1,322,1.0,1.0,1.0,1,528,1.0,1.0,1.0,1,974,1.0,1.0,1.0 +bm25_name_only,q015,Make a task in my project tracker Linear,146,49,12480,0.0,0.0,0.0,30,12220,0.0,0.0,0.0,15,11860,0.0,0.0,0.0 +bm25_name_only,q016,Post a comment on a GitHub issue,2,1,307,1.0,1.0,1.0,1,429,1.0,1.0,1.0,1,732,1.0,1.0,1.0 +bm25_name_only,q017,Merge the pull request,1,1,354,1.0,1.0,1.0,1,586,1.0,1.0,1.0,1,1108,1.0,1.0,1.0 +bm25_name_only,q018,Trigger a GitHub Actions workflow,1,1,294,1.0,1.0,1.0,1,416,1.0,1.0,1.0,1,719,1.0,1.0,1.0 +bm25_name_only,q019,Get Dependabot alerts,1,1,287,1.0,1.0,1.0,1,421,1.0,1.0,1.0,1,801,1.0,1.0,1.0 +bm25_name_only,q020,List Sentry issues,1,1,289,1.0,1.0,1.0,1,425,1.0,1.0,1.0,1,796,1.0,1.0,1.0 +bm25_name_only,q021,Create a Stripe customer,1,1,186,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +bm25_name_only,q022,Refund a Stripe payment,1,1,234,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,662,1.0,1.0,1.0 +bm25_name_only,q023,Send an email with Gmail,1,1,262,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,716,1.0,1.0,1.0 +bm25_name_only,q024,Search my Gmail inbox,1,1,238,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,714,1.0,1.0,1.0 +bm25_name_only,q025,Upload a file to Google Drive,254,85,20407,0.0,0.0,0.0,51,19659,0.0,0.0,0.0,26,19548,0.0,0.0,0.0 +bm25_name_only,q026,Create a calendar event,1,1,300,1.0,1.0,1.0,1,480,1.0,1.0,1.0,1,844,1.0,1.0,1.0 +bm25_name_only,q027,List Kubernetes pods,1,1,251,1.0,1.0,1.0,1,393,1.0,1.0,1.0,1,750,1.0,1.0,1.0 +bm25_name_only,q028,Get logs from a k8s pod,1,1,262,1.0,1.0,1.0,1,455,1.0,1.0,1.0,1,803,1.0,1.0,1.0 +bm25_name_only,q029,Run a docker container,1,1,221,1.0,1.0,1.0,1,327,1.0,1.0,1.0,1,709,1.0,1.0,1.0 +bm25_name_only,q030,Show docker container logs,1,1,241,1.0,1.0,1.0,1,380,1.0,1.0,1.0,1,648,1.0,1.0,1.0 +bm25_name_only,q031,Navigate the browser to a URL,1,1,169,1.0,1.0,1.0,1,309,1.0,1.0,1.0,1,756,1.0,1.0,1.0 +bm25_name_only,q032,Take a screenshot of the page,1,1,226,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,713,1.0,1.0,1.0 +bm25_name_only,q033,Create a GitLab merge request,2,1,323,1.0,1.0,1.0,1,552,1.0,1.0,1.0,1,881,1.0,1.0,1.0 +bm25_name_only,q034,Search Notion pages,1,1,229,1.0,1.0,1.0,1,351,1.0,1.0,1.0,1,686,1.0,1.0,1.0 +bm25_name_only,q035,Create a Notion page,1,1,230,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,748,1.0,1.0,1.0 +bm25_name_only,q036,Query a Notion database,1,1,245,1.0,1.0,1.0,1,360,1.0,1.0,1.0,1,687,1.0,1.0,1.0 +bm25_name_only,q037,Create a Zendesk ticket,1,1,257,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,813,1.0,1.0,1.0 +bm25_name_only,q038,Create a HubSpot contact,1,1,244,1.0,1.0,1.0,1,476,1.0,1.0,1.0,1,813,1.0,1.0,1.0 +bm25_name_only,q039,Create a Salesforce lead,1,1,252,1.0,1.0,1.0,1,402,1.0,1.0,1.0,1,784,1.0,1.0,1.0 +bm25_name_only,q040,Run a SOQL query,4,2,408,0.0,1.0,1.0,1,334,1.0,1.0,1.0,1,713,1.0,1.0,1.0 +bm25_name_only,q041,List Shopify orders,1,1,208,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,728,1.0,1.0,1.0 +bm25_name_only,q042,Create a PagerDuty incident,1,1,259,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,818,1.0,1.0,1.0 +bm25_name_only,q043,Who is on call in PagerDuty,244,82,19786,0.0,0.0,0.0,49,18956,0.0,0.0,0.0,25,18737,0.0,0.0,0.0 +bm25_name_only,q044,Query Datadog metrics,1,1,225,1.0,1.0,1.0,1,340,1.0,1.0,1.0,1,619,1.0,1.0,1.0 +bm25_name_only,q045,Search Datadog logs,1,1,280,1.0,1.0,1.0,1,444,1.0,1.0,1.0,1,871,1.0,1.0,1.0 +bm25_name_only,q046,List S3 buckets,1,1,236,1.0,1.0,1.0,1,378,1.0,1.0,1.0,1,735,1.0,1.0,1.0 +bm25_name_only,q047,Upload an object to S3,2,1,250,1.0,1.0,1.0,1,437,1.0,1.0,1.0,1,839,1.0,1.0,1.0 +bm25_name_only,q048,Purge Cloudflare cache,1,1,183,1.0,1.0,1.0,1,341,1.0,1.0,1.0,1,756,1.0,1.0,1.0 +bm25_name_only,q049,Create a Cloudflare DNS record,1,1,249,1.0,1.0,1.0,1,415,1.0,1.0,1.0,1,879,1.0,1.0,1.0 +bm25_name_only,q050,List Vercel deployments,1,1,173,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,710,1.0,1.0,1.0 +bm25_name_only,q051,Insert a row into Supabase,1,1,230,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,679,1.0,1.0,1.0 +bm25_name_only,q052,Export images from Figma,1,1,196,1.0,1.0,1.0,1,325,1.0,1.0,1.0,1,651,1.0,1.0,1.0 +bm25_name_only,q053,Send an SMS with Twilio,1,1,261,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,718,1.0,1.0,1.0 +bm25_name_only,q054,Create an Okta user,1,1,183,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,631,1.0,1.0,1.0 +bm25_name_only,q055,Instant query Prometheus,6,2,365,0.0,1.0,1.0,2,704,0.0,1.0,1.0,1,684,1.0,1.0,1.0 +bm25_name_only,q056,Search Grafana dashboards,1,1,217,1.0,1.0,1.0,1,376,1.0,1.0,1.0,1,685,1.0,1.0,1.0 +bm25_name_only,q057,Trigger a CircleCI pipeline,1,1,308,1.0,1.0,1.0,1,430,1.0,1.0,1.0,1,701,1.0,1.0,1.0 +bm25_name_only,q058,Run terraform plan,1,1,275,1.0,1.0,1.0,1,430,1.0,1.0,1.0,1,712,1.0,1.0,1.0 +bm25_name_only,q059,Create an Asana task,1,1,267,1.0,1.0,1.0,1,429,1.0,1.0,1.0,1,821,1.0,1.0,1.0 +bm25_name_only,q060,Create a Trello card,1,1,245,1.0,1.0,1.0,1,396,1.0,1.0,1.0,1,809,1.0,1.0,1.0 +bm25_name_only,q061,Create a ClickUp task,4,2,496,0.0,1.0,1.0,1,386,1.0,1.0,1.0,1,800,1.0,1.0,1.0 +bm25_name_only,q062,Create a Confluence page,1,1,260,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,835,1.0,1.0,1.0 +bm25_name_only,q063,Send a Discord message,1,1,250,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,664,1.0,1.0,1.0 +bm25_name_only,q064,Search Elasticsearch,290,97,22834,0.0,0.0,0.0,58,22047,0.0,0.0,0.0,29,21525,0.0,0.0,0.0 +bm25_name_only,q065,Get a Redis key,1,1,189,1.0,1.0,1.0,1,280,1.0,1.0,1.0,1,628,1.0,1.0,1.0 +bm25_name_only,q066,Find MongoDB documents,1,1,258,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,738,1.0,1.0,1.0 +bm25_name_only,q067,List SQLite tables,2,1,151,1.0,1.0,1.0,1,294,1.0,1.0,1.0,1,675,1.0,1.0,1.0 +bm25_name_only,q068,Describe a postgres table,1,1,187,1.0,1.0,1.0,1,292,1.0,1.0,1.0,1,714,1.0,1.0,1.0 +bm25_name_only,q069,github_create_issue,1,1,398,1.0,1.0,1.0,1,580,1.0,1.0,1.0,1,1139,1.0,1.0,1.0 +bm25_name_only,q070,createIssue,1,1,397,1.0,1.0,1.0,1,579,1.0,1.0,1.0,1,1138,1.0,1.0,1.0 +bm25_name_only,q071,list-directory,1,1,182,1.0,1.0,1.0,1,344,1.0,1.0,1.0,1,725,1.0,1.0,1.0 +bm25_name_only,q072,sequential thinking next thought,116,39,9799,0.0,0.0,0.0,24,9727,0.0,0.0,0.0,12,9475,0.0,0.0,0.0 +bm25_name_only,q073,What time is it in Tokyo,1,1,204,1.0,1.0,1.0,1,309,1.0,1.0,1.0,1,731,1.0,1.0,1.0 +bm25_name_only,q074,Convert time between timezones,1,1,204,1.0,1.0,1.0,1,296,1.0,1.0,1.0,1,732,1.0,1.0,1.0 +bm25_name_only,q075,Star a GitHub repo,1,1,204,1.0,1.0,1.0,1,333,1.0,1.0,1.0,1,612,1.0,1.0,1.0 +bm25_name_only,q076,Fork a repository,1,1,248,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,729,1.0,1.0,1.0 +bm25_name_only,q077,Get the latest GitHub release,1,1,245,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,704,1.0,1.0,1.0 +bm25_name_only,q078,Create a gist,1,1,229,1.0,1.0,1.0,1,409,1.0,1.0,1.0,1,800,1.0,1.0,1.0 +bm25_name_only,q079,Search GitHub code,1,1,204,1.0,1.0,1.0,1,333,1.0,1.0,1.0,1,612,1.0,1.0,1.0 +bm25_name_only,q080,List GitHub notifications,1,1,175,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,601,1.0,1.0,1.0 +bm25_name_only,q081,Assign Copilot to an issue,1,1,267,1.0,1.0,1.0,1,457,1.0,1.0,1.0,1,1083,1.0,1.0,1.0 +bm25_name_only,q082,Get secret scanning alerts,1,1,295,1.0,1.0,1.0,1,470,1.0,1.0,1.0,1,818,1.0,1.0,1.0 +bm25_name_only,q083,Create or update a file on GitHub,1,1,333,1.0,1.0,1.0,1,508,1.0,1.0,1.0,1,861,1.0,1.0,1.0 +bm25_name_only,q084,List collaborators on a repo,1,1,274,1.0,1.0,1.0,1,416,1.0,1.0,1.0,1,773,1.0,1.0,1.0 +bm25_name_only,q085,Reply in a Slack thread,1,1,257,1.0,1.0,1.0,1,414,1.0,1.0,1.0,1,740,1.0,1.0,1.0 +bm25_name_only,q086,Add an emoji reaction on Slack,1,1,256,1.0,1.0,1.0,1,426,1.0,1.0,1.0,1,792,1.0,1.0,1.0 +bm25_name_only,q087,Search Slack messages,1,1,231,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,665,1.0,1.0,1.0 +bm25_name_only,q088,Create a Slack channel,1,1,196,1.0,1.0,1.0,1,342,1.0,1.0,1.0,1,685,1.0,1.0,1.0 +bm25_name_only,q089,Transition a Jira issue,1,1,270,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,763,1.0,1.0,1.0 +bm25_name_only,q090,Search Jira with JQL,1,1,267,1.0,1.0,1.0,1,385,1.0,1.0,1.0,1,733,1.0,1.0,1.0 +bm25_name_only,q091,Add a worklog to Jira,1,1,334,1.0,1.0,1.0,1,455,1.0,1.0,1.0,1,914,1.0,1.0,1.0 +bm25_name_only,q092,List Linear teams,1,1,176,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,728,1.0,1.0,1.0 +bm25_name_only,q093,Comment on a Linear issue,1,1,334,1.0,1.0,1.0,1,464,1.0,1.0,1.0,1,885,1.0,1.0,1.0 +bm25_name_only,q094,Move a local file,1,1,271,1.0,1.0,1.0,1,398,1.0,1.0,1.0,1,760,1.0,1.0,1.0 +bm25_name_only,q095,Recursively search files by pattern,1,1,280,1.0,1.0,1.0,1,463,1.0,1.0,1.0,1,772,1.0,1.0,1.0 +bm25_name_only,q096,Edit a file in place with old and new string,1,1,227,1.0,1.0,1.0,1,398,1.0,1.0,1.0,1,746,1.0,1.0,1.0 +bm25_name_only,q097,Read the whole knowledge graph,1,1,211,1.0,1.0,1.0,1,387,1.0,1.0,1.0,1,642,1.0,1.0,1.0 +bm25_name_only,q098,Search memory nodes,1,1,202,1.0,1.0,1.0,1,361,1.0,1.0,1.0,1,670,1.0,1.0,1.0 +bm25_name_only,q099,Show unstaged git diff,1,1,200,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,604,1.0,1.0,1.0 +bm25_name_only,q100,Stage files with git add,1,1,269,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,687,1.0,1.0,1.0 +bm25_name_only,q101,Create a Shopify product,1,1,240,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +bm25_name_only,q102,Cancel a Stripe subscription,1,1,188,1.0,1.0,1.0,1,301,1.0,1.0,1.0,1,632,1.0,1.0,1.0 +bm25_name_only,q103,Create a payment intent,1,1,245,1.0,1.0,1.0,1,477,1.0,1.0,1.0,1,814,1.0,1.0,1.0 +bm25_name_only,q104,Acknowledge a PagerDuty incident,1,1,261,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,787,1.0,1.0,1.0 +bm25_name_only,q105,Create a Datadog monitor,1,1,297,1.0,1.0,1.0,1,459,1.0,1.0,1.0,1,904,1.0,1.0,1.0 +bm25_name_only,q106,Resolve a Sentry issue,2,1,245,1.0,1.0,1.0,1,412,1.0,1.0,1.0,1,935,1.0,1.0,1.0 +bm25_name_only,q107,Scale a Kubernetes deployment,1,1,232,1.0,1.0,1.0,1,337,1.0,1.0,1.0,1,759,1.0,1.0,1.0 +bm25_name_only,q108,Apply a Kubernetes manifest,1,1,202,1.0,1.0,1.0,1,307,1.0,1.0,1.0,1,729,1.0,1.0,1.0 +bm25_name_only,q109,Stop a docker container,1,1,174,1.0,1.0,1.0,1,327,1.0,1.0,1.0,1,581,1.0,1.0,1.0 +bm25_name_only,q110,Build a docker image,1,1,193,1.0,1.0,1.0,1,332,1.0,1.0,1.0,1,581,1.0,1.0,1.0 +bm25_name_only,q111,Click an element in Playwright,1,1,193,1.0,1.0,1.0,1,298,1.0,1.0,1.0,1,720,1.0,1.0,1.0 +bm25_name_only,q112,Type text into a browser input,1,1,236,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,598,1.0,1.0,1.0 +bm25_name_only,q113,List GitLab projects,5,2,326,0.0,1.0,1.0,1,268,1.0,1.0,1.0,1,573,1.0,1.0,1.0 +bm25_name_only,q114,Create a Bitbucket pull request,7,3,1123,0.0,0.0,1.0,2,1177,0.0,1.0,1.0,1,1155,1.0,1.0,1.0 +bm25_name_only,q115,Create a Gmail draft,1,1,265,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,814,1.0,1.0,1.0 +bm25_name_only,q116,Share a Google Drive file,1,1,278,1.0,1.0,1.0,1,440,1.0,1.0,1.0,1,771,1.0,1.0,1.0 +bm25_name_only,q117,List Google Calendar events,2,1,233,1.0,1.0,1.0,1,408,1.0,1.0,1.0,1,789,1.0,1.0,1.0 +bm25_name_only,q118,Delete a calendar event,1,1,300,1.0,1.0,1.0,1,455,1.0,1.0,1.0,1,798,1.0,1.0,1.0 +bm25_name_only,q119,List Vercel projects,1,1,183,1.0,1.0,1.0,1,263,1.0,1.0,1.0,1,576,1.0,1.0,1.0 +bm25_name_only,q120,Select rows from Supabase,1,1,237,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,711,1.0,1.0,1.0 +bm25_name_only,q121,Post a Figma comment,1,1,218,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,721,1.0,1.0,1.0 +bm25_name_only,q122,List Okta users,1,1,209,1.0,1.0,1.0,1,327,1.0,1.0,1.0,1,606,1.0,1.0,1.0 +bm25_name_only,q123,Range query Prometheus,1,1,227,1.0,1.0,1.0,1,339,1.0,1.0,1.0,1,684,1.0,1.0,1.0 +bm25_name_only,q124,Create a Grafana annotation,1,1,218,1.0,1.0,1.0,1,398,1.0,1.0,1.0,1,762,1.0,1.0,1.0 +bm25_name_only,q125,Apply terraform,2,1,200,1.0,1.0,1.0,1,305,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +bm25_name_only,q126,List Asana tasks,1,1,245,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,774,1.0,1.0,1.0 +bm25_name_only,q127,Comment on a Trello card,2,1,260,1.0,1.0,1.0,1,412,1.0,1.0,1.0,1,729,1.0,1.0,1.0 +bm25_name_only,q128,Update a ClickUp task,2,1,264,1.0,1.0,1.0,1,423,1.0,1.0,1.0,1,788,1.0,1.0,1.0 +bm25_name_only,q129,Search Confluence,1,1,221,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,712,1.0,1.0,1.0 +bm25_name_only,q130,Create a HubSpot deal,1,1,292,1.0,1.0,1.0,1,502,1.0,1.0,1.0,1,856,1.0,1.0,1.0 +bm25_name_only,q131,List Zendesk tickets,1,1,219,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,786,1.0,1.0,1.0 +bm25_name_only,q132,List Discord channels,1,1,184,1.0,1.0,1.0,1,300,1.0,1.0,1.0,1,705,1.0,1.0,1.0 +bm25_name_only,q133,Set a Redis key,1,1,189,1.0,1.0,1.0,1,280,1.0,1.0,1.0,1,603,1.0,1.0,1.0 +bm25_name_only,q134,Insert MongoDB documents,1,1,237,1.0,1.0,1.0,1,399,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +bm25_name_only,q135,Run a MySQL query,6,2,408,0.0,1.0,1.0,2,734,0.0,1.0,1.0,1,713,1.0,1.0,1.0 +bm25_name_only,q136,Index a document in Elasticsearch,1,1,198,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,771,1.0,1.0,1.0 +bm25_name_only,q137,Get an S3 object,1,1,249,1.0,1.0,1.0,1,341,1.0,1.0,1.0,1,735,1.0,1.0,1.0 +bm25_name_only,q138,List Cloudflare zones,1,1,232,1.0,1.0,1.0,1,374,1.0,1.0,1.0,1,731,1.0,1.0,1.0 +bm25_name_only,q139,Create a Stripe invoice,1,1,204,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +bm25_name_only,q140,List GitHub Actions workflows,1,1,293,1.0,1.0,1.0,1,409,1.0,1.0,1.0,1,710,1.0,1.0,1.0 +bm25_name_only,q141,Get GitHub Actions job logs,1,1,256,1.0,1.0,1.0,1,385,1.0,1.0,1.0,1,822,1.0,1.0,1.0 +bm25_name_only,q142,Create a GitHub repository,1,1,252,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,717,1.0,1.0,1.0 +bm25_name_only,q143,Checkout a git branch,1,1,212,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,667,1.0,1.0,1.0 +bm25_name_only,q144,Show git log,1,1,194,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,597,1.0,1.0,1.0 +bm25_name_only,q145,List postgres schemas,1,1,231,1.0,1.0,1.0,1,373,1.0,1.0,1.0,1,730,1.0,1.0,1.0 +bm25_name_only,q146,Open nodes in the knowledge graph,1,1,163,1.0,1.0,1.0,1,293,1.0,1.0,1.0,1,699,1.0,1.0,1.0 +bm25_name_only,q147,Get file metadata from the filesystem,4,2,503,0.0,1.0,1.0,1,390,1.0,1.0,1.0,1,782,1.0,1.0,1.0 +bm25_name_only,q148,Write a new local file,1,1,217,1.0,1.0,1.0,1,437,1.0,1.0,1.0,1,959,1.0,1.0,1.0 +bm25_name_only,q149,Search GitHub issues,1,1,233,1.0,1.0,1.0,1,387,1.0,1.0,1.0,1,690,1.0,1.0,1.0 +bm25_name_only,q150,Get the authenticated GitHub user,11,4,796,0.0,0.0,0.0,3,1039,0.0,0.0,1.0,2,1422,0.0,1.0,1.0 +bm25_plain,q001,Create a GitHub issue,1,1,380,1.0,1.0,1.0,1,570,1.0,1.0,1.0,1,1093,1.0,1.0,1.0 +bm25_plain,q002,Create a Jira ticket,1,1,281,1.0,1.0,1.0,1,466,1.0,1.0,1.0,1,951,1.0,1.0,1.0 +bm25_plain,q003,Send a Slack message,2,1,244,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,760,1.0,1.0,1.0 +bm25_plain,q004,List open pull requests,1,1,237,1.0,1.0,1.0,1,460,1.0,1.0,1.0,1,836,1.0,1.0,1.0 +bm25_plain,q005,Open a pull request on GitHub,1,1,364,1.0,1.0,1.0,1,564,1.0,1.0,1.0,1,1180,1.0,1.0,1.0 +bm25_plain,q006,Run a SQL query against postgres,1,1,180,1.0,1.0,1.0,1,295,1.0,1.0,1.0,1,592,1.0,1.0,1.0 +bm25_plain,q007,Read a local file from disk,1,1,223,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,809,1.0,1.0,1.0 +bm25_plain,q008,What files are in this folder,9,3,717,0.0,0.0,1.0,2,756,0.0,1.0,1.0,1,734,1.0,1.0,1.0 +bm25_plain,q009,Remember this fact in the knowledge graph,2,1,154,1.0,1.0,1.0,1,261,1.0,1.0,1.0,1,507,1.0,1.0,1.0 +bm25_plain,q010,Search the web with Brave,1,1,258,1.0,1.0,1.0,1,410,1.0,1.0,1.0,1,684,1.0,1.0,1.0 +bm25_plain,q011,Fetch the contents of a URL,1,1,208,1.0,1.0,1.0,1,330,1.0,1.0,1.0,1,663,1.0,1.0,1.0 +bm25_plain,q012,Show git status,1,1,186,1.0,1.0,1.0,1,299,1.0,1.0,1.0,1,613,1.0,1.0,1.0 +bm25_plain,q013,Commit staged changes,1,1,182,1.0,1.0,1.0,1,327,1.0,1.0,1.0,1,663,1.0,1.0,1.0 +bm25_plain,q014,Create a Linear issue,2,1,333,1.0,1.0,1.0,1,463,1.0,1.0,1.0,1,917,1.0,1.0,1.0 +bm25_plain,q015,Make a task in my project tracker Linear,146,49,12414,0.0,0.0,0.0,30,12154,0.0,0.0,0.0,15,11794,0.0,0.0,0.0 +bm25_plain,q016,Post a comment on a GitHub issue,3,1,307,1.0,1.0,1.0,1,442,1.0,1.0,1.0,1,876,1.0,1.0,1.0 +bm25_plain,q017,Merge the pull request,1,1,283,1.0,1.0,1.0,1,468,1.0,1.0,1.0,1,1121,1.0,1.0,1.0 +bm25_plain,q018,Trigger a GitHub Actions workflow,1,1,294,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,721,1.0,1.0,1.0 +bm25_plain,q019,Get Dependabot alerts,1,1,269,1.0,1.0,1.0,1,412,1.0,1.0,1.0,1,715,1.0,1.0,1.0 +bm25_plain,q020,List Sentry issues,1,1,238,1.0,1.0,1.0,1,395,1.0,1.0,1.0,1,796,1.0,1.0,1.0 +bm25_plain,q021,Create a Stripe customer,1,1,186,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +bm25_plain,q022,Refund a Stripe payment,1,1,234,1.0,1.0,1.0,1,331,1.0,1.0,1.0,1,647,1.0,1.0,1.0 +bm25_plain,q023,Send an email with Gmail,1,1,224,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,713,1.0,1.0,1.0 +bm25_plain,q024,Search my Gmail inbox,1,1,178,1.0,1.0,1.0,1,315,1.0,1.0,1.0,1,691,1.0,1.0,1.0 +bm25_plain,q025,Upload a file to Google Drive,4,2,523,0.0,1.0,1.0,1,446,1.0,1.0,1.0,1,774,1.0,1.0,1.0 +bm25_plain,q026,Create a calendar event,1,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,801,1.0,1.0,1.0 +bm25_plain,q027,List Kubernetes pods,1,1,184,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,718,1.0,1.0,1.0 +bm25_plain,q028,Get logs from a k8s pod,1,1,262,1.0,1.0,1.0,1,455,1.0,1.0,1.0,1,830,1.0,1.0,1.0 +bm25_plain,q029,Run a docker container,1,1,223,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,597,1.0,1.0,1.0 +bm25_plain,q030,Show docker container logs,1,1,188,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,649,1.0,1.0,1.0 +bm25_plain,q031,Navigate the browser to a URL,1,1,180,1.0,1.0,1.0,1,346,1.0,1.0,1.0,1,617,1.0,1.0,1.0 +bm25_plain,q032,Take a screenshot of the page,1,1,212,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,608,1.0,1.0,1.0 +bm25_plain,q033,Create a GitLab merge request,3,1,297,1.0,1.0,1.0,1,475,1.0,1.0,1.0,1,975,1.0,1.0,1.0 +bm25_plain,q034,Search Notion pages,1,1,180,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,671,1.0,1.0,1.0 +bm25_plain,q035,Create a Notion page,1,1,230,1.0,1.0,1.0,1,371,1.0,1.0,1.0,1,822,1.0,1.0,1.0 +bm25_plain,q036,Query a Notion database,1,1,261,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,668,1.0,1.0,1.0 +bm25_plain,q037,Create a Zendesk ticket,1,1,257,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,714,1.0,1.0,1.0 +bm25_plain,q038,Create a HubSpot contact,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,663,1.0,1.0,1.0 +bm25_plain,q039,Create a Salesforce lead,2,1,252,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,710,1.0,1.0,1.0 +bm25_plain,q040,Run a SOQL query,1,1,186,1.0,1.0,1.0,1,332,1.0,1.0,1.0,1,662,1.0,1.0,1.0 +bm25_plain,q041,List Shopify orders,1,1,208,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,682,1.0,1.0,1.0 +bm25_plain,q042,Create a PagerDuty incident,1,1,259,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,696,1.0,1.0,1.0 +bm25_plain,q043,Who is on call in PagerDuty,1,1,237,1.0,1.0,1.0,1,373,1.0,1.0,1.0,1,724,1.0,1.0,1.0 +bm25_plain,q044,Query Datadog metrics,1,1,300,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,774,1.0,1.0,1.0 +bm25_plain,q045,Search Datadog logs,1,1,242,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,855,1.0,1.0,1.0 +bm25_plain,q046,List S3 buckets,1,1,190,1.0,1.0,1.0,1,258,1.0,1.0,1.0,1,474,1.0,1.0,1.0 +bm25_plain,q047,Upload an object to S3,1,1,250,1.0,1.0,1.0,1,437,1.0,1.0,1.0,1,850,1.0,1.0,1.0 +bm25_plain,q048,Purge Cloudflare cache,1,1,183,1.0,1.0,1.0,1,340,1.0,1.0,1.0,1,746,1.0,1.0,1.0 +bm25_plain,q049,Create a Cloudflare DNS record,1,1,249,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,708,1.0,1.0,1.0 +bm25_plain,q050,List Vercel deployments,1,1,173,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,529,1.0,1.0,1.0 +bm25_plain,q051,Insert a row into Supabase,1,1,203,1.0,1.0,1.0,1,347,1.0,1.0,1.0,1,665,1.0,1.0,1.0 +bm25_plain,q052,Export images from Figma,1,1,218,1.0,1.0,1.0,1,328,1.0,1.0,1.0,1,637,1.0,1.0,1.0 +bm25_plain,q053,Send an SMS with Twilio,1,1,254,1.0,1.0,1.0,1,396,1.0,1.0,1.0,1,725,1.0,1.0,1.0 +bm25_plain,q054,Create an Okta user,1,1,183,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,622,1.0,1.0,1.0 +bm25_plain,q055,Instant query Prometheus,1,1,221,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,680,1.0,1.0,1.0 +bm25_plain,q056,Search Grafana dashboards,1,1,217,1.0,1.0,1.0,1,360,1.0,1.0,1.0,1,660,1.0,1.0,1.0 +bm25_plain,q057,Trigger a CircleCI pipeline,1,1,200,1.0,1.0,1.0,1,373,1.0,1.0,1.0,1,755,1.0,1.0,1.0 +bm25_plain,q058,Run terraform plan,1,1,172,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,658,1.0,1.0,1.0 +bm25_plain,q059,Create an Asana task,1,1,267,1.0,1.0,1.0,1,429,1.0,1.0,1.0,1,825,1.0,1.0,1.0 +bm25_plain,q060,Create a Trello card,1,1,240,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,679,1.0,1.0,1.0 +bm25_plain,q061,Create a ClickUp task,1,1,257,1.0,1.0,1.0,1,414,1.0,1.0,1.0,1,719,1.0,1.0,1.0 +bm25_plain,q062,Create a Confluence page,1,1,260,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,699,1.0,1.0,1.0 +bm25_plain,q063,Send a Discord message,1,1,224,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,682,1.0,1.0,1.0 +bm25_plain,q064,Search Elasticsearch,1,1,239,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,672,1.0,1.0,1.0 +bm25_plain,q065,Get a Redis key,1,1,182,1.0,1.0,1.0,1,281,1.0,1.0,1.0,1,589,1.0,1.0,1.0 +bm25_plain,q066,Find MongoDB documents,1,1,244,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,738,1.0,1.0,1.0 +bm25_plain,q067,List SQLite tables,1,1,142,1.0,1.0,1.0,1,240,1.0,1.0,1.0,1,496,1.0,1.0,1.0 +bm25_plain,q068,Describe a postgres table,1,1,200,1.0,1.0,1.0,1,349,1.0,1.0,1.0,1,597,1.0,1.0,1.0 +bm25_plain,q069,github_create_issue,1,1,378,1.0,1.0,1.0,1,568,1.0,1.0,1.0,1,1091,1.0,1.0,1.0 +bm25_plain,q070,createIssue,29,10,2752,0.0,0.0,0.0,6,2680,0.0,0.0,0.0,3,2626,0.0,0.0,1.0 +bm25_plain,q071,list-directory,2,1,186,1.0,1.0,1.0,1,296,1.0,1.0,1.0,1,607,1.0,1.0,1.0 +bm25_plain,q072,sequential thinking next thought,1,1,224,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,797,1.0,1.0,1.0 +bm25_plain,q073,What time is it in Tokyo,1,1,217,1.0,1.0,1.0,1,378,1.0,1.0,1.0,1,703,1.0,1.0,1.0 +bm25_plain,q074,Convert time between timezones,1,1,204,1.0,1.0,1.0,1,360,1.0,1.0,1.0,1,704,1.0,1.0,1.0 +bm25_plain,q075,Star a GitHub repo,1,1,226,1.0,1.0,1.0,1,394,1.0,1.0,1.0,1,779,1.0,1.0,1.0 +bm25_plain,q076,Fork a repository,1,1,235,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,792,1.0,1.0,1.0 +bm25_plain,q077,Get the latest GitHub release,1,1,239,1.0,1.0,1.0,1,361,1.0,1.0,1.0,1,712,1.0,1.0,1.0 +bm25_plain,q078,Create a gist,1,1,229,1.0,1.0,1.0,1,335,1.0,1.0,1.0,1,649,1.0,1.0,1.0 +bm25_plain,q079,Search GitHub code,1,1,283,1.0,1.0,1.0,1,385,1.0,1.0,1.0,1,757,1.0,1.0,1.0 +bm25_plain,q080,List GitHub notifications,1,1,204,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +bm25_plain,q081,Assign Copilot to an issue,1,1,267,1.0,1.0,1.0,1,402,1.0,1.0,1.0,1,833,1.0,1.0,1.0 +bm25_plain,q082,Get secret scanning alerts,1,1,295,1.0,1.0,1.0,1,470,1.0,1.0,1.0,1,765,1.0,1.0,1.0 +bm25_plain,q083,Create or update a file on GitHub,1,1,306,1.0,1.0,1.0,1,481,1.0,1.0,1.0,1,889,1.0,1.0,1.0 +bm25_plain,q084,List collaborators on a repo,1,1,278,1.0,1.0,1.0,1,433,1.0,1.0,1.0,1,932,1.0,1.0,1.0 +bm25_plain,q085,Reply in a Slack thread,1,1,240,1.0,1.0,1.0,1,350,1.0,1.0,1.0,1,632,1.0,1.0,1.0 +bm25_plain,q086,Add an emoji reaction on Slack,1,1,271,1.0,1.0,1.0,1,428,1.0,1.0,1.0,1,811,1.0,1.0,1.0 +bm25_plain,q087,Search Slack messages,1,1,232,1.0,1.0,1.0,1,373,1.0,1.0,1.0,1,664,1.0,1.0,1.0 +bm25_plain,q088,Create a Slack channel,1,1,196,1.0,1.0,1.0,1,342,1.0,1.0,1.0,1,751,1.0,1.0,1.0 +bm25_plain,q089,Transition a Jira issue,1,1,208,1.0,1.0,1.0,1,342,1.0,1.0,1.0,1,778,1.0,1.0,1.0 +bm25_plain,q090,Search Jira with JQL,1,1,241,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,619,1.0,1.0,1.0 +bm25_plain,q091,Add a worklog to Jira,1,1,236,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,874,1.0,1.0,1.0 +bm25_plain,q092,List Linear teams,1,1,137,1.0,1.0,1.0,1,295,1.0,1.0,1.0,1,646,1.0,1.0,1.0 +bm25_plain,q093,Comment on a Linear issue,1,1,244,1.0,1.0,1.0,1,360,1.0,1.0,1.0,1,777,1.0,1.0,1.0 +bm25_plain,q094,Move a local file,1,1,224,1.0,1.0,1.0,1,358,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +bm25_plain,q095,Recursively search files by pattern,1,1,200,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,721,1.0,1.0,1.0 +bm25_plain,q096,Edit a file in place with old and new string,1,1,247,1.0,1.0,1.0,1,358,1.0,1.0,1.0,1,673,1.0,1.0,1.0 +bm25_plain,q097,Read the whole knowledge graph,1,1,152,1.0,1.0,1.0,1,259,1.0,1.0,1.0,1,561,1.0,1.0,1.0 +bm25_plain,q098,Search memory nodes,1,1,202,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,635,1.0,1.0,1.0 +bm25_plain,q099,Show unstaged git diff,1,1,190,1.0,1.0,1.0,1,313,1.0,1.0,1.0,1,602,1.0,1.0,1.0 +bm25_plain,q100,Stage files with git add,1,1,197,1.0,1.0,1.0,1,417,1.0,1.0,1.0,1,872,1.0,1.0,1.0 +bm25_plain,q101,Create a Shopify product,1,1,240,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +bm25_plain,q102,Cancel a Stripe subscription,1,1,167,1.0,1.0,1.0,1,285,1.0,1.0,1.0,1,610,1.0,1.0,1.0 +bm25_plain,q103,Create a payment intent,1,1,234,1.0,1.0,1.0,1,340,1.0,1.0,1.0,1,654,1.0,1.0,1.0 +bm25_plain,q104,Acknowledge a PagerDuty incident,1,1,223,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,717,1.0,1.0,1.0 +bm25_plain,q105,Create a Datadog monitor,1,1,259,1.0,1.0,1.0,1,459,1.0,1.0,1.0,1,758,1.0,1.0,1.0 +bm25_plain,q106,Resolve a Sentry issue,2,1,251,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,721,1.0,1.0,1.0 +bm25_plain,q107,Scale a Kubernetes deployment,1,1,245,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +bm25_plain,q108,Apply a Kubernetes manifest,1,1,204,1.0,1.0,1.0,1,308,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +bm25_plain,q109,Stop a docker container,1,1,188,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,641,1.0,1.0,1.0 +bm25_plain,q110,Build a docker image,1,1,211,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,601,1.0,1.0,1.0 +bm25_plain,q111,Click an element in Playwright,1,1,216,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,612,1.0,1.0,1.0 +bm25_plain,q112,Type text into a browser input,1,1,235,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,648,1.0,1.0,1.0 +bm25_plain,q113,List GitLab projects,1,1,143,1.0,1.0,1.0,1,240,1.0,1.0,1.0,1,623,1.0,1.0,1.0 +bm25_plain,q114,Create a Bitbucket pull request,1,1,313,1.0,1.0,1.0,1,546,1.0,1.0,1.0,1,1139,1.0,1.0,1.0 +bm25_plain,q115,Create a Gmail draft,1,1,205,1.0,1.0,1.0,1,315,1.0,1.0,1.0,1,758,1.0,1.0,1.0 +bm25_plain,q116,Share a Google Drive file,1,1,254,1.0,1.0,1.0,1,417,1.0,1.0,1.0,1,725,1.0,1.0,1.0 +bm25_plain,q117,List Google Calendar events,1,1,219,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,753,1.0,1.0,1.0 +bm25_plain,q118,Delete a calendar event,1,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,735,1.0,1.0,1.0 +bm25_plain,q119,List Vercel projects,1,1,151,1.0,1.0,1.0,1,248,1.0,1.0,1.0,1,550,1.0,1.0,1.0 +bm25_plain,q120,Select rows from Supabase,1,1,237,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,660,1.0,1.0,1.0 +bm25_plain,q121,Post a Figma comment,1,1,204,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,646,1.0,1.0,1.0 +bm25_plain,q122,List Okta users,1,1,177,1.0,1.0,1.0,1,271,1.0,1.0,1.0,1,606,1.0,1.0,1.0 +bm25_plain,q123,Range query Prometheus,1,1,221,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,680,1.0,1.0,1.0 +bm25_plain,q124,Create a Grafana annotation,1,1,218,1.0,1.0,1.0,1,324,1.0,1.0,1.0,1,638,1.0,1.0,1.0 +bm25_plain,q125,Apply terraform,1,1,190,1.0,1.0,1.0,1,292,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +bm25_plain,q126,List Asana tasks,1,1,192,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,642,1.0,1.0,1.0 +bm25_plain,q127,Comment on a Trello card,1,1,260,1.0,1.0,1.0,1,422,1.0,1.0,1.0,1,703,1.0,1.0,1.0 +bm25_plain,q128,Update a ClickUp task,1,1,257,1.0,1.0,1.0,1,398,1.0,1.0,1.0,1,789,1.0,1.0,1.0 +bm25_plain,q129,Search Confluence,1,1,221,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,691,1.0,1.0,1.0 +bm25_plain,q130,Create a HubSpot deal,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,663,1.0,1.0,1.0 +bm25_plain,q131,List Zendesk tickets,1,1,189,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,605,1.0,1.0,1.0 +bm25_plain,q132,List Discord channels,1,1,184,1.0,1.0,1.0,1,300,1.0,1.0,1.0,1,592,1.0,1.0,1.0 +bm25_plain,q133,Set a Redis key,1,1,182,1.0,1.0,1.0,1,302,1.0,1.0,1.0,1,600,1.0,1.0,1.0 +bm25_plain,q134,Insert MongoDB documents,1,1,244,1.0,1.0,1.0,1,399,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +bm25_plain,q135,Run a MySQL query,1,1,181,1.0,1.0,1.0,1,286,1.0,1.0,1.0,1,608,1.0,1.0,1.0 +bm25_plain,q136,Index a document in Elasticsearch,1,1,242,1.0,1.0,1.0,1,346,1.0,1.0,1.0,1,605,1.0,1.0,1.0 +bm25_plain,q137,Get an S3 object,1,1,249,1.0,1.0,1.0,1,355,1.0,1.0,1.0,1,668,1.0,1.0,1.0 +bm25_plain,q138,List Cloudflare zones,1,1,182,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,579,1.0,1.0,1.0 +bm25_plain,q139,Create a Stripe invoice,1,1,204,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +bm25_plain,q140,List GitHub Actions workflows,1,1,293,1.0,1.0,1.0,1,468,1.0,1.0,1.0,1,845,1.0,1.0,1.0 +bm25_plain,q141,Get GitHub Actions job logs,1,1,256,1.0,1.0,1.0,1,421,1.0,1.0,1.0,1,822,1.0,1.0,1.0 +bm25_plain,q142,Create a GitHub repository,1,1,268,1.0,1.0,1.0,1,406,1.0,1.0,1.0,1,949,1.0,1.0,1.0 +bm25_plain,q143,Checkout a git branch,1,1,266,1.0,1.0,1.0,1,429,1.0,1.0,1.0,1,1001,1.0,1.0,1.0 +bm25_plain,q144,Show git log,1,1,196,1.0,1.0,1.0,1,299,1.0,1.0,1.0,1,600,1.0,1.0,1.0 +bm25_plain,q145,List postgres schemas,1,1,160,1.0,1.0,1.0,1,277,1.0,1.0,1.0,1,515,1.0,1.0,1.0 +bm25_plain,q146,Open nodes in the knowledge graph,1,1,163,1.0,1.0,1.0,1,260,1.0,1.0,1.0,1,531,1.0,1.0,1.0 +bm25_plain,q147,Get file metadata from the filesystem,2,1,220,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,766,1.0,1.0,1.0 +bm25_plain,q148,Write a new local file,1,1,234,1.0,1.0,1.0,1,384,1.0,1.0,1.0,1,791,1.0,1.0,1.0 +bm25_plain,q149,Search GitHub issues,1,1,233,1.0,1.0,1.0,1,385,1.0,1.0,1.0,1,700,1.0,1.0,1.0 +bm25_plain,q150,Get the authenticated GitHub user,2,1,179,1.0,1.0,1.0,1,288,1.0,1.0,1.0,1,569,1.0,1.0,1.0 +bm25_tfidf_hybrid,q001,Create a GitHub issue,2,1,349,1.0,1.0,1.0,1,521,1.0,1.0,1.0,1,972,1.0,1.0,1.0 +bm25_tfidf_hybrid,q002,Create a Jira ticket,1,1,281,1.0,1.0,1.0,1,466,1.0,1.0,1.0,1,951,1.0,1.0,1.0 +bm25_tfidf_hybrid,q003,Send a Slack message,2,1,250,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,753,1.0,1.0,1.0 +bm25_tfidf_hybrid,q004,List open pull requests,1,1,237,1.0,1.0,1.0,1,460,1.0,1.0,1.0,1,836,1.0,1.0,1.0 +bm25_tfidf_hybrid,q005,Open a pull request on GitHub,1,1,364,1.0,1.0,1.0,1,611,1.0,1.0,1.0,1,1144,1.0,1.0,1.0 +bm25_tfidf_hybrid,q006,Run a SQL query against postgres,1,1,180,1.0,1.0,1.0,1,295,1.0,1.0,1.0,1,553,1.0,1.0,1.0 +bm25_tfidf_hybrid,q007,Read a local file from disk,1,1,223,1.0,1.0,1.0,1,428,1.0,1.0,1.0,1,780,1.0,1.0,1.0 +bm25_tfidf_hybrid,q008,What files are in this folder,8,3,712,0.0,0.0,1.0,2,745,0.0,1.0,1.0,1,723,1.0,1.0,1.0 +bm25_tfidf_hybrid,q009,Remember this fact in the knowledge graph,2,1,154,1.0,1.0,1.0,1,262,1.0,1.0,1.0,1,544,1.0,1.0,1.0 +bm25_tfidf_hybrid,q010,Search the web with Brave,1,1,258,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,710,1.0,1.0,1.0 +bm25_tfidf_hybrid,q011,Fetch the contents of a URL,1,1,208,1.0,1.0,1.0,1,372,1.0,1.0,1.0,1,640,1.0,1.0,1.0 +bm25_tfidf_hybrid,q012,Show git status,2,1,183,1.0,1.0,1.0,1,299,1.0,1.0,1.0,1,645,1.0,1.0,1.0 +bm25_tfidf_hybrid,q013,Commit staged changes,1,1,182,1.0,1.0,1.0,1,327,1.0,1.0,1.0,1,663,1.0,1.0,1.0 +bm25_tfidf_hybrid,q014,Create a Linear issue,4,2,574,0.0,1.0,1.0,1,463,1.0,1.0,1.0,1,921,1.0,1.0,1.0 +bm25_tfidf_hybrid,q015,Make a task in my project tracker Linear,145,49,12448,0.0,0.0,0.0,29,11769,0.0,0.0,0.0,15,11810,0.0,0.0,0.0 +bm25_tfidf_hybrid,q016,Post a comment on a GitHub issue,3,1,307,1.0,1.0,1.0,1,444,1.0,1.0,1.0,1,819,1.0,1.0,1.0 +bm25_tfidf_hybrid,q017,Merge the pull request,1,1,283,1.0,1.0,1.0,1,495,1.0,1.0,1.0,1,1121,1.0,1.0,1.0 +bm25_tfidf_hybrid,q018,Trigger a GitHub Actions workflow,1,1,294,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,795,1.0,1.0,1.0 +bm25_tfidf_hybrid,q019,Get Dependabot alerts,1,1,269,1.0,1.0,1.0,1,420,1.0,1.0,1.0,1,684,1.0,1.0,1.0 +bm25_tfidf_hybrid,q020,List Sentry issues,1,1,236,1.0,1.0,1.0,1,380,1.0,1.0,1.0,1,796,1.0,1.0,1.0 +bm25_tfidf_hybrid,q021,Create a Stripe customer,1,1,186,1.0,1.0,1.0,1,300,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +bm25_tfidf_hybrid,q022,Refund a Stripe payment,1,1,234,1.0,1.0,1.0,1,370,1.0,1.0,1.0,1,647,1.0,1.0,1.0 +bm25_tfidf_hybrid,q023,Send an email with Gmail,1,1,224,1.0,1.0,1.0,1,344,1.0,1.0,1.0,1,711,1.0,1.0,1.0 +bm25_tfidf_hybrid,q024,Search my Gmail inbox,1,1,178,1.0,1.0,1.0,1,315,1.0,1.0,1.0,1,652,1.0,1.0,1.0 +bm25_tfidf_hybrid,q025,Upload a file to Google Drive,4,2,523,0.0,1.0,1.0,1,446,1.0,1.0,1.0,1,735,1.0,1.0,1.0 +bm25_tfidf_hybrid,q026,Create a calendar event,3,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,962,1.0,1.0,1.0 +bm25_tfidf_hybrid,q027,List Kubernetes pods,1,1,184,1.0,1.0,1.0,1,324,1.0,1.0,1.0,1,699,1.0,1.0,1.0 +bm25_tfidf_hybrid,q028,Get logs from a k8s pod,1,1,262,1.0,1.0,1.0,1,455,1.0,1.0,1.0,1,834,1.0,1.0,1.0 +bm25_tfidf_hybrid,q029,Run a docker container,1,1,223,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,597,1.0,1.0,1.0 +bm25_tfidf_hybrid,q030,Show docker container logs,1,1,188,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,645,1.0,1.0,1.0 +bm25_tfidf_hybrid,q031,Navigate the browser to a URL,1,1,180,1.0,1.0,1.0,1,294,1.0,1.0,1.0,1,596,1.0,1.0,1.0 +bm25_tfidf_hybrid,q032,Take a screenshot of the page,1,1,226,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,678,1.0,1.0,1.0 +bm25_tfidf_hybrid,q033,Create a GitLab merge request,3,1,297,1.0,1.0,1.0,1,475,1.0,1.0,1.0,1,939,1.0,1.0,1.0 +bm25_tfidf_hybrid,q034,Search Notion pages,1,1,180,1.0,1.0,1.0,1,286,1.0,1.0,1.0,1,639,1.0,1.0,1.0 +bm25_tfidf_hybrid,q035,Create a Notion page,3,1,230,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,707,1.0,1.0,1.0 +bm25_tfidf_hybrid,q036,Query a Notion database,1,1,279,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,709,1.0,1.0,1.0 +bm25_tfidf_hybrid,q037,Create a Zendesk ticket,3,1,257,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,875,1.0,1.0,1.0 +bm25_tfidf_hybrid,q038,Create a HubSpot contact,2,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,899,1.0,1.0,1.0 +bm25_tfidf_hybrid,q039,Create a Salesforce lead,2,1,252,1.0,1.0,1.0,1,416,1.0,1.0,1.0,1,796,1.0,1.0,1.0 +bm25_tfidf_hybrid,q040,Run a SOQL query,1,1,189,1.0,1.0,1.0,1,310,1.0,1.0,1.0,1,662,1.0,1.0,1.0 +bm25_tfidf_hybrid,q041,List Shopify orders,1,1,208,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,662,1.0,1.0,1.0 +bm25_tfidf_hybrid,q042,Create a PagerDuty incident,2,1,255,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,857,1.0,1.0,1.0 +bm25_tfidf_hybrid,q043,Who is on call in PagerDuty,1,1,227,1.0,1.0,1.0,1,419,1.0,1.0,1.0,1,772,1.0,1.0,1.0 +bm25_tfidf_hybrid,q044,Query Datadog metrics,1,1,300,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,804,1.0,1.0,1.0 +bm25_tfidf_hybrid,q045,Search Datadog logs,1,1,242,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,824,1.0,1.0,1.0 +bm25_tfidf_hybrid,q046,List S3 buckets,1,1,170,1.0,1.0,1.0,1,312,1.0,1.0,1.0,1,616,1.0,1.0,1.0 +bm25_tfidf_hybrid,q047,Upload an object to S3,1,1,250,1.0,1.0,1.0,1,437,1.0,1.0,1.0,1,862,1.0,1.0,1.0 +bm25_tfidf_hybrid,q048,Purge Cloudflare cache,1,1,231,1.0,1.0,1.0,1,340,1.0,1.0,1.0,1,746,1.0,1.0,1.0 +bm25_tfidf_hybrid,q049,Create a Cloudflare DNS record,1,1,249,1.0,1.0,1.0,1,360,1.0,1.0,1.0,1,757,1.0,1.0,1.0 +bm25_tfidf_hybrid,q050,List Vercel deployments,1,1,173,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,618,1.0,1.0,1.0 +bm25_tfidf_hybrid,q051,Insert a row into Supabase,1,1,203,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,665,1.0,1.0,1.0 +bm25_tfidf_hybrid,q052,Export images from Figma,1,1,196,1.0,1.0,1.0,1,328,1.0,1.0,1.0,1,777,1.0,1.0,1.0 +bm25_tfidf_hybrid,q053,Send an SMS with Twilio,1,1,254,1.0,1.0,1.0,1,409,1.0,1.0,1.0,1,725,1.0,1.0,1.0 +bm25_tfidf_hybrid,q054,Create an Okta user,1,1,183,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,683,1.0,1.0,1.0 +bm25_tfidf_hybrid,q055,Instant query Prometheus,1,1,221,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,680,1.0,1.0,1.0 +bm25_tfidf_hybrid,q056,Search Grafana dashboards,1,1,217,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,636,1.0,1.0,1.0 +bm25_tfidf_hybrid,q057,Trigger a CircleCI pipeline,1,1,216,1.0,1.0,1.0,1,325,1.0,1.0,1.0,1,755,1.0,1.0,1.0 +bm25_tfidf_hybrid,q058,Run terraform plan,1,1,172,1.0,1.0,1.0,1,280,1.0,1.0,1.0,1,684,1.0,1.0,1.0 +bm25_tfidf_hybrid,q059,Create an Asana task,2,1,267,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,825,1.0,1.0,1.0 +bm25_tfidf_hybrid,q060,Create a Trello card,1,1,240,1.0,1.0,1.0,1,413,1.0,1.0,1.0,1,903,1.0,1.0,1.0 +bm25_tfidf_hybrid,q061,Create a ClickUp task,1,1,257,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,770,1.0,1.0,1.0 +bm25_tfidf_hybrid,q062,Create a Confluence page,2,1,260,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,755,1.0,1.0,1.0 +bm25_tfidf_hybrid,q063,Send a Discord message,1,1,224,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,740,1.0,1.0,1.0 +bm25_tfidf_hybrid,q064,Search Elasticsearch,1,1,239,1.0,1.0,1.0,1,375,1.0,1.0,1.0,1,658,1.0,1.0,1.0 +bm25_tfidf_hybrid,q065,Get a Redis key,1,1,182,1.0,1.0,1.0,1,281,1.0,1.0,1.0,1,589,1.0,1.0,1.0 +bm25_tfidf_hybrid,q066,Find MongoDB documents,1,1,258,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,738,1.0,1.0,1.0 +bm25_tfidf_hybrid,q067,List SQLite tables,1,1,147,1.0,1.0,1.0,1,245,1.0,1.0,1.0,1,543,1.0,1.0,1.0 +bm25_tfidf_hybrid,q068,Describe a postgres table,1,1,215,1.0,1.0,1.0,1,349,1.0,1.0,1.0,1,597,1.0,1.0,1.0 +bm25_tfidf_hybrid,q069,github_create_issue,2,1,347,1.0,1.0,1.0,1,511,1.0,1.0,1.0,1,931,1.0,1.0,1.0 +bm25_tfidf_hybrid,q070,createIssue,2,1,332,1.0,1.0,1.0,1,472,1.0,1.0,1.0,1,880,1.0,1.0,1.0 +bm25_tfidf_hybrid,q071,list-directory,2,1,182,1.0,1.0,1.0,1,348,1.0,1.0,1.0,1,654,1.0,1.0,1.0 +bm25_tfidf_hybrid,q072,sequential thinking next thought,1,1,224,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,797,1.0,1.0,1.0 +bm25_tfidf_hybrid,q073,What time is it in Tokyo,1,1,222,1.0,1.0,1.0,1,378,1.0,1.0,1.0,1,699,1.0,1.0,1.0 +bm25_tfidf_hybrid,q074,Convert time between timezones,1,1,204,1.0,1.0,1.0,1,360,1.0,1.0,1.0,1,704,1.0,1.0,1.0 +bm25_tfidf_hybrid,q075,Star a GitHub repo,1,1,238,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,733,1.0,1.0,1.0 +bm25_tfidf_hybrid,q076,Fork a repository,1,1,248,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,825,1.0,1.0,1.0 +bm25_tfidf_hybrid,q077,Get the latest GitHub release,1,1,201,1.0,1.0,1.0,1,361,1.0,1.0,1.0,1,753,1.0,1.0,1.0 +bm25_tfidf_hybrid,q078,Create a gist,3,1,229,1.0,1.0,1.0,1,421,1.0,1.0,1.0,1,911,1.0,1.0,1.0 +bm25_tfidf_hybrid,q079,Search GitHub code,1,1,283,1.0,1.0,1.0,1,419,1.0,1.0,1.0,1,725,1.0,1.0,1.0 +bm25_tfidf_hybrid,q080,List GitHub notifications,1,1,193,1.0,1.0,1.0,1,328,1.0,1.0,1.0,1,687,1.0,1.0,1.0 +bm25_tfidf_hybrid,q081,Assign Copilot to an issue,1,1,267,1.0,1.0,1.0,1,434,1.0,1.0,1.0,1,875,1.0,1.0,1.0 +bm25_tfidf_hybrid,q082,Get secret scanning alerts,1,1,295,1.0,1.0,1.0,1,470,1.0,1.0,1.0,1,733,1.0,1.0,1.0 +bm25_tfidf_hybrid,q083,Create or update a file on GitHub,1,1,341,1.0,1.0,1.0,1,545,1.0,1.0,1.0,1,1020,1.0,1.0,1.0 +bm25_tfidf_hybrid,q084,List collaborators on a repo,1,1,307,1.0,1.0,1.0,1,445,1.0,1.0,1.0,1,962,1.0,1.0,1.0 +bm25_tfidf_hybrid,q085,Reply in a Slack thread,1,1,240,1.0,1.0,1.0,1,346,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +bm25_tfidf_hybrid,q086,Add an emoji reaction on Slack,1,1,268,1.0,1.0,1.0,1,467,1.0,1.0,1.0,1,812,1.0,1.0,1.0 +bm25_tfidf_hybrid,q087,Search Slack messages,1,1,211,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,632,1.0,1.0,1.0 +bm25_tfidf_hybrid,q088,Create a Slack channel,1,1,196,1.0,1.0,1.0,1,339,1.0,1.0,1.0,1,718,1.0,1.0,1.0 +bm25_tfidf_hybrid,q089,Transition a Jira issue,1,1,207,1.0,1.0,1.0,1,342,1.0,1.0,1.0,1,781,1.0,1.0,1.0 +bm25_tfidf_hybrid,q090,Search Jira with JQL,1,1,202,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,605,1.0,1.0,1.0 +bm25_tfidf_hybrid,q091,Add a worklog to Jira,1,1,285,1.0,1.0,1.0,1,406,1.0,1.0,1.0,1,839,1.0,1.0,1.0 +bm25_tfidf_hybrid,q092,List Linear teams,1,1,137,1.0,1.0,1.0,1,283,1.0,1.0,1.0,1,639,1.0,1.0,1.0 +bm25_tfidf_hybrid,q093,Comment on a Linear issue,1,1,244,1.0,1.0,1.0,1,360,1.0,1.0,1.0,1,798,1.0,1.0,1.0 +bm25_tfidf_hybrid,q094,Move a local file,1,1,224,1.0,1.0,1.0,1,358,1.0,1.0,1.0,1,684,1.0,1.0,1.0 +bm25_tfidf_hybrid,q095,Recursively search files by pattern,1,1,192,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,716,1.0,1.0,1.0 +bm25_tfidf_hybrid,q096,Edit a file in place with old and new string,1,1,247,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,735,1.0,1.0,1.0 +bm25_tfidf_hybrid,q097,Read the whole knowledge graph,1,1,152,1.0,1.0,1.0,1,260,1.0,1.0,1.0,1,561,1.0,1.0,1.0 +bm25_tfidf_hybrid,q098,Search memory nodes,1,1,202,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,621,1.0,1.0,1.0 +bm25_tfidf_hybrid,q099,Show unstaged git diff,1,1,190,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,621,1.0,1.0,1.0 +bm25_tfidf_hybrid,q100,Stage files with git add,1,1,220,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,774,1.0,1.0,1.0 +bm25_tfidf_hybrid,q101,Create a Shopify product,3,1,240,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +bm25_tfidf_hybrid,q102,Cancel a Stripe subscription,1,1,206,1.0,1.0,1.0,1,321,1.0,1.0,1.0,1,610,1.0,1.0,1.0 +bm25_tfidf_hybrid,q103,Create a payment intent,2,1,234,1.0,1.0,1.0,1,426,1.0,1.0,1.0,1,916,1.0,1.0,1.0 +bm25_tfidf_hybrid,q104,Acknowledge a PagerDuty incident,1,1,261,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,717,1.0,1.0,1.0 +bm25_tfidf_hybrid,q105,Create a Datadog monitor,2,1,259,1.0,1.0,1.0,1,459,1.0,1.0,1.0,1,948,1.0,1.0,1.0 +bm25_tfidf_hybrid,q106,Resolve a Sentry issue,2,1,251,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,745,1.0,1.0,1.0 +bm25_tfidf_hybrid,q107,Scale a Kubernetes deployment,1,1,230,1.0,1.0,1.0,1,374,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +bm25_tfidf_hybrid,q108,Apply a Kubernetes manifest,1,1,200,1.0,1.0,1.0,1,344,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +bm25_tfidf_hybrid,q109,Stop a docker container,1,1,188,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,641,1.0,1.0,1.0 +bm25_tfidf_hybrid,q110,Build a docker image,1,1,211,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,601,1.0,1.0,1.0 +bm25_tfidf_hybrid,q111,Click an element in Playwright,1,1,216,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,654,1.0,1.0,1.0 +bm25_tfidf_hybrid,q112,Type text into a browser input,1,1,235,1.0,1.0,1.0,1,337,1.0,1.0,1.0,1,647,1.0,1.0,1.0 +bm25_tfidf_hybrid,q113,List GitLab projects,1,1,171,1.0,1.0,1.0,1,255,1.0,1.0,1.0,1,609,1.0,1.0,1.0 +bm25_tfidf_hybrid,q114,Create a Bitbucket pull request,1,1,313,1.0,1.0,1.0,1,508,1.0,1.0,1.0,1,1091,1.0,1.0,1.0 +bm25_tfidf_hybrid,q115,Create a Gmail draft,1,1,188,1.0,1.0,1.0,1,315,1.0,1.0,1.0,1,844,1.0,1.0,1.0 +bm25_tfidf_hybrid,q116,Share a Google Drive file,1,1,254,1.0,1.0,1.0,1,417,1.0,1.0,1.0,1,684,1.0,1.0,1.0 +bm25_tfidf_hybrid,q117,List Google Calendar events,1,1,269,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,753,1.0,1.0,1.0 +bm25_tfidf_hybrid,q118,Delete a calendar event,1,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,796,1.0,1.0,1.0 +bm25_tfidf_hybrid,q119,List Vercel projects,1,1,175,1.0,1.0,1.0,1,269,1.0,1.0,1.0,1,550,1.0,1.0,1.0 +bm25_tfidf_hybrid,q120,Select rows from Supabase,1,1,237,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,815,1.0,1.0,1.0 +bm25_tfidf_hybrid,q121,Post a Figma comment,1,1,218,1.0,1.0,1.0,1,350,1.0,1.0,1.0,1,658,1.0,1.0,1.0 +bm25_tfidf_hybrid,q122,List Okta users,1,1,177,1.0,1.0,1.0,1,306,1.0,1.0,1.0,1,606,1.0,1.0,1.0 +bm25_tfidf_hybrid,q123,Range query Prometheus,1,1,221,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,680,1.0,1.0,1.0 +bm25_tfidf_hybrid,q124,Create a Grafana annotation,1,1,218,1.0,1.0,1.0,1,410,1.0,1.0,1.0,1,900,1.0,1.0,1.0 +bm25_tfidf_hybrid,q125,Apply terraform,1,1,190,1.0,1.0,1.0,1,292,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +bm25_tfidf_hybrid,q126,List Asana tasks,1,1,211,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,708,1.0,1.0,1.0 +bm25_tfidf_hybrid,q127,Comment on a Trello card,1,1,241,1.0,1.0,1.0,1,422,1.0,1.0,1.0,1,695,1.0,1.0,1.0 +bm25_tfidf_hybrid,q128,Update a ClickUp task,1,1,257,1.0,1.0,1.0,1,399,1.0,1.0,1.0,1,740,1.0,1.0,1.0 +bm25_tfidf_hybrid,q129,Search Confluence,1,1,221,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,677,1.0,1.0,1.0 +bm25_tfidf_hybrid,q130,Create a HubSpot deal,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,899,1.0,1.0,1.0 +bm25_tfidf_hybrid,q131,List Zendesk tickets,1,1,189,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,694,1.0,1.0,1.0 +bm25_tfidf_hybrid,q132,List Discord channels,1,1,184,1.0,1.0,1.0,1,300,1.0,1.0,1.0,1,639,1.0,1.0,1.0 +bm25_tfidf_hybrid,q133,Set a Redis key,1,1,182,1.0,1.0,1.0,1,302,1.0,1.0,1.0,1,595,1.0,1.0,1.0 +bm25_tfidf_hybrid,q134,Insert MongoDB documents,1,1,256,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +bm25_tfidf_hybrid,q135,Run a MySQL query,1,1,179,1.0,1.0,1.0,1,281,1.0,1.0,1.0,1,619,1.0,1.0,1.0 +bm25_tfidf_hybrid,q136,Index a document in Elasticsearch,1,1,242,1.0,1.0,1.0,1,344,1.0,1.0,1.0,1,607,1.0,1.0,1.0 +bm25_tfidf_hybrid,q137,Get an S3 object,1,1,249,1.0,1.0,1.0,1,355,1.0,1.0,1.0,1,742,1.0,1.0,1.0 +bm25_tfidf_hybrid,q138,List Cloudflare zones,1,1,215,1.0,1.0,1.0,1,319,1.0,1.0,1.0,1,663,1.0,1.0,1.0 +bm25_tfidf_hybrid,q139,Create a Stripe invoice,1,1,204,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +bm25_tfidf_hybrid,q140,List GitHub Actions workflows,1,1,268,1.0,1.0,1.0,1,457,1.0,1.0,1.0,1,782,1.0,1.0,1.0 +bm25_tfidf_hybrid,q141,Get GitHub Actions job logs,1,1,237,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,822,1.0,1.0,1.0 +bm25_tfidf_hybrid,q142,Create a GitHub repository,1,1,235,1.0,1.0,1.0,1,406,1.0,1.0,1.0,1,837,1.0,1.0,1.0 +bm25_tfidf_hybrid,q143,Checkout a git branch,1,1,266,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,1001,1.0,1.0,1.0 +bm25_tfidf_hybrid,q144,Show git log,1,1,194,1.0,1.0,1.0,1,299,1.0,1.0,1.0,1,619,1.0,1.0,1.0 +bm25_tfidf_hybrid,q145,List postgres schemas,1,1,158,1.0,1.0,1.0,1,277,1.0,1.0,1.0,1,604,1.0,1.0,1.0 +bm25_tfidf_hybrid,q146,Open nodes in the knowledge graph,1,1,163,1.0,1.0,1.0,1,260,1.0,1.0,1.0,1,531,1.0,1.0,1.0 +bm25_tfidf_hybrid,q147,Get file metadata from the filesystem,1,1,220,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,769,1.0,1.0,1.0 +bm25_tfidf_hybrid,q148,Write a new local file,1,1,234,1.0,1.0,1.0,1,428,1.0,1.0,1.0,1,752,1.0,1.0,1.0 +bm25_tfidf_hybrid,q149,Search GitHub issues,1,1,263,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,710,1.0,1.0,1.0 +bm25_tfidf_hybrid,q150,Get the authenticated GitHub user,3,1,179,1.0,1.0,1.0,1,284,1.0,1.0,1.0,1,606,1.0,1.0,1.0 +bm25_weighted,q001,Create a GitHub issue,1,1,388,1.0,1.0,1.0,1,598,1.0,1.0,1.0,1,955,1.0,1.0,1.0 +bm25_weighted,q002,Create a Jira ticket,1,1,312,1.0,1.0,1.0,1,412,1.0,1.0,1.0,1,763,1.0,1.0,1.0 +bm25_weighted,q003,Send a Slack message,1,1,232,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,707,1.0,1.0,1.0 +bm25_weighted,q004,List open pull requests,1,1,234,1.0,1.0,1.0,1,429,1.0,1.0,1.0,1,1035,1.0,1.0,1.0 +bm25_weighted,q005,Open a pull request on GitHub,1,1,368,1.0,1.0,1.0,1,653,1.0,1.0,1.0,1,1052,1.0,1.0,1.0 +bm25_weighted,q006,Run a SQL query against postgres,1,1,167,1.0,1.0,1.0,1,280,1.0,1.0,1.0,1,590,1.0,1.0,1.0 +bm25_weighted,q007,Read a local file from disk,1,1,223,1.0,1.0,1.0,1,385,1.0,1.0,1.0,1,782,1.0,1.0,1.0 +bm25_weighted,q008,What files are in this folder,6,2,473,0.0,1.0,1.0,2,730,0.0,1.0,1.0,1,708,1.0,1.0,1.0 +bm25_weighted,q009,Remember this fact in the knowledge graph,2,1,154,1.0,1.0,1.0,1,261,1.0,1.0,1.0,1,524,1.0,1.0,1.0 +bm25_weighted,q010,Search the web with Brave,1,1,245,1.0,1.0,1.0,1,412,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +bm25_weighted,q011,Fetch the contents of a URL,1,1,258,1.0,1.0,1.0,1,372,1.0,1.0,1.0,1,707,1.0,1.0,1.0 +bm25_weighted,q012,Show git status,1,1,196,1.0,1.0,1.0,1,304,1.0,1.0,1.0,1,600,1.0,1.0,1.0 +bm25_weighted,q013,Commit staged changes,1,1,218,1.0,1.0,1.0,1,328,1.0,1.0,1.0,1,663,1.0,1.0,1.0 +bm25_weighted,q014,Create a Linear issue,1,1,340,1.0,1.0,1.0,1,463,1.0,1.0,1.0,1,765,1.0,1.0,1.0 +bm25_weighted,q015,Make a task in my project tracker Linear,2,1,352,1.0,1.0,1.0,1,432,1.0,1.0,1.0,1,769,1.0,1.0,1.0 +bm25_weighted,q016,Post a comment on a GitHub issue,2,1,333,1.0,1.0,1.0,1,601,1.0,1.0,1.0,1,938,1.0,1.0,1.0 +bm25_weighted,q017,Merge the pull request,1,1,283,1.0,1.0,1.0,1,523,1.0,1.0,1.0,1,1121,1.0,1.0,1.0 +bm25_weighted,q018,Trigger a GitHub Actions workflow,1,1,294,1.0,1.0,1.0,1,450,1.0,1.0,1.0,1,737,1.0,1.0,1.0 +bm25_weighted,q019,Get Dependabot alerts,1,1,287,1.0,1.0,1.0,1,445,1.0,1.0,1.0,1,798,1.0,1.0,1.0 +bm25_weighted,q020,List Sentry issues,1,1,238,1.0,1.0,1.0,1,396,1.0,1.0,1.0,1,796,1.0,1.0,1.0 +bm25_weighted,q021,Create a Stripe customer,2,1,186,1.0,1.0,1.0,1,300,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +bm25_weighted,q022,Refund a Stripe payment,1,1,234,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,662,1.0,1.0,1.0 +bm25_weighted,q023,Send an email with Gmail,1,1,224,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,703,1.0,1.0,1.0 +bm25_weighted,q024,Search my Gmail inbox,1,1,205,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,714,1.0,1.0,1.0 +bm25_weighted,q025,Upload a file to Google Drive,4,2,496,0.0,1.0,1.0,1,418,1.0,1.0,1.0,1,791,1.0,1.0,1.0 +bm25_weighted,q026,Create a calendar event,2,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,895,1.0,1.0,1.0 +bm25_weighted,q027,List Kubernetes pods,1,1,184,1.0,1.0,1.0,1,314,1.0,1.0,1.0,1,718,1.0,1.0,1.0 +bm25_weighted,q028,Get logs from a k8s pod,1,1,262,1.0,1.0,1.0,1,455,1.0,1.0,1.0,1,834,1.0,1.0,1.0 +bm25_weighted,q029,Run a docker container,1,1,219,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,644,1.0,1.0,1.0 +bm25_weighted,q030,Show docker container logs,1,1,188,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,686,1.0,1.0,1.0 +bm25_weighted,q031,Navigate the browser to a URL,1,1,180,1.0,1.0,1.0,1,265,1.0,1.0,1.0,1,536,1.0,1.0,1.0 +bm25_weighted,q032,Take a screenshot of the page,1,1,226,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,711,1.0,1.0,1.0 +bm25_weighted,q033,Create a GitLab merge request,3,1,297,1.0,1.0,1.0,1,475,1.0,1.0,1.0,1,792,1.0,1.0,1.0 +bm25_weighted,q034,Search Notion pages,1,1,195,1.0,1.0,1.0,1,366,1.0,1.0,1.0,1,686,1.0,1.0,1.0 +bm25_weighted,q035,Create a Notion page,3,1,230,1.0,1.0,1.0,1,371,1.0,1.0,1.0,1,687,1.0,1.0,1.0 +bm25_weighted,q036,Query a Notion database,1,1,279,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,687,1.0,1.0,1.0 +bm25_weighted,q037,Create a Zendesk ticket,1,1,257,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,856,1.0,1.0,1.0 +bm25_weighted,q038,Create a HubSpot contact,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,904,1.0,1.0,1.0 +bm25_weighted,q039,Create a Salesforce lead,2,1,252,1.0,1.0,1.0,1,402,1.0,1.0,1.0,1,787,1.0,1.0,1.0 +bm25_weighted,q040,Run a SOQL query,1,1,186,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,724,1.0,1.0,1.0 +bm25_weighted,q041,List Shopify orders,1,1,208,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,682,1.0,1.0,1.0 +bm25_weighted,q042,Create a PagerDuty incident,1,1,259,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,838,1.0,1.0,1.0 +bm25_weighted,q043,Who is on call in PagerDuty,1,1,227,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,721,1.0,1.0,1.0 +bm25_weighted,q044,Query Datadog metrics,1,1,300,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,728,1.0,1.0,1.0 +bm25_weighted,q045,Search Datadog logs,1,1,242,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,877,1.0,1.0,1.0 +bm25_weighted,q046,List S3 buckets,1,1,186,1.0,1.0,1.0,1,318,1.0,1.0,1.0,1,667,1.0,1.0,1.0 +bm25_weighted,q047,Upload an object to S3,1,1,250,1.0,1.0,1.0,1,437,1.0,1.0,1.0,1,950,1.0,1.0,1.0 +bm25_weighted,q048,Purge Cloudflare cache,1,1,231,1.0,1.0,1.0,1,340,1.0,1.0,1.0,1,746,1.0,1.0,1.0 +bm25_weighted,q049,Create a Cloudflare DNS record,1,1,217,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,738,1.0,1.0,1.0 +bm25_weighted,q050,List Vercel deployments,1,1,177,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,579,1.0,1.0,1.0 +bm25_weighted,q051,Insert a row into Supabase,1,1,238,1.0,1.0,1.0,1,355,1.0,1.0,1.0,1,665,1.0,1.0,1.0 +bm25_weighted,q052,Export images from Figma,1,1,239,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,658,1.0,1.0,1.0 +bm25_weighted,q053,Send an SMS with Twilio,1,1,249,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,740,1.0,1.0,1.0 +bm25_weighted,q054,Create an Okta user,3,1,183,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,656,1.0,1.0,1.0 +bm25_weighted,q055,Instant query Prometheus,1,1,221,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,680,1.0,1.0,1.0 +bm25_weighted,q056,Search Grafana dashboards,1,1,217,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,690,1.0,1.0,1.0 +bm25_weighted,q057,Trigger a CircleCI pipeline,1,1,200,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,755,1.0,1.0,1.0 +bm25_weighted,q058,Run terraform plan,1,1,172,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,717,1.0,1.0,1.0 +bm25_weighted,q059,Create an Asana task,2,1,267,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,825,1.0,1.0,1.0 +bm25_weighted,q060,Create a Trello card,1,1,240,1.0,1.0,1.0,1,394,1.0,1.0,1.0,1,884,1.0,1.0,1.0 +bm25_weighted,q061,Create a ClickUp task,2,1,257,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,751,1.0,1.0,1.0 +bm25_weighted,q062,Create a Confluence page,1,1,260,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,703,1.0,1.0,1.0 +bm25_weighted,q063,Send a Discord message,1,1,224,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,682,1.0,1.0,1.0 +bm25_weighted,q064,Search Elasticsearch,1,1,239,1.0,1.0,1.0,1,404,1.0,1.0,1.0,1,699,1.0,1.0,1.0 +bm25_weighted,q065,Get a Redis key,1,1,182,1.0,1.0,1.0,1,280,1.0,1.0,1.0,1,589,1.0,1.0,1.0 +bm25_weighted,q066,Find MongoDB documents,1,1,258,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,738,1.0,1.0,1.0 +bm25_weighted,q067,List SQLite tables,1,1,151,1.0,1.0,1.0,1,240,1.0,1.0,1.0,1,514,1.0,1.0,1.0 +bm25_weighted,q068,Describe a postgres table,1,1,186,1.0,1.0,1.0,1,272,1.0,1.0,1.0,1,597,1.0,1.0,1.0 +bm25_weighted,q069,github_create_issue,1,1,386,1.0,1.0,1.0,1,596,1.0,1.0,1.0,1,1019,1.0,1.0,1.0 +bm25_weighted,q070,createIssue,1,1,340,1.0,1.0,1.0,1,546,1.0,1.0,1.0,1,961,1.0,1.0,1.0 +bm25_weighted,q071,list-directory,2,1,182,1.0,1.0,1.0,1,296,1.0,1.0,1.0,1,607,1.0,1.0,1.0 +bm25_weighted,q072,sequential thinking next thought,1,1,224,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,797,1.0,1.0,1.0 +bm25_weighted,q073,What time is it in Tokyo,1,1,217,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,783,1.0,1.0,1.0 +bm25_weighted,q074,Convert time between timezones,1,1,204,1.0,1.0,1.0,1,335,1.0,1.0,1.0,1,704,1.0,1.0,1.0 +bm25_weighted,q075,Star a GitHub repo,1,1,195,1.0,1.0,1.0,1,309,1.0,1.0,1.0,1,604,1.0,1.0,1.0 +bm25_weighted,q076,Fork a repository,1,1,235,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,792,1.0,1.0,1.0 +bm25_weighted,q077,Get the latest GitHub release,1,1,249,1.0,1.0,1.0,1,366,1.0,1.0,1.0,1,647,1.0,1.0,1.0 +bm25_weighted,q078,Create a gist,1,1,229,1.0,1.0,1.0,1,429,1.0,1.0,1.0,1,912,1.0,1.0,1.0 +bm25_weighted,q079,Search GitHub code,1,1,283,1.0,1.0,1.0,1,406,1.0,1.0,1.0,1,697,1.0,1.0,1.0 +bm25_weighted,q080,List GitHub notifications,1,1,169,1.0,1.0,1.0,1,292,1.0,1.0,1.0,1,585,1.0,1.0,1.0 +bm25_weighted,q081,Assign Copilot to an issue,1,1,267,1.0,1.0,1.0,1,434,1.0,1.0,1.0,1,928,1.0,1.0,1.0 +bm25_weighted,q082,Get secret scanning alerts,1,1,295,1.0,1.0,1.0,1,470,1.0,1.0,1.0,1,830,1.0,1.0,1.0 +bm25_weighted,q083,Create or update a file on GitHub,1,1,347,1.0,1.0,1.0,1,519,1.0,1.0,1.0,1,869,1.0,1.0,1.0 +bm25_weighted,q084,List collaborators on a repo,1,1,278,1.0,1.0,1.0,1,433,1.0,1.0,1.0,1,816,1.0,1.0,1.0 +bm25_weighted,q085,Reply in a Slack thread,1,1,254,1.0,1.0,1.0,1,358,1.0,1.0,1.0,1,717,1.0,1.0,1.0 +bm25_weighted,q086,Add an emoji reaction on Slack,1,1,216,1.0,1.0,1.0,1,357,1.0,1.0,1.0,1,709,1.0,1.0,1.0 +bm25_weighted,q087,Search Slack messages,1,1,213,1.0,1.0,1.0,1,337,1.0,1.0,1.0,1,689,1.0,1.0,1.0 +bm25_weighted,q088,Create a Slack channel,3,1,196,1.0,1.0,1.0,1,342,1.0,1.0,1.0,1,719,1.0,1.0,1.0 +bm25_weighted,q089,Transition a Jira issue,1,1,208,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,763,1.0,1.0,1.0 +bm25_weighted,q090,Search Jira with JQL,1,1,194,1.0,1.0,1.0,1,370,1.0,1.0,1.0,1,764,1.0,1.0,1.0 +bm25_weighted,q091,Add a worklog to Jira,1,1,249,1.0,1.0,1.0,1,350,1.0,1.0,1.0,1,765,1.0,1.0,1.0 +bm25_weighted,q092,List Linear teams,1,1,184,1.0,1.0,1.0,1,380,1.0,1.0,1.0,1,764,1.0,1.0,1.0 +bm25_weighted,q093,Comment on a Linear issue,1,1,334,1.0,1.0,1.0,1,464,1.0,1.0,1.0,1,760,1.0,1.0,1.0 +bm25_weighted,q094,Move a local file,1,1,214,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,699,1.0,1.0,1.0 +bm25_weighted,q095,Recursively search files by pattern,1,1,280,1.0,1.0,1.0,1,433,1.0,1.0,1.0,1,705,1.0,1.0,1.0 +bm25_weighted,q096,Edit a file in place with old and new string,1,1,227,1.0,1.0,1.0,1,394,1.0,1.0,1.0,1,682,1.0,1.0,1.0 +bm25_weighted,q097,Read the whole knowledge graph,1,1,152,1.0,1.0,1.0,1,259,1.0,1.0,1.0,1,529,1.0,1.0,1.0 +bm25_weighted,q098,Search memory nodes,1,1,202,1.0,1.0,1.0,1,301,1.0,1.0,1.0,1,546,1.0,1.0,1.0 +bm25_weighted,q099,Show unstaged git diff,1,1,190,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,602,1.0,1.0,1.0 +bm25_weighted,q100,Stage files with git add,3,1,213,1.0,1.0,1.0,1,336,1.0,1.0,1.0,1,677,1.0,1.0,1.0 +bm25_weighted,q101,Create a Shopify product,1,1,240,1.0,1.0,1.0,1,380,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +bm25_weighted,q102,Cancel a Stripe subscription,1,1,188,1.0,1.0,1.0,1,301,1.0,1.0,1.0,1,632,1.0,1.0,1.0 +bm25_weighted,q103,Create a payment intent,1,1,234,1.0,1.0,1.0,1,417,1.0,1.0,1.0,1,897,1.0,1.0,1.0 +bm25_weighted,q104,Acknowledge a PagerDuty incident,1,1,261,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,717,1.0,1.0,1.0 +bm25_weighted,q105,Create a Datadog monitor,2,1,259,1.0,1.0,1.0,1,459,1.0,1.0,1.0,1,927,1.0,1.0,1.0 +bm25_weighted,q106,Resolve a Sentry issue,2,1,251,1.0,1.0,1.0,1,397,1.0,1.0,1.0,1,817,1.0,1.0,1.0 +bm25_weighted,q107,Scale a Kubernetes deployment,1,1,263,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +bm25_weighted,q108,Apply a Kubernetes manifest,1,1,253,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +bm25_weighted,q109,Stop a docker container,1,1,188,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,641,1.0,1.0,1.0 +bm25_weighted,q110,Build a docker image,1,1,224,1.0,1.0,1.0,1,317,1.0,1.0,1.0,1,601,1.0,1.0,1.0 +bm25_weighted,q111,Click an element in Playwright,1,1,216,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,566,1.0,1.0,1.0 +bm25_weighted,q112,Type text into a browser input,1,1,217,1.0,1.0,1.0,1,302,1.0,1.0,1.0,1,554,1.0,1.0,1.0 +bm25_weighted,q113,List GitLab projects,1,1,179,1.0,1.0,1.0,1,304,1.0,1.0,1.0,1,599,1.0,1.0,1.0 +bm25_weighted,q114,Create a Bitbucket pull request,1,1,313,1.0,1.0,1.0,1,471,1.0,1.0,1.0,1,1091,1.0,1.0,1.0 +bm25_weighted,q115,Create a Gmail draft,1,1,205,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,835,1.0,1.0,1.0 +bm25_weighted,q116,Share a Google Drive file,1,1,254,1.0,1.0,1.0,1,417,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +bm25_weighted,q117,List Google Calendar events,1,1,269,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,784,1.0,1.0,1.0 +bm25_weighted,q118,Delete a calendar event,1,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,751,1.0,1.0,1.0 +bm25_weighted,q119,List Vercel projects,1,1,177,1.0,1.0,1.0,1,301,1.0,1.0,1.0,1,550,1.0,1.0,1.0 +bm25_weighted,q120,Select rows from Supabase,1,1,237,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,709,1.0,1.0,1.0 +bm25_weighted,q121,Post a Figma comment,1,1,204,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +bm25_weighted,q122,List Okta users,1,1,177,1.0,1.0,1.0,1,306,1.0,1.0,1.0,1,606,1.0,1.0,1.0 +bm25_weighted,q123,Range query Prometheus,1,1,221,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,680,1.0,1.0,1.0 +bm25_weighted,q124,Create a Grafana annotation,1,1,218,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,881,1.0,1.0,1.0 +bm25_weighted,q125,Apply terraform,1,1,171,1.0,1.0,1.0,1,292,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +bm25_weighted,q126,List Asana tasks,1,1,244,1.0,1.0,1.0,1,404,1.0,1.0,1.0,1,682,1.0,1.0,1.0 +bm25_weighted,q127,Comment on a Trello card,1,1,260,1.0,1.0,1.0,1,422,1.0,1.0,1.0,1,757,1.0,1.0,1.0 +bm25_weighted,q128,Update a ClickUp task,1,1,257,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,761,1.0,1.0,1.0 +bm25_weighted,q129,Search Confluence,1,1,221,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,731,1.0,1.0,1.0 +bm25_weighted,q130,Create a HubSpot deal,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,904,1.0,1.0,1.0 +bm25_weighted,q131,List Zendesk tickets,1,1,189,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,645,1.0,1.0,1.0 +bm25_weighted,q132,List Discord channels,1,1,166,1.0,1.0,1.0,1,321,1.0,1.0,1.0,1,625,1.0,1.0,1.0 +bm25_weighted,q133,Set a Redis key,1,1,182,1.0,1.0,1.0,1,280,1.0,1.0,1.0,1,600,1.0,1.0,1.0 +bm25_weighted,q134,Insert MongoDB documents,1,1,256,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +bm25_weighted,q135,Run a MySQL query,3,1,182,1.0,1.0,1.0,1,268,1.0,1.0,1.0,1,543,1.0,1.0,1.0 +bm25_weighted,q136,Index a document in Elasticsearch,1,1,242,1.0,1.0,1.0,1,396,1.0,1.0,1.0,1,753,1.0,1.0,1.0 +bm25_weighted,q137,Get an S3 object,1,1,249,1.0,1.0,1.0,1,355,1.0,1.0,1.0,1,706,1.0,1.0,1.0 +bm25_weighted,q138,List Cloudflare zones,1,1,182,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,629,1.0,1.0,1.0 +bm25_weighted,q139,Create a Stripe invoice,1,1,204,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,632,1.0,1.0,1.0 +bm25_weighted,q140,List GitHub Actions workflows,1,1,293,1.0,1.0,1.0,1,433,1.0,1.0,1.0,1,736,1.0,1.0,1.0 +bm25_weighted,q141,Get GitHub Actions job logs,1,1,269,1.0,1.0,1.0,1,450,1.0,1.0,1.0,1,747,1.0,1.0,1.0 +bm25_weighted,q142,Create a GitHub repository,1,1,250,1.0,1.0,1.0,1,388,1.0,1.0,1.0,1,750,1.0,1.0,1.0 +bm25_weighted,q143,Checkout a git branch,1,1,212,1.0,1.0,1.0,1,373,1.0,1.0,1.0,1,666,1.0,1.0,1.0 +bm25_weighted,q144,Show git log,1,1,206,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,600,1.0,1.0,1.0 +bm25_weighted,q145,List postgres schemas,1,1,158,1.0,1.0,1.0,1,277,1.0,1.0,1.0,1,555,1.0,1.0,1.0 +bm25_weighted,q146,Open nodes in the knowledge graph,1,1,163,1.0,1.0,1.0,1,260,1.0,1.0,1.0,1,549,1.0,1.0,1.0 +bm25_weighted,q147,Get file metadata from the filesystem,4,2,462,0.0,1.0,1.0,1,348,1.0,1.0,1.0,1,688,1.0,1.0,1.0 +bm25_weighted,q148,Write a new local file,2,1,217,1.0,1.0,1.0,1,366,1.0,1.0,1.0,1,714,1.0,1.0,1.0 +bm25_weighted,q149,Search GitHub issues,1,1,270,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,670,1.0,1.0,1.0 +bm25_weighted,q150,Get the authenticated GitHub user,11,4,771,0.0,0.0,0.0,3,1050,0.0,0.0,1.0,2,1402,0.0,1.0,1.0 +boolean_and,q001,Create a GitHub issue,1,1,343,1.0,1.0,1.0,1,448,1.0,1.0,1.0,1,870,1.0,1.0,1.0 +boolean_and,q002,Create a Jira ticket,1,1,227,1.0,1.0,1.0,1,385,1.0,1.0,1.0,1,800,1.0,1.0,1.0 +boolean_and,q003,Send a Slack message,121,41,10292,0.0,0.0,0.0,25,10122,0.0,0.0,0.0,13,10175,0.0,0.0,0.0 +boolean_and,q004,List open pull requests,37,13,3710,0.0,0.0,0.0,8,3752,0.0,0.0,0.0,4,3672,0.0,0.0,0.0 +boolean_and,q005,Open a pull request on GitHub,1,1,265,1.0,1.0,1.0,1,423,1.0,1.0,1.0,1,838,1.0,1.0,1.0 +boolean_and,q006,Run a SQL query against postgres,1,1,167,1.0,1.0,1.0,1,325,1.0,1.0,1.0,1,740,1.0,1.0,1.0 +boolean_and,q007,Read a local file from disk,76,26,7282,0.0,0.0,0.0,16,7224,0.0,0.0,0.0,8,7048,0.0,0.0,0.0 +boolean_and,q008,What files are in this folder,82,28,7711,0.0,0.0,0.0,17,7530,0.0,0.0,0.0,9,7616,0.0,0.0,0.0 +boolean_and,q009,Remember this fact in the knowledge graph,89,30,8108,0.0,0.0,0.0,18,7832,0.0,0.0,0.0,9,7625,0.0,0.0,0.0 +boolean_and,q010,Search the web with Brave,119,40,10063,0.0,0.0,0.0,24,9727,0.0,0.0,0.0,12,9475,0.0,0.0,0.0 +boolean_and,q011,Fetch the contents of a URL,103,35,8982,0.0,0.0,0.0,21,8674,0.0,0.0,0.0,11,8746,0.0,0.0,0.0 +boolean_and,q012,Show git status,1,1,163,1.0,1.0,1.0,1,321,1.0,1.0,1.0,1,736,1.0,1.0,1.0 +boolean_and,q013,Commit staged changes,1,1,175,1.0,1.0,1.0,1,333,1.0,1.0,1.0,1,748,1.0,1.0,1.0 +boolean_and,q014,Create a Linear issue,1,1,273,1.0,1.0,1.0,1,378,1.0,1.0,1.0,1,800,1.0,1.0,1.0 +boolean_and,q015,Make a task in my project tracker Linear,136,46,11712,0.0,0.0,0.0,28,11391,0.0,0.0,0.0,14,11055,0.0,0.0,0.0 +boolean_and,q016,Post a comment on a GitHub issue,34,12,3507,0.0,0.0,0.0,7,3237,0.0,0.0,0.0,4,3688,0.0,0.0,0.0 +boolean_and,q017,Merge the pull request,41,14,4072,0.0,0.0,0.0,9,4373,0.0,0.0,0.0,5,4766,0.0,0.0,0.0 +boolean_and,q018,Trigger a GitHub Actions workflow,1,1,233,1.0,1.0,1.0,1,391,1.0,1.0,1.0,1,806,1.0,1.0,1.0 +boolean_and,q019,Get Dependabot alerts,51,17,5097,0.0,0.0,0.0,11,5225,0.0,0.0,0.0,6,5482,0.0,0.0,0.0 +boolean_and,q020,List Sentry issues,1,1,207,1.0,1.0,1.0,1,365,1.0,1.0,1.0,1,780,1.0,1.0,1.0 +boolean_and,q021,Create a Stripe customer,1,1,223,1.0,1.0,1.0,1,315,1.0,1.0,1.0,1,751,1.0,1.0,1.0 +boolean_and,q022,Refund a Stripe payment,1,1,181,1.0,1.0,1.0,1,339,1.0,1.0,1.0,1,754,1.0,1.0,1.0 +boolean_and,q023,Send an email with Gmail,244,82,19588,0.0,0.0,0.0,49,18841,0.0,0.0,0.0,25,18655,0.0,0.0,0.0 +boolean_and,q024,Search my Gmail inbox,245,82,19506,0.0,0.0,0.0,49,18792,0.0,0.0,0.0,25,18630,0.0,0.0,0.0 +boolean_and,q025,Upload a file to Google Drive,253,85,20366,0.0,0.0,0.0,51,19618,0.0,0.0,0.0,26,19523,0.0,0.0,0.0 +boolean_and,q026,Create a calendar event,1,1,232,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,805,1.0,1.0,1.0 +boolean_and,q027,List Kubernetes pods,1,1,168,1.0,1.0,1.0,1,326,1.0,1.0,1.0,1,741,1.0,1.0,1.0 +boolean_and,q028,Get logs from a k8s pod,1,1,214,1.0,1.0,1.0,1,372,1.0,1.0,1.0,1,787,1.0,1.0,1.0 +boolean_and,q029,Run a docker container,1,1,231,1.0,1.0,1.0,1,336,1.0,1.0,1.0,1,758,1.0,1.0,1.0 +boolean_and,q030,Show docker container logs,184,62,14888,0.0,0.0,0.0,37,14342,0.0,0.0,0.0,19,14250,0.0,0.0,0.0 +boolean_and,q031,Navigate the browser to a URL,1,1,165,1.0,1.0,1.0,1,323,1.0,1.0,1.0,1,738,1.0,1.0,1.0 +boolean_and,q032,Take a screenshot of the page,1,1,188,1.0,1.0,1.0,1,346,1.0,1.0,1.0,1,761,1.0,1.0,1.0 +boolean_and,q033,Create a GitLab merge request,1,1,231,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,804,1.0,1.0,1.0 +boolean_and,q034,Search Notion pages,1,1,181,1.0,1.0,1.0,1,339,1.0,1.0,1.0,1,754,1.0,1.0,1.0 +boolean_and,q035,Create a Notion page,1,1,200,1.0,1.0,1.0,1,358,1.0,1.0,1.0,1,773,1.0,1.0,1.0 +boolean_and,q036,Query a Notion database,1,1,198,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,771,1.0,1.0,1.0 +boolean_and,q037,Create a Zendesk ticket,1,1,201,1.0,1.0,1.0,1,359,1.0,1.0,1.0,1,774,1.0,1.0,1.0 +boolean_and,q038,Create a HubSpot contact,1,1,198,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,771,1.0,1.0,1.0 +boolean_and,q039,Create a Salesforce lead,1,1,220,1.0,1.0,1.0,1,378,1.0,1.0,1.0,1,793,1.0,1.0,1.0 +boolean_and,q040,Run a SOQL query,1,1,173,1.0,1.0,1.0,1,331,1.0,1.0,1.0,1,746,1.0,1.0,1.0 +boolean_and,q041,List Shopify orders,1,1,177,1.0,1.0,1.0,1,335,1.0,1.0,1.0,1,750,1.0,1.0,1.0 +boolean_and,q042,Create a PagerDuty incident,1,1,217,1.0,1.0,1.0,1,375,1.0,1.0,1.0,1,790,1.0,1.0,1.0 +boolean_and,q043,Who is on call in PagerDuty,243,81,19511,0.0,0.0,0.0,49,18939,0.0,0.0,0.0,25,18705,0.0,0.0,0.0 +boolean_and,q044,Query Datadog metrics,1,1,214,1.0,1.0,1.0,1,372,1.0,1.0,1.0,1,787,1.0,1.0,1.0 +boolean_and,q045,Search Datadog logs,1,1,212,1.0,1.0,1.0,1,370,1.0,1.0,1.0,1,785,1.0,1.0,1.0 +boolean_and,q046,List S3 buckets,1,1,153,1.0,1.0,1.0,1,311,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +boolean_and,q047,Upload an object to S3,1,1,201,1.0,1.0,1.0,1,359,1.0,1.0,1.0,1,774,1.0,1.0,1.0 +boolean_and,q048,Purge Cloudflare cache,1,1,183,1.0,1.0,1.0,1,341,1.0,1.0,1.0,1,756,1.0,1.0,1.0 +boolean_and,q049,Create a Cloudflare DNS record,1,1,217,1.0,1.0,1.0,1,375,1.0,1.0,1.0,1,790,1.0,1.0,1.0 +boolean_and,q050,List Vercel deployments,1,1,171,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,744,1.0,1.0,1.0 +boolean_and,q051,Insert a row into Supabase,308,103,24500,0.0,0.0,0.0,62,23675,0.0,0.0,0.0,31,22993,0.0,0.0,0.0 +boolean_and,q052,Export images from Figma,1,1,201,1.0,1.0,1.0,1,359,1.0,1.0,1.0,1,774,1.0,1.0,1.0 +boolean_and,q053,Send an SMS with Twilio,316,106,25217,0.0,0.0,0.0,64,24398,0.0,0.0,0.0,32,23694,0.0,0.0,0.0 +boolean_and,q054,Create an Okta user,1,1,165,1.0,1.0,1.0,1,323,1.0,1.0,1.0,1,738,1.0,1.0,1.0 +boolean_and,q055,Instant query Prometheus,1,1,164,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,737,1.0,1.0,1.0 +boolean_and,q056,Search Grafana dashboards,1,1,171,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,744,1.0,1.0,1.0 +boolean_and,q057,Trigger a CircleCI pipeline,1,1,183,1.0,1.0,1.0,1,341,1.0,1.0,1.0,1,756,1.0,1.0,1.0 +boolean_and,q058,Run terraform plan,1,1,155,1.0,1.0,1.0,1,313,1.0,1.0,1.0,1,728,1.0,1.0,1.0 +boolean_and,q059,Create an Asana task,1,1,218,1.0,1.0,1.0,1,376,1.0,1.0,1.0,1,791,1.0,1.0,1.0 +boolean_and,q060,Create a Trello card,1,1,204,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,777,1.0,1.0,1.0 +boolean_and,q061,Create a ClickUp task,1,1,204,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,777,1.0,1.0,1.0 +boolean_and,q062,Create a Confluence page,1,1,204,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,777,1.0,1.0,1.0 +boolean_and,q063,Send a Discord message,1,1,179,1.0,1.0,1.0,1,337,1.0,1.0,1.0,1,752,1.0,1.0,1.0 +boolean_and,q064,Search Elasticsearch,1,1,176,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,749,1.0,1.0,1.0 +boolean_and,q065,Get a Redis key,1,1,157,1.0,1.0,1.0,1,315,1.0,1.0,1.0,1,730,1.0,1.0,1.0 +boolean_and,q066,Find MongoDB documents,1,1,188,1.0,1.0,1.0,1,346,1.0,1.0,1.0,1,761,1.0,1.0,1.0 +boolean_and,q067,List SQLite tables,1,1,147,1.0,1.0,1.0,1,305,1.0,1.0,1.0,1,720,1.0,1.0,1.0 +boolean_and,q068,Describe a postgres table,1,1,176,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,749,1.0,1.0,1.0 +boolean_and,q069,github_create_issue,1,1,341,1.0,1.0,1.0,1,446,1.0,1.0,1.0,1,868,1.0,1.0,1.0 +boolean_and,q070,createIssue,1,1,418,1.0,1.0,1.0,1,600,1.0,1.0,1.0,1,941,1.0,1.0,1.0 +boolean_and,q071,list-directory,1,1,185,1.0,1.0,1.0,1,290,1.0,1.0,1.0,1,712,1.0,1.0,1.0 +boolean_and,q072,sequential thinking next thought,1,1,224,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,797,1.0,1.0,1.0 +boolean_and,q073,What time is it in Tokyo,117,39,9838,0.0,0.0,0.0,24,9751,0.0,0.0,0.0,12,9487,0.0,0.0,0.0 +boolean_and,q074,Convert time between timezones,1,1,191,1.0,1.0,1.0,1,349,1.0,1.0,1.0,1,764,1.0,1.0,1.0 +boolean_and,q075,Star a GitHub repo,1,1,180,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,753,1.0,1.0,1.0 +boolean_and,q076,Fork a repository,1,1,193,1.0,1.0,1.0,1,351,1.0,1.0,1.0,1,766,1.0,1.0,1.0 +boolean_and,q077,Get the latest GitHub release,1,1,185,1.0,1.0,1.0,1,343,1.0,1.0,1.0,1,758,1.0,1.0,1.0 +boolean_and,q078,Create a gist,1,1,189,1.0,1.0,1.0,1,347,1.0,1.0,1.0,1,762,1.0,1.0,1.0 +boolean_and,q079,Search GitHub code,1,1,180,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,782,1.0,1.0,1.0 +boolean_and,q080,List GitHub notifications,1,1,177,1.0,1.0,1.0,1,335,1.0,1.0,1.0,1,750,1.0,1.0,1.0 +boolean_and,q081,Assign Copilot to an issue,1,1,218,1.0,1.0,1.0,1,376,1.0,1.0,1.0,1,791,1.0,1.0,1.0 +boolean_and,q082,Get secret scanning alerts,49,17,5097,0.0,0.0,0.0,10,4866,0.0,0.0,0.0,5,4766,0.0,0.0,0.0 +boolean_and,q083,Create or update a file on GitHub,17,6,1593,0.0,0.0,0.0,4,1694,0.0,0.0,0.0,2,1646,0.0,1.0,1.0 +boolean_and,q084,List collaborators on a repo,1,1,191,1.0,1.0,1.0,1,349,1.0,1.0,1.0,1,764,1.0,1.0,1.0 +boolean_and,q085,Reply in a Slack thread,122,41,10333,0.0,0.0,0.0,25,10147,0.0,0.0,0.0,13,10188,0.0,0.0,0.0 +boolean_and,q086,Add an emoji reaction on Slack,123,41,10374,0.0,0.0,0.0,25,10172,0.0,0.0,0.0,13,10201,0.0,0.0,0.0 +boolean_and,q087,Search Slack messages,1,1,180,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,753,1.0,1.0,1.0 +boolean_and,q088,Create a Slack channel,1,1,184,1.0,1.0,1.0,1,342,1.0,1.0,1.0,1,757,1.0,1.0,1.0 +boolean_and,q089,Transition a Jira issue,1,1,181,1.0,1.0,1.0,1,339,1.0,1.0,1.0,1,754,1.0,1.0,1.0 +boolean_and,q090,Search Jira with JQL,1,1,185,1.0,1.0,1.0,1,343,1.0,1.0,1.0,1,758,1.0,1.0,1.0 +boolean_and,q091,Add a worklog to Jira,1,1,204,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,777,1.0,1.0,1.0 +boolean_and,q092,List Linear teams,1,1,145,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,718,1.0,1.0,1.0 +boolean_and,q093,Comment on a Linear issue,141,47,11773,0.0,0.0,0.0,29,11631,0.0,0.0,0.0,15,11710,0.0,0.0,0.0 +boolean_and,q094,Move a local file,85,29,7859,0.0,0.0,0.0,17,7496,0.0,0.0,0.0,9,7598,0.0,0.0,0.0 +boolean_and,q095,Recursively search files by pattern,86,29,7917,0.0,0.0,0.0,18,7814,0.0,0.0,0.0,9,7616,0.0,0.0,0.0 +boolean_and,q096,Edit a file in place with old and new string,80,27,7629,0.0,0.0,0.0,16,7288,0.0,0.0,0.0,8,7080,0.0,0.0,0.0 +boolean_and,q097,Read the whole knowledge graph,95,32,8392,0.0,0.0,0.0,19,8062,0.0,0.0,0.0,10,8146,0.0,0.0,0.0 +boolean_and,q098,Search memory nodes,1,1,168,1.0,1.0,1.0,1,326,1.0,1.0,1.0,1,741,1.0,1.0,1.0 +boolean_and,q099,Show unstaged git diff,1,1,167,1.0,1.0,1.0,1,325,1.0,1.0,1.0,1,740,1.0,1.0,1.0 +boolean_and,q100,Stage files with git add,109,37,9343,0.0,0.0,0.0,22,8966,0.0,0.0,0.0,11,8735,0.0,0.0,0.0 +boolean_and,q101,Create a Shopify product,1,1,202,1.0,1.0,1.0,1,360,1.0,1.0,1.0,1,775,1.0,1.0,1.0 +boolean_and,q102,Cancel a Stripe subscription,1,1,162,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,735,1.0,1.0,1.0 +boolean_and,q103,Create a payment intent,1,1,230,1.0,1.0,1.0,1,335,1.0,1.0,1.0,1,757,1.0,1.0,1.0 +boolean_and,q104,Acknowledge a PagerDuty incident,1,1,185,1.0,1.0,1.0,1,343,1.0,1.0,1.0,1,758,1.0,1.0,1.0 +boolean_and,q105,Create a Datadog monitor,1,1,229,1.0,1.0,1.0,1,387,1.0,1.0,1.0,1,802,1.0,1.0,1.0 +boolean_and,q106,Resolve a Sentry issue,231,77,18333,0.0,0.0,0.0,47,17977,0.0,0.0,0.0,24,17954,0.0,0.0,0.0 +boolean_and,q107,Scale a Kubernetes deployment,1,1,210,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,783,1.0,1.0,1.0 +boolean_and,q108,Apply a Kubernetes manifest,1,1,179,1.0,1.0,1.0,1,337,1.0,1.0,1.0,1,752,1.0,1.0,1.0 +boolean_and,q109,Stop a docker container,1,1,158,1.0,1.0,1.0,1,316,1.0,1.0,1.0,1,731,1.0,1.0,1.0 +boolean_and,q110,Build a docker image,1,1,177,1.0,1.0,1.0,1,335,1.0,1.0,1.0,1,750,1.0,1.0,1.0 +boolean_and,q111,Click an element in Playwright,168,56,13816,0.0,0.0,0.0,34,13458,0.0,0.0,0.0,17,13084,0.0,0.0,0.0 +boolean_and,q112,Type text into a browser input,169,57,14038,0.0,0.0,0.0,34,13458,0.0,0.0,0.0,17,13084,0.0,0.0,0.0 +boolean_and,q113,List GitLab projects,1,1,163,1.0,1.0,1.0,1,321,1.0,1.0,1.0,1,736,1.0,1.0,1.0 +boolean_and,q114,Create a Bitbucket pull request,1,1,238,1.0,1.0,1.0,1,396,1.0,1.0,1.0,1,811,1.0,1.0,1.0 +boolean_and,q115,Create a Gmail draft,1,1,193,1.0,1.0,1.0,1,351,1.0,1.0,1.0,1,766,1.0,1.0,1.0 +boolean_and,q116,Share a Google Drive file,1,1,199,1.0,1.0,1.0,1,357,1.0,1.0,1.0,1,772,1.0,1.0,1.0 +boolean_and,q117,List Google Calendar events,1,1,201,1.0,1.0,1.0,1,359,1.0,1.0,1.0,1,774,1.0,1.0,1.0 +boolean_and,q118,Delete a calendar event,1,1,184,1.0,1.0,1.0,1,342,1.0,1.0,1.0,1,757,1.0,1.0,1.0 +boolean_and,q119,List Vercel projects,1,1,151,1.0,1.0,1.0,1,309,1.0,1.0,1.0,1,724,1.0,1.0,1.0 +boolean_and,q120,Select rows from Supabase,1,1,180,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,753,1.0,1.0,1.0 +boolean_and,q121,Post a Figma comment,1,1,186,1.0,1.0,1.0,1,344,1.0,1.0,1.0,1,759,1.0,1.0,1.0 +boolean_and,q122,List Okta users,1,1,163,1.0,1.0,1.0,1,321,1.0,1.0,1.0,1,736,1.0,1.0,1.0 +boolean_and,q123,Range query Prometheus,1,1,211,1.0,1.0,1.0,1,369,1.0,1.0,1.0,1,784,1.0,1.0,1.0 +boolean_and,q124,Create a Grafana annotation,1,1,198,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,771,1.0,1.0,1.0 +boolean_and,q125,Apply terraform,1,1,171,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,744,1.0,1.0,1.0 +boolean_and,q126,List Asana tasks,1,1,179,1.0,1.0,1.0,1,337,1.0,1.0,1.0,1,752,1.0,1.0,1.0 +boolean_and,q127,Comment on a Trello card,1,1,185,1.0,1.0,1.0,1,343,1.0,1.0,1.0,1,758,1.0,1.0,1.0 +boolean_and,q128,Update a ClickUp task,1,1,204,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,777,1.0,1.0,1.0 +boolean_and,q129,Search Confluence,1,1,165,1.0,1.0,1.0,1,323,1.0,1.0,1.0,1,738,1.0,1.0,1.0 +boolean_and,q130,Create a HubSpot deal,1,1,204,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,777,1.0,1.0,1.0 +boolean_and,q131,List Zendesk tickets,1,1,166,1.0,1.0,1.0,1,324,1.0,1.0,1.0,1,739,1.0,1.0,1.0 +boolean_and,q132,List Discord channels,1,1,160,1.0,1.0,1.0,1,318,1.0,1.0,1.0,1,733,1.0,1.0,1.0 +boolean_and,q133,Set a Redis key,1,1,185,1.0,1.0,1.0,1,343,1.0,1.0,1.0,1,758,1.0,1.0,1.0 +boolean_and,q134,Insert MongoDB documents,1,1,186,1.0,1.0,1.0,1,344,1.0,1.0,1.0,1,759,1.0,1.0,1.0 +boolean_and,q135,Run a MySQL query,1,1,166,1.0,1.0,1.0,1,324,1.0,1.0,1.0,1,739,1.0,1.0,1.0 +boolean_and,q136,Index a document in Elasticsearch,1,1,198,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,771,1.0,1.0,1.0 +boolean_and,q137,Get an S3 object,1,1,184,1.0,1.0,1.0,1,342,1.0,1.0,1.0,1,757,1.0,1.0,1.0 +boolean_and,q138,List Cloudflare zones,1,1,149,1.0,1.0,1.0,1,307,1.0,1.0,1.0,1,722,1.0,1.0,1.0 +boolean_and,q139,Create a Stripe invoice,1,1,160,1.0,1.0,1.0,1,318,1.0,1.0,1.0,1,733,1.0,1.0,1.0 +boolean_and,q140,List GitHub Actions workflows,1,1,180,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,753,1.0,1.0,1.0 +boolean_and,q141,Get GitHub Actions job logs,1,1,208,1.0,1.0,1.0,1,366,1.0,1.0,1.0,1,781,1.0,1.0,1.0 +boolean_and,q142,Create a GitHub repository,2,1,346,1.0,1.0,1.0,1,532,1.0,1.0,1.0,1,938,1.0,1.0,1.0 +boolean_and,q143,Checkout a git branch,1,1,174,1.0,1.0,1.0,1,332,1.0,1.0,1.0,1,747,1.0,1.0,1.0 +boolean_and,q144,Show git log,1,1,173,1.0,1.0,1.0,1,331,1.0,1.0,1.0,1,746,1.0,1.0,1.0 +boolean_and,q145,List postgres schemas,1,1,148,1.0,1.0,1.0,1,306,1.0,1.0,1.0,1,721,1.0,1.0,1.0 +boolean_and,q146,Open nodes in the knowledge graph,1,1,164,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,737,1.0,1.0,1.0 +boolean_and,q147,Get file metadata from the filesystem,87,29,7917,0.0,0.0,0.0,18,7814,0.0,0.0,0.0,9,7616,0.0,0.0,0.0 +boolean_and,q148,Write a new local file,79,27,7494,0.0,0.0,0.0,16,7208,0.0,0.0,0.0,8,7040,0.0,0.0,0.0 +boolean_and,q149,Search GitHub issues,1,1,198,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,771,1.0,1.0,1.0 +boolean_and,q150,Get the authenticated GitHub user,1,1,181,1.0,1.0,1.0,1,379,1.0,1.0,1.0,1,784,1.0,1.0,1.0 +boolean_or,q001,Create a GitHub issue,1,1,434,1.0,1.0,1.0,1,614,1.0,1.0,1.0,1,1216,1.0,1.0,1.0 +boolean_or,q002,Create a Jira ticket,1,1,333,1.0,1.0,1.0,1,528,1.0,1.0,1.0,1,1058,1.0,1.0,1.0 +boolean_or,q003,Send a Slack message,1,1,247,1.0,1.0,1.0,1,507,1.0,1.0,1.0,1,924,1.0,1.0,1.0 +boolean_or,q004,List open pull requests,1,1,334,1.0,1.0,1.0,1,423,1.0,1.0,1.0,1,808,1.0,1.0,1.0 +boolean_or,q005,Open a pull request on GitHub,1,1,364,1.0,1.0,1.0,1,611,1.0,1.0,1.0,1,1194,1.0,1.0,1.0 +boolean_or,q006,Run a SQL query against postgres,1,1,180,1.0,1.0,1.0,1,295,1.0,1.0,1.0,1,604,1.0,1.0,1.0 +boolean_or,q007,Read a local file from disk,1,1,223,1.0,1.0,1.0,1,444,1.0,1.0,1.0,1,881,1.0,1.0,1.0 +boolean_or,q008,What files are in this folder,85,29,7919,0.0,0.0,0.0,17,7541,0.0,0.0,0.0,9,7663,0.0,0.0,0.0 +boolean_or,q009,Remember this fact in the knowledge graph,1,1,178,1.0,1.0,1.0,1,285,1.0,1.0,1.0,1,506,1.0,1.0,1.0 +boolean_or,q010,Search the web with Brave,1,1,262,1.0,1.0,1.0,1,410,1.0,1.0,1.0,1,714,1.0,1.0,1.0 +boolean_or,q011,Fetch the contents of a URL,1,1,211,1.0,1.0,1.0,1,369,1.0,1.0,1.0,1,671,1.0,1.0,1.0 +boolean_or,q012,Show git status,1,1,176,1.0,1.0,1.0,1,302,1.0,1.0,1.0,1,613,1.0,1.0,1.0 +boolean_or,q013,Commit staged changes,1,1,215,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,663,1.0,1.0,1.0 +boolean_or,q014,Create a Linear issue,1,1,351,1.0,1.0,1.0,1,601,1.0,1.0,1.0,1,1019,1.0,1.0,1.0 +boolean_or,q015,Make a task in my project tracker Linear,1,1,353,1.0,1.0,1.0,1,488,1.0,1.0,1.0,1,823,1.0,1.0,1.0 +boolean_or,q016,Post a comment on a GitHub issue,2,1,338,1.0,1.0,1.0,1,519,1.0,1.0,1.0,1,1190,1.0,1.0,1.0 +boolean_or,q017,Merge the pull request,1,1,370,1.0,1.0,1.0,1,602,1.0,1.0,1.0,1,1196,1.0,1.0,1.0 +boolean_or,q018,Trigger a GitHub Actions workflow,1,1,294,1.0,1.0,1.0,1,442,1.0,1.0,1.0,1,848,1.0,1.0,1.0 +boolean_or,q019,Get Dependabot alerts,1,1,243,1.0,1.0,1.0,1,348,1.0,1.0,1.0,1,792,1.0,1.0,1.0 +boolean_or,q020,List Sentry issues,1,1,302,1.0,1.0,1.0,1,425,1.0,1.0,1.0,1,809,1.0,1.0,1.0 +boolean_or,q021,Create a Stripe customer,1,1,223,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,749,1.0,1.0,1.0 +boolean_or,q022,Refund a Stripe payment,1,1,234,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,662,1.0,1.0,1.0 +boolean_or,q023,Send an email with Gmail,1,1,237,1.0,1.0,1.0,1,385,1.0,1.0,1.0,1,859,1.0,1.0,1.0 +boolean_or,q024,Search my Gmail inbox,1,1,250,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +boolean_or,q025,Upload a file to Google Drive,3,1,269,1.0,1.0,1.0,1,446,1.0,1.0,1.0,1,801,1.0,1.0,1.0 +boolean_or,q026,Create a calendar event,1,1,371,1.0,1.0,1.0,1,588,1.0,1.0,1.0,1,1186,1.0,1.0,1.0 +boolean_or,q027,List Kubernetes pods,1,1,184,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,675,1.0,1.0,1.0 +boolean_or,q028,Get logs from a k8s pod,1,1,314,1.0,1.0,1.0,1,481,1.0,1.0,1.0,1,881,1.0,1.0,1.0 +boolean_or,q029,Run a docker container,1,1,239,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,682,1.0,1.0,1.0 +boolean_or,q030,Show docker container logs,1,1,237,1.0,1.0,1.0,1,343,1.0,1.0,1.0,1,705,1.0,1.0,1.0 +boolean_or,q031,Navigate the browser to a URL,1,1,170,1.0,1.0,1.0,1,304,1.0,1.0,1.0,1,609,1.0,1.0,1.0 +boolean_or,q032,Take a screenshot of the page,1,1,189,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,642,1.0,1.0,1.0 +boolean_or,q033,Create a GitLab merge request,1,1,297,1.0,1.0,1.0,1,552,1.0,1.0,1.0,1,1075,1.0,1.0,1.0 +boolean_or,q034,Search Notion pages,1,1,248,1.0,1.0,1.0,1,387,1.0,1.0,1.0,1,667,1.0,1.0,1.0 +boolean_or,q035,Create a Notion page,1,1,230,1.0,1.0,1.0,1,371,1.0,1.0,1.0,1,856,1.0,1.0,1.0 +boolean_or,q036,Query a Notion database,1,1,262,1.0,1.0,1.0,1,439,1.0,1.0,1.0,1,749,1.0,1.0,1.0 +boolean_or,q037,Create a Zendesk ticket,1,1,282,1.0,1.0,1.0,1,517,1.0,1.0,1.0,1,1115,1.0,1.0,1.0 +boolean_or,q038,Create a HubSpot contact,1,1,255,1.0,1.0,1.0,1,480,1.0,1.0,1.0,1,1141,1.0,1.0,1.0 +boolean_or,q039,Create a Salesforce lead,1,1,252,1.0,1.0,1.0,1,477,1.0,1.0,1.0,1,1138,1.0,1.0,1.0 +boolean_or,q040,Run a SOQL query,1,1,186,1.0,1.0,1.0,1,330,1.0,1.0,1.0,1,676,1.0,1.0,1.0 +boolean_or,q041,List Shopify orders,1,1,208,1.0,1.0,1.0,1,378,1.0,1.0,1.0,1,725,1.0,1.0,1.0 +boolean_or,q042,Create a PagerDuty incident,1,1,259,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,922,1.0,1.0,1.0 +boolean_or,q043,Who is on call in PagerDuty,1,1,220,1.0,1.0,1.0,1,388,1.0,1.0,1.0,1,796,1.0,1.0,1.0 +boolean_or,q044,Query Datadog metrics,1,1,300,1.0,1.0,1.0,1,490,1.0,1.0,1.0,1,800,1.0,1.0,1.0 +boolean_or,q045,Search Datadog logs,1,1,279,1.0,1.0,1.0,1,418,1.0,1.0,1.0,1,735,1.0,1.0,1.0 +boolean_or,q046,List S3 buckets,1,1,231,1.0,1.0,1.0,1,373,1.0,1.0,1.0,1,768,1.0,1.0,1.0 +boolean_or,q047,Upload an object to S3,1,1,303,1.0,1.0,1.0,1,479,1.0,1.0,1.0,1,954,1.0,1.0,1.0 +boolean_or,q048,Purge Cloudflare cache,1,1,183,1.0,1.0,1.0,1,340,1.0,1.0,1.0,1,746,1.0,1.0,1.0 +boolean_or,q049,Create a Cloudflare DNS record,1,1,249,1.0,1.0,1.0,1,474,1.0,1.0,1.0,1,1135,1.0,1.0,1.0 +boolean_or,q050,List Vercel deployments,1,1,173,1.0,1.0,1.0,1,343,1.0,1.0,1.0,1,690,1.0,1.0,1.0 +boolean_or,q051,Insert a row into Supabase,1,1,240,1.0,1.0,1.0,1,347,1.0,1.0,1.0,1,665,1.0,1.0,1.0 +boolean_or,q052,Export images from Figma,1,1,292,1.0,1.0,1.0,1,477,1.0,1.0,1.0,1,844,1.0,1.0,1.0 +boolean_or,q053,Send an SMS with Twilio,1,1,232,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,836,1.0,1.0,1.0 +boolean_or,q054,Create an Okta user,1,1,183,1.0,1.0,1.0,1,385,1.0,1.0,1.0,1,748,1.0,1.0,1.0 +boolean_or,q055,Instant query Prometheus,1,1,262,1.0,1.0,1.0,1,383,1.0,1.0,1.0,1,681,1.0,1.0,1.0 +boolean_or,q056,Search Grafana dashboards,1,1,238,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,657,1.0,1.0,1.0 +boolean_or,q057,Trigger a CircleCI pipeline,1,1,216,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,755,1.0,1.0,1.0 +boolean_or,q058,Run terraform plan,1,1,210,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,706,1.0,1.0,1.0 +boolean_or,q059,Create an Asana task,1,1,267,1.0,1.0,1.0,1,470,1.0,1.0,1.0,1,908,1.0,1.0,1.0 +boolean_or,q060,Create a Trello card,1,1,240,1.0,1.0,1.0,1,465,1.0,1.0,1.0,1,1126,1.0,1.0,1.0 +boolean_or,q061,Create a ClickUp task,1,1,257,1.0,1.0,1.0,1,415,1.0,1.0,1.0,1,1008,1.0,1.0,1.0 +boolean_or,q062,Create a Confluence page,1,1,256,1.0,1.0,1.0,1,438,1.0,1.0,1.0,1,934,1.0,1.0,1.0 +boolean_or,q063,Send a Discord message,1,1,320,1.0,1.0,1.0,1,549,1.0,1.0,1.0,1,981,1.0,1.0,1.0 +boolean_or,q064,Search Elasticsearch,1,1,243,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,662,1.0,1.0,1.0 +boolean_or,q065,Get a Redis key,1,1,190,1.0,1.0,1.0,1,315,1.0,1.0,1.0,1,627,1.0,1.0,1.0 +boolean_or,q066,Find MongoDB documents,1,1,258,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,738,1.0,1.0,1.0 +boolean_or,q067,List SQLite tables,1,1,156,1.0,1.0,1.0,1,252,1.0,1.0,1.0,1,604,1.0,1.0,1.0 +boolean_or,q068,Describe a postgres table,1,1,186,1.0,1.0,1.0,1,272,1.0,1.0,1.0,1,597,1.0,1.0,1.0 +boolean_or,q069,github_create_issue,1,1,432,1.0,1.0,1.0,1,612,1.0,1.0,1.0,1,1214,1.0,1.0,1.0 +boolean_or,q070,createIssue,1,1,418,1.0,1.0,1.0,1,600,1.0,1.0,1.0,1,1107,1.0,1.0,1.0 +boolean_or,q071,list-directory,1,1,198,1.0,1.0,1.0,1,408,1.0,1.0,1.0,1,755,1.0,1.0,1.0 +boolean_or,q072,sequential thinking next thought,1,1,224,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,797,1.0,1.0,1.0 +boolean_or,q073,What time is it in Tokyo,1,1,241,1.0,1.0,1.0,1,456,1.0,1.0,1.0,1,993,1.0,1.0,1.0 +boolean_or,q074,Convert time between timezones,1,1,214,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,704,1.0,1.0,1.0 +boolean_or,q075,Star a GitHub repo,1,1,298,1.0,1.0,1.0,1,479,1.0,1.0,1.0,1,849,1.0,1.0,1.0 +boolean_or,q076,Fork a repository,1,1,311,1.0,1.0,1.0,1,480,1.0,1.0,1.0,1,932,1.0,1.0,1.0 +boolean_or,q077,Get the latest GitHub release,1,1,185,1.0,1.0,1.0,1,383,1.0,1.0,1.0,1,823,1.0,1.0,1.0 +boolean_or,q078,Create a gist,1,1,322,1.0,1.0,1.0,1,554,1.0,1.0,1.0,1,1115,1.0,1.0,1.0 +boolean_or,q079,Search GitHub code,1,1,232,1.0,1.0,1.0,1,371,1.0,1.0,1.0,1,722,1.0,1.0,1.0 +boolean_or,q080,List GitHub notifications,1,1,255,1.0,1.0,1.0,1,397,1.0,1.0,1.0,1,792,1.0,1.0,1.0 +boolean_or,q081,Assign Copilot to an issue,1,1,332,1.0,1.0,1.0,1,527,1.0,1.0,1.0,1,864,1.0,1.0,1.0 +boolean_or,q082,Get secret scanning alerts,1,1,294,1.0,1.0,1.0,1,446,1.0,1.0,1.0,1,839,1.0,1.0,1.0 +boolean_or,q083,Create or update a file on GitHub,1,1,389,1.0,1.0,1.0,1,612,1.0,1.0,1.0,1,1243,1.0,1.0,1.0 +boolean_or,q084,List collaborators on a repo,1,1,274,1.0,1.0,1.0,1,416,1.0,1.0,1.0,1,965,1.0,1.0,1.0 +boolean_or,q085,Reply in a Slack thread,1,1,254,1.0,1.0,1.0,1,411,1.0,1.0,1.0,1,781,1.0,1.0,1.0 +boolean_or,q086,Add an emoji reaction on Slack,1,1,304,1.0,1.0,1.0,1,461,1.0,1.0,1.0,1,950,1.0,1.0,1.0 +boolean_or,q087,Search Slack messages,1,1,232,1.0,1.0,1.0,1,391,1.0,1.0,1.0,1,685,1.0,1.0,1.0 +boolean_or,q088,Create a Slack channel,1,1,251,1.0,1.0,1.0,1,414,1.0,1.0,1.0,1,751,1.0,1.0,1.0 +boolean_or,q089,Transition a Jira issue,1,1,270,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,914,1.0,1.0,1.0 +boolean_or,q090,Search Jira with JQL,1,1,252,1.0,1.0,1.0,1,391,1.0,1.0,1.0,1,700,1.0,1.0,1.0 +boolean_or,q091,Add a worklog to Jira,1,1,285,1.0,1.0,1.0,1,495,1.0,1.0,1.0,1,874,1.0,1.0,1.0 +boolean_or,q092,List Linear teams,1,1,184,1.0,1.0,1.0,1,330,1.0,1.0,1.0,1,658,1.0,1.0,1.0 +boolean_or,q093,Comment on a Linear issue,2,1,298,1.0,1.0,1.0,1,534,1.0,1.0,1.0,1,936,1.0,1.0,1.0 +boolean_or,q094,Move a local file,4,2,506,0.0,1.0,1.0,1,389,1.0,1.0,1.0,1,907,1.0,1.0,1.0 +boolean_or,q095,Recursively search files by pattern,1,1,236,1.0,1.0,1.0,1,359,1.0,1.0,1.0,1,758,1.0,1.0,1.0 +boolean_or,q096,Edit a file in place with old and new string,1,1,360,1.0,1.0,1.0,1,489,1.0,1.0,1.0,1,875,1.0,1.0,1.0 +boolean_or,q097,Read the whole knowledge graph,1,1,154,1.0,1.0,1.0,1,262,1.0,1.0,1.0,1,561,1.0,1.0,1.0 +boolean_or,q098,Search memory nodes,1,1,217,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,647,1.0,1.0,1.0 +boolean_or,q099,Show unstaged git diff,1,1,190,1.0,1.0,1.0,1,304,1.0,1.0,1.0,1,615,1.0,1.0,1.0 +boolean_or,q100,Stage files with git add,1,1,306,1.0,1.0,1.0,1,538,1.0,1.0,1.0,1,836,1.0,1.0,1.0 +boolean_or,q101,Create a Shopify product,1,1,258,1.0,1.0,1.0,1,388,1.0,1.0,1.0,1,914,1.0,1.0,1.0 +boolean_or,q102,Cancel a Stripe subscription,1,1,188,1.0,1.0,1.0,1,340,1.0,1.0,1.0,1,643,1.0,1.0,1.0 +boolean_or,q103,Create a payment intent,1,1,234,1.0,1.0,1.0,1,459,1.0,1.0,1.0,1,1120,1.0,1.0,1.0 +boolean_or,q104,Acknowledge a PagerDuty incident,1,1,261,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,717,1.0,1.0,1.0 +boolean_or,q105,Create a Datadog monitor,1,1,341,1.0,1.0,1.0,1,521,1.0,1.0,1.0,1,1160,1.0,1.0,1.0 +boolean_or,q106,Resolve a Sentry issue,3,1,251,1.0,1.0,1.0,1,519,1.0,1.0,1.0,1,1064,1.0,1.0,1.0 +boolean_or,q107,Scale a Kubernetes deployment,1,1,253,1.0,1.0,1.0,1,418,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +boolean_or,q108,Apply a Kubernetes manifest,1,1,222,1.0,1.0,1.0,1,408,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +boolean_or,q109,Stop a docker container,1,1,221,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,641,1.0,1.0,1.0 +boolean_or,q110,Build a docker image,1,1,236,1.0,1.0,1.0,1,344,1.0,1.0,1.0,1,601,1.0,1.0,1.0 +boolean_or,q111,Click an element in Playwright,1,1,236,1.0,1.0,1.0,1,422,1.0,1.0,1.0,1,772,1.0,1.0,1.0 +boolean_or,q112,Type text into a browser input,1,1,232,1.0,1.0,1.0,1,410,1.0,1.0,1.0,1,714,1.0,1.0,1.0 +boolean_or,q113,List GitLab projects,1,1,209,1.0,1.0,1.0,1,349,1.0,1.0,1.0,1,721,1.0,1.0,1.0 +boolean_or,q114,Create a Bitbucket pull request,1,1,439,1.0,1.0,1.0,1,671,1.0,1.0,1.0,1,1243,1.0,1.0,1.0 +boolean_or,q115,Create a Gmail draft,1,1,396,1.0,1.0,1.0,1,576,1.0,1.0,1.0,1,1149,1.0,1.0,1.0 +boolean_or,q116,Share a Google Drive file,1,1,271,1.0,1.0,1.0,1,417,1.0,1.0,1.0,1,936,1.0,1.0,1.0 +boolean_or,q117,List Google Calendar events,1,1,199,1.0,1.0,1.0,1,327,1.0,1.0,1.0,1,777,1.0,1.0,1.0 +boolean_or,q118,Delete a calendar event,1,1,300,1.0,1.0,1.0,1,481,1.0,1.0,1.0,1,895,1.0,1.0,1.0 +boolean_or,q119,List Vercel projects,1,1,171,1.0,1.0,1.0,1,256,1.0,1.0,1.0,1,607,1.0,1.0,1.0 +boolean_or,q120,Select rows from Supabase,1,1,237,1.0,1.0,1.0,1,429,1.0,1.0,1.0,1,844,1.0,1.0,1.0 +boolean_or,q121,Post a Figma comment,1,1,304,1.0,1.0,1.0,1,528,1.0,1.0,1.0,1,883,1.0,1.0,1.0 +boolean_or,q122,List Okta users,1,1,203,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +boolean_or,q123,Range query Prometheus,1,1,262,1.0,1.0,1.0,1,383,1.0,1.0,1.0,1,681,1.0,1.0,1.0 +boolean_or,q124,Create a Grafana annotation,1,1,331,1.0,1.0,1.0,1,563,1.0,1.0,1.0,1,1154,1.0,1.0,1.0 +boolean_or,q125,Apply terraform,1,1,190,1.0,1.0,1.0,1,292,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +boolean_or,q126,List Asana tasks,1,1,207,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,759,1.0,1.0,1.0 +boolean_or,q127,Comment on a Trello card,1,1,271,1.0,1.0,1.0,1,438,1.0,1.0,1.0,1,1028,1.0,1.0,1.0 +boolean_or,q128,Update a ClickUp task,1,1,257,1.0,1.0,1.0,1,399,1.0,1.0,1.0,1,1022,1.0,1.0,1.0 +boolean_or,q129,Search Confluence,1,1,232,1.0,1.0,1.0,1,371,1.0,1.0,1.0,1,651,1.0,1.0,1.0 +boolean_or,q130,Create a HubSpot deal,1,1,342,1.0,1.0,1.0,1,522,1.0,1.0,1.0,1,1161,1.0,1.0,1.0 +boolean_or,q131,List Zendesk tickets,1,1,198,1.0,1.0,1.0,1,373,1.0,1.0,1.0,1,750,1.0,1.0,1.0 +boolean_or,q132,List Discord channels,1,1,184,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,701,1.0,1.0,1.0 +boolean_or,q133,Set a Redis key,1,1,211,1.0,1.0,1.0,1,392,1.0,1.0,1.0,1,738,1.0,1.0,1.0 +boolean_or,q134,Insert MongoDB documents,1,1,258,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +boolean_or,q135,Run a MySQL query,1,1,179,1.0,1.0,1.0,1,294,1.0,1.0,1.0,1,639,1.0,1.0,1.0 +boolean_or,q136,Index a document in Elasticsearch,1,1,242,1.0,1.0,1.0,1,411,1.0,1.0,1.0,1,973,1.0,1.0,1.0 +boolean_or,q137,Get an S3 object,1,1,249,1.0,1.0,1.0,1,355,1.0,1.0,1.0,1,705,1.0,1.0,1.0 +boolean_or,q138,List Cloudflare zones,1,1,179,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,731,1.0,1.0,1.0 +boolean_or,q139,Create a Stripe invoice,1,1,223,1.0,1.0,1.0,1,371,1.0,1.0,1.0,1,897,1.0,1.0,1.0 +boolean_or,q140,List GitHub Actions workflows,1,1,258,1.0,1.0,1.0,1,400,1.0,1.0,1.0,1,795,1.0,1.0,1.0 +boolean_or,q141,Get GitHub Actions job logs,1,1,256,1.0,1.0,1.0,1,361,1.0,1.0,1.0,1,805,1.0,1.0,1.0 +boolean_or,q142,Create a GitHub repository,2,1,346,1.0,1.0,1.0,1,585,1.0,1.0,1.0,1,1012,1.0,1.0,1.0 +boolean_or,q143,Checkout a git branch,1,1,330,1.0,1.0,1.0,1,545,1.0,1.0,1.0,1,960,1.0,1.0,1.0 +boolean_or,q144,Show git log,1,1,187,1.0,1.0,1.0,1,302,1.0,1.0,1.0,1,613,1.0,1.0,1.0 +boolean_or,q145,List postgres schemas,1,1,172,1.0,1.0,1.0,1,347,1.0,1.0,1.0,1,724,1.0,1.0,1.0 +boolean_or,q146,Open nodes in the knowledge graph,1,1,177,1.0,1.0,1.0,1,284,1.0,1.0,1.0,1,505,1.0,1.0,1.0 +boolean_or,q147,Get file metadata from the filesystem,4,2,463,0.0,1.0,1.0,1,385,1.0,1.0,1.0,1,750,1.0,1.0,1.0 +boolean_or,q148,Write a new local file,1,1,234,1.0,1.0,1.0,1,384,1.0,1.0,1.0,1,927,1.0,1.0,1.0 +boolean_or,q149,Search GitHub issues,1,1,265,1.0,1.0,1.0,1,420,1.0,1.0,1.0,1,667,1.0,1.0,1.0 +boolean_or,q150,Get the authenticated GitHub user,1,1,179,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,796,1.0,1.0,1.0 +char_trigram,q001,Create a GitHub issue,1,1,380,1.0,1.0,1.0,1,557,1.0,1.0,1.0,1,1021,1.0,1.0,1.0 +char_trigram,q002,Create a Jira ticket,1,1,318,1.0,1.0,1.0,1,513,1.0,1.0,1.0,1,1004,1.0,1.0,1.0 +char_trigram,q003,Send a Slack message,2,1,249,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,653,1.0,1.0,1.0 +char_trigram,q004,List open pull requests,2,1,237,1.0,1.0,1.0,1,477,1.0,1.0,1.0,1,1034,1.0,1.0,1.0 +char_trigram,q005,Open a pull request on GitHub,1,1,327,1.0,1.0,1.0,1,505,1.0,1.0,1.0,1,1144,1.0,1.0,1.0 +char_trigram,q006,Run a SQL query against postgres,1,1,180,1.0,1.0,1.0,1,295,1.0,1.0,1.0,1,592,1.0,1.0,1.0 +char_trigram,q007,Read a local file from disk,1,1,223,1.0,1.0,1.0,1,428,1.0,1.0,1.0,1,849,1.0,1.0,1.0 +char_trigram,q008,What files are in this folder,91,31,8311,0.0,0.0,0.0,19,8176,0.0,0.0,0.0,10,8240,0.0,0.0,0.0 +char_trigram,q009,Remember this fact in the knowledge graph,4,2,335,0.0,1.0,1.0,1,255,1.0,1.0,1.0,1,543,1.0,1.0,1.0 +char_trigram,q010,Search the web with Brave,1,1,258,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,687,1.0,1.0,1.0 +char_trigram,q011,Fetch the contents of a URL,1,1,222,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,634,1.0,1.0,1.0 +char_trigram,q012,Show git status,2,1,217,1.0,1.0,1.0,1,344,1.0,1.0,1.0,1,724,1.0,1.0,1.0 +char_trigram,q013,Commit staged changes,1,1,182,1.0,1.0,1.0,1,330,1.0,1.0,1.0,1,664,1.0,1.0,1.0 +char_trigram,q014,Create a Linear issue,1,1,293,1.0,1.0,1.0,1,463,1.0,1.0,1.0,1,925,1.0,1.0,1.0 +char_trigram,q015,Make a task in my project tracker Linear,146,49,12254,0.0,0.0,0.0,30,11994,0.0,0.0,0.0,15,11634,0.0,0.0,0.0 +char_trigram,q016,Post a comment on a GitHub issue,2,1,307,1.0,1.0,1.0,1,468,1.0,1.0,1.0,1,892,1.0,1.0,1.0 +char_trigram,q017,Merge the pull request,1,1,299,1.0,1.0,1.0,1,428,1.0,1.0,1.0,1,859,1.0,1.0,1.0 +char_trigram,q018,Trigger a GitHub Actions workflow,1,1,294,1.0,1.0,1.0,1,470,1.0,1.0,1.0,1,847,1.0,1.0,1.0 +char_trigram,q019,Get Dependabot alerts,1,1,286,1.0,1.0,1.0,1,463,1.0,1.0,1.0,1,738,1.0,1.0,1.0 +char_trigram,q020,List Sentry issues,1,1,238,1.0,1.0,1.0,1,363,1.0,1.0,1.0,1,796,1.0,1.0,1.0 +char_trigram,q021,Create a Stripe customer,2,1,223,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +char_trigram,q022,Refund a Stripe payment,1,1,234,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,647,1.0,1.0,1.0 +char_trigram,q023,Send an email with Gmail,1,1,239,1.0,1.0,1.0,1,380,1.0,1.0,1.0,1,671,1.0,1.0,1.0 +char_trigram,q024,Search my Gmail inbox,1,1,226,1.0,1.0,1.0,1,331,1.0,1.0,1.0,1,579,1.0,1.0,1.0 +char_trigram,q025,Upload a file to Google Drive,6,2,523,0.0,1.0,1.0,2,786,0.0,1.0,1.0,1,764,1.0,1.0,1.0 +char_trigram,q026,Create a calendar event,1,1,300,1.0,1.0,1.0,1,484,1.0,1.0,1.0,1,939,1.0,1.0,1.0 +char_trigram,q027,List Kubernetes pods,1,1,180,1.0,1.0,1.0,1,314,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +char_trigram,q028,Get logs from a k8s pod,1,1,262,1.0,1.0,1.0,1,441,1.0,1.0,1.0,1,858,1.0,1.0,1.0 +char_trigram,q029,Run a docker container,2,1,223,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,640,1.0,1.0,1.0 +char_trigram,q030,Show docker container logs,1,1,188,1.0,1.0,1.0,1,298,1.0,1.0,1.0,1,649,1.0,1.0,1.0 +char_trigram,q031,Navigate the browser to a URL,1,1,180,1.0,1.0,1.0,1,282,1.0,1.0,1.0,1,574,1.0,1.0,1.0 +char_trigram,q032,Take a screenshot of the page,1,1,212,1.0,1.0,1.0,1,314,1.0,1.0,1.0,1,646,1.0,1.0,1.0 +char_trigram,q033,Create a GitLab merge request,1,1,297,1.0,1.0,1.0,1,475,1.0,1.0,1.0,1,967,1.0,1.0,1.0 +char_trigram,q034,Search Notion pages,1,1,211,1.0,1.0,1.0,1,335,1.0,1.0,1.0,1,693,1.0,1.0,1.0 +char_trigram,q035,Create a Notion page,1,1,230,1.0,1.0,1.0,1,413,1.0,1.0,1.0,1,860,1.0,1.0,1.0 +char_trigram,q036,Query a Notion database,1,1,261,1.0,1.0,1.0,1,404,1.0,1.0,1.0,1,730,1.0,1.0,1.0 +char_trigram,q037,Create a Zendesk ticket,2,1,257,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,892,1.0,1.0,1.0 +char_trigram,q038,Create a HubSpot contact,1,1,216,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,797,1.0,1.0,1.0 +char_trigram,q039,Create a Salesforce lead,1,1,252,1.0,1.0,1.0,1,416,1.0,1.0,1.0,1,710,1.0,1.0,1.0 +char_trigram,q040,Run a SOQL query,1,1,186,1.0,1.0,1.0,1,294,1.0,1.0,1.0,1,592,1.0,1.0,1.0 +char_trigram,q041,List Shopify orders,1,1,193,1.0,1.0,1.0,1,333,1.0,1.0,1.0,1,628,1.0,1.0,1.0 +char_trigram,q042,Create a PagerDuty incident,1,1,255,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,853,1.0,1.0,1.0 +char_trigram,q043,Who is on call in PagerDuty,1,1,215,1.0,1.0,1.0,1,363,1.0,1.0,1.0,1,765,1.0,1.0,1.0 +char_trigram,q044,Query Datadog metrics,1,1,300,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,780,1.0,1.0,1.0 +char_trigram,q045,Search Datadog logs,1,1,248,1.0,1.0,1.0,1,363,1.0,1.0,1.0,1,646,1.0,1.0,1.0 +char_trigram,q046,List S3 buckets,1,1,187,1.0,1.0,1.0,1,316,1.0,1.0,1.0,1,616,1.0,1.0,1.0 +char_trigram,q047,Upload an object to S3,1,1,250,1.0,1.0,1.0,1,426,1.0,1.0,1.0,1,830,1.0,1.0,1.0 +char_trigram,q048,Purge Cloudflare cache,1,1,183,1.0,1.0,1.0,1,335,1.0,1.0,1.0,1,548,1.0,1.0,1.0 +char_trigram,q049,Create a Cloudflare DNS record,1,1,249,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,781,1.0,1.0,1.0 +char_trigram,q050,List Vercel deployments,1,1,177,1.0,1.0,1.0,1,306,1.0,1.0,1.0,1,710,1.0,1.0,1.0 +char_trigram,q051,Insert a row into Supabase,1,1,245,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,681,1.0,1.0,1.0 +char_trigram,q052,Export images from Figma,1,1,218,1.0,1.0,1.0,1,349,1.0,1.0,1.0,1,692,1.0,1.0,1.0 +char_trigram,q053,Send an SMS with Twilio,1,1,215,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,740,1.0,1.0,1.0 +char_trigram,q054,Create an Okta user,1,1,183,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,719,1.0,1.0,1.0 +char_trigram,q055,Instant query Prometheus,1,1,221,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,660,1.0,1.0,1.0 +char_trigram,q056,Search Grafana dashboards,1,1,182,1.0,1.0,1.0,1,287,1.0,1.0,1.0,1,666,1.0,1.0,1.0 +char_trigram,q057,Trigger a CircleCI pipeline,1,1,200,1.0,1.0,1.0,1,325,1.0,1.0,1.0,1,755,1.0,1.0,1.0 +char_trigram,q058,Run terraform plan,1,1,172,1.0,1.0,1.0,1,281,1.0,1.0,1.0,1,593,1.0,1.0,1.0 +char_trigram,q059,Create an Asana task,1,1,267,1.0,1.0,1.0,1,429,1.0,1.0,1.0,1,822,1.0,1.0,1.0 +char_trigram,q060,Create a Trello card,1,1,240,1.0,1.0,1.0,1,411,1.0,1.0,1.0,1,745,1.0,1.0,1.0 +char_trigram,q061,Create a ClickUp task,1,1,257,1.0,1.0,1.0,1,415,1.0,1.0,1.0,1,802,1.0,1.0,1.0 +char_trigram,q062,Create a Confluence page,1,1,260,1.0,1.0,1.0,1,418,1.0,1.0,1.0,1,870,1.0,1.0,1.0 +char_trigram,q063,Send a Discord message,1,1,224,1.0,1.0,1.0,1,327,1.0,1.0,1.0,1,701,1.0,1.0,1.0 +char_trigram,q064,Search Elasticsearch,1,1,239,1.0,1.0,1.0,1,358,1.0,1.0,1.0,1,630,1.0,1.0,1.0 +char_trigram,q065,Get a Redis key,1,1,182,1.0,1.0,1.0,1,283,1.0,1.0,1.0,1,582,1.0,1.0,1.0 +char_trigram,q066,Find MongoDB documents,1,1,258,1.0,1.0,1.0,1,417,1.0,1.0,1.0,1,742,1.0,1.0,1.0 +char_trigram,q067,List SQLite tables,1,1,151,1.0,1.0,1.0,1,240,1.0,1.0,1.0,1,487,1.0,1.0,1.0 +char_trigram,q068,Describe a postgres table,1,1,188,1.0,1.0,1.0,1,274,1.0,1.0,1.0,1,697,1.0,1.0,1.0 +char_trigram,q069,github_create_issue,2,1,347,1.0,1.0,1.0,1,516,1.0,1.0,1.0,1,1032,1.0,1.0,1.0 +char_trigram,q070,createIssue,3,1,332,1.0,1.0,1.0,1,579,1.0,1.0,1.0,1,982,1.0,1.0,1.0 +char_trigram,q071,list-directory,1,1,186,1.0,1.0,1.0,1,333,1.0,1.0,1.0,1,586,1.0,1.0,1.0 +char_trigram,q072,sequential thinking next thought,1,1,278,1.0,1.0,1.0,1,434,1.0,1.0,1.0,1,862,1.0,1.0,1.0 +char_trigram,q073,What time is it in Tokyo,1,1,217,1.0,1.0,1.0,1,333,1.0,1.0,1.0,1,804,1.0,1.0,1.0 +char_trigram,q074,Convert time between timezones,1,1,213,1.0,1.0,1.0,1,319,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +char_trigram,q075,Star a GitHub repo,1,1,226,1.0,1.0,1.0,1,416,1.0,1.0,1.0,1,778,1.0,1.0,1.0 +char_trigram,q076,Fork a repository,1,1,240,1.0,1.0,1.0,1,378,1.0,1.0,1.0,1,764,1.0,1.0,1.0 +char_trigram,q077,Get the latest GitHub release,1,1,257,1.0,1.0,1.0,1,349,1.0,1.0,1.0,1,728,1.0,1.0,1.0 +char_trigram,q078,Create a gist,1,1,259,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,817,1.0,1.0,1.0 +char_trigram,q079,Search GitHub code,2,1,191,1.0,1.0,1.0,1,296,1.0,1.0,1.0,1,719,1.0,1.0,1.0 +char_trigram,q080,List GitHub notifications,1,1,176,1.0,1.0,1.0,1,295,1.0,1.0,1.0,1,606,1.0,1.0,1.0 +char_trigram,q081,Assign Copilot to an issue,1,1,267,1.0,1.0,1.0,1,467,1.0,1.0,1.0,1,883,1.0,1.0,1.0 +char_trigram,q082,Get secret scanning alerts,1,1,294,1.0,1.0,1.0,1,483,1.0,1.0,1.0,1,777,1.0,1.0,1.0 +char_trigram,q083,Create or update a file on GitHub,2,1,318,1.0,1.0,1.0,1,469,1.0,1.0,1.0,1,881,1.0,1.0,1.0 +char_trigram,q084,List collaborators on a repo,1,1,271,1.0,1.0,1.0,1,413,1.0,1.0,1.0,1,813,1.0,1.0,1.0 +char_trigram,q085,Reply in a Slack thread,2,1,242,1.0,1.0,1.0,1,347,1.0,1.0,1.0,1,621,1.0,1.0,1.0 +char_trigram,q086,Add an emoji reaction on Slack,1,1,302,1.0,1.0,1.0,1,496,1.0,1.0,1.0,1,919,1.0,1.0,1.0 +char_trigram,q087,Search Slack messages,1,1,232,1.0,1.0,1.0,1,392,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +char_trigram,q088,Create a Slack channel,1,1,196,1.0,1.0,1.0,1,339,1.0,1.0,1.0,1,801,1.0,1.0,1.0 +char_trigram,q089,Transition a Jira issue,1,1,208,1.0,1.0,1.0,1,391,1.0,1.0,1.0,1,740,1.0,1.0,1.0 +char_trigram,q090,Search Jira with JQL,1,1,196,1.0,1.0,1.0,1,301,1.0,1.0,1.0,1,627,1.0,1.0,1.0 +char_trigram,q091,Add a worklog to Jira,1,1,233,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,645,1.0,1.0,1.0 +char_trigram,q092,List Linear teams,1,1,184,1.0,1.0,1.0,1,330,1.0,1.0,1.0,1,686,1.0,1.0,1.0 +char_trigram,q093,Comment on a Linear issue,1,1,190,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,831,1.0,1.0,1.0 +char_trigram,q094,Move a local file,5,2,422,0.0,1.0,1.0,1,335,1.0,1.0,1.0,1,689,1.0,1.0,1.0 +char_trigram,q095,Recursively search files by pattern,1,1,200,1.0,1.0,1.0,1,324,1.0,1.0,1.0,1,658,1.0,1.0,1.0 +char_trigram,q096,Edit a file in place with old and new string,1,1,224,1.0,1.0,1.0,1,350,1.0,1.0,1.0,1,733,1.0,1.0,1.0 +char_trigram,q097,Read the whole knowledge graph,1,1,152,1.0,1.0,1.0,1,253,1.0,1.0,1.0,1,574,1.0,1.0,1.0 +char_trigram,q098,Search memory nodes,1,1,195,1.0,1.0,1.0,1,319,1.0,1.0,1.0,1,621,1.0,1.0,1.0 +char_trigram,q099,Show unstaged git diff,1,1,174,1.0,1.0,1.0,1,299,1.0,1.0,1.0,1,614,1.0,1.0,1.0 +char_trigram,q100,Stage files with git add,1,1,209,1.0,1.0,1.0,1,349,1.0,1.0,1.0,1,745,1.0,1.0,1.0 +char_trigram,q101,Create a Shopify product,1,1,240,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,724,1.0,1.0,1.0 +char_trigram,q102,Cancel a Stripe subscription,1,1,203,1.0,1.0,1.0,1,361,1.0,1.0,1.0,1,724,1.0,1.0,1.0 +char_trigram,q103,Create a payment intent,2,1,234,1.0,1.0,1.0,1,439,1.0,1.0,1.0,1,931,1.0,1.0,1.0 +char_trigram,q104,Acknowledge a PagerDuty incident,1,1,223,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,665,1.0,1.0,1.0 +char_trigram,q105,Create a Datadog monitor,1,1,259,1.0,1.0,1.0,1,448,1.0,1.0,1.0,1,805,1.0,1.0,1.0 +char_trigram,q106,Resolve a Sentry issue,2,1,251,1.0,1.0,1.0,1,412,1.0,1.0,1.0,1,848,1.0,1.0,1.0 +char_trigram,q107,Scale a Kubernetes deployment,1,1,262,1.0,1.0,1.0,1,400,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +char_trigram,q108,Apply a Kubernetes manifest,1,1,222,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,713,1.0,1.0,1.0 +char_trigram,q109,Stop a docker container,1,1,188,1.0,1.0,1.0,1,327,1.0,1.0,1.0,1,655,1.0,1.0,1.0 +char_trigram,q110,Build a docker image,1,1,211,1.0,1.0,1.0,1,317,1.0,1.0,1.0,1,598,1.0,1.0,1.0 +char_trigram,q111,Click an element in Playwright,1,1,216,1.0,1.0,1.0,1,339,1.0,1.0,1.0,1,566,1.0,1.0,1.0 +char_trigram,q112,Type text into a browser input,1,1,233,1.0,1.0,1.0,1,333,1.0,1.0,1.0,1,629,1.0,1.0,1.0 +char_trigram,q113,List GitLab projects,1,1,209,1.0,1.0,1.0,1,328,1.0,1.0,1.0,1,609,1.0,1.0,1.0 +char_trigram,q114,Create a Bitbucket pull request,1,1,313,1.0,1.0,1.0,1,489,1.0,1.0,1.0,1,1072,1.0,1.0,1.0 +char_trigram,q115,Create a Gmail draft,1,1,238,1.0,1.0,1.0,1,431,1.0,1.0,1.0,1,780,1.0,1.0,1.0 +char_trigram,q116,Share a Google Drive file,1,1,254,1.0,1.0,1.0,1,417,1.0,1.0,1.0,1,753,1.0,1.0,1.0 +char_trigram,q117,List Google Calendar events,1,1,219,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,757,1.0,1.0,1.0 +char_trigram,q118,Delete a calendar event,1,1,231,1.0,1.0,1.0,1,404,1.0,1.0,1.0,1,704,1.0,1.0,1.0 +char_trigram,q119,List Vercel projects,1,1,163,1.0,1.0,1.0,1,255,1.0,1.0,1.0,1,530,1.0,1.0,1.0 +char_trigram,q120,Select rows from Supabase,1,1,237,1.0,1.0,1.0,1,351,1.0,1.0,1.0,1,670,1.0,1.0,1.0 +char_trigram,q121,Post a Figma comment,1,1,201,1.0,1.0,1.0,1,319,1.0,1.0,1.0,1,612,1.0,1.0,1.0 +char_trigram,q122,List Okta users,1,1,211,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,607,1.0,1.0,1.0 +char_trigram,q123,Range query Prometheus,1,1,221,1.0,1.0,1.0,1,347,1.0,1.0,1.0,1,776,1.0,1.0,1.0 +char_trigram,q124,Create a Grafana annotation,1,1,275,1.0,1.0,1.0,1,432,1.0,1.0,1.0,1,893,1.0,1.0,1.0 +char_trigram,q125,Apply terraform,1,1,171,1.0,1.0,1.0,1,327,1.0,1.0,1.0,1,644,1.0,1.0,1.0 +char_trigram,q126,List Asana tasks,1,1,192,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,631,1.0,1.0,1.0 +char_trigram,q127,Comment on a Trello card,1,1,260,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,712,1.0,1.0,1.0 +char_trigram,q128,Update a ClickUp task,1,1,257,1.0,1.0,1.0,1,399,1.0,1.0,1.0,1,760,1.0,1.0,1.0 +char_trigram,q129,Search Confluence,1,1,221,1.0,1.0,1.0,1,369,1.0,1.0,1.0,1,660,1.0,1.0,1.0 +char_trigram,q130,Create a HubSpot deal,1,1,255,1.0,1.0,1.0,1,428,1.0,1.0,1.0,1,805,1.0,1.0,1.0 +char_trigram,q131,List Zendesk tickets,1,1,189,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,593,1.0,1.0,1.0 +char_trigram,q132,List Discord channels,1,1,184,1.0,1.0,1.0,1,321,1.0,1.0,1.0,1,598,1.0,1.0,1.0 +char_trigram,q133,Set a Redis key,2,1,182,1.0,1.0,1.0,1,284,1.0,1.0,1.0,1,550,1.0,1.0,1.0 +char_trigram,q134,Insert MongoDB documents,1,1,258,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,751,1.0,1.0,1.0 +char_trigram,q135,Run a MySQL query,1,1,179,1.0,1.0,1.0,1,294,1.0,1.0,1.0,1,554,1.0,1.0,1.0 +char_trigram,q136,Index a document in Elasticsearch,1,1,242,1.0,1.0,1.0,1,369,1.0,1.0,1.0,1,638,1.0,1.0,1.0 +char_trigram,q137,Get an S3 object,1,1,234,1.0,1.0,1.0,1,397,1.0,1.0,1.0,1,696,1.0,1.0,1.0 +char_trigram,q138,List Cloudflare zones,1,1,182,1.0,1.0,1.0,1,366,1.0,1.0,1.0,1,711,1.0,1.0,1.0 +char_trigram,q139,Create a Stripe invoice,1,1,216,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,762,1.0,1.0,1.0 +char_trigram,q140,List GitHub Actions workflows,1,1,293,1.0,1.0,1.0,1,459,1.0,1.0,1.0,1,763,1.0,1.0,1.0 +char_trigram,q141,Get GitHub Actions job logs,1,1,269,1.0,1.0,1.0,1,374,1.0,1.0,1.0,1,645,1.0,1.0,1.0 +char_trigram,q142,Create a GitHub repository,3,1,268,1.0,1.0,1.0,1,406,1.0,1.0,1.0,1,844,1.0,1.0,1.0 +char_trigram,q143,Checkout a git branch,1,1,200,1.0,1.0,1.0,1,384,1.0,1.0,1.0,1,814,1.0,1.0,1.0 +char_trigram,q144,Show git log,1,1,193,1.0,1.0,1.0,1,311,1.0,1.0,1.0,1,646,1.0,1.0,1.0 +char_trigram,q145,List postgres schemas,1,1,152,1.0,1.0,1.0,1,269,1.0,1.0,1.0,1,525,1.0,1.0,1.0 +char_trigram,q146,Open nodes in the knowledge graph,1,1,177,1.0,1.0,1.0,1,262,1.0,1.0,1.0,1,525,1.0,1.0,1.0 +char_trigram,q147,Get file metadata from the filesystem,1,1,220,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,635,1.0,1.0,1.0 +char_trigram,q148,Write a new local file,1,1,234,1.0,1.0,1.0,1,342,1.0,1.0,1.0,1,740,1.0,1.0,1.0 +char_trigram,q149,Search GitHub issues,1,1,209,1.0,1.0,1.0,1,333,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +char_trigram,q150,Get the authenticated GitHub user,2,1,179,1.0,1.0,1.0,1,300,1.0,1.0,1.0,1,611,1.0,1.0,1.0 +codedb_find,q001,Create a GitHub issue,1,1,326,1.0,1.0,1.0,1,521,1.0,1.0,1.0,1,1013,1.0,1.0,1.0 +codedb_find,q002,Create a Jira ticket,7,3,963,0.0,0.0,1.0,2,1040,0.0,1.0,1.0,1,1020,1.0,1.0,1.0 +codedb_find,q003,Send a Slack message,130,44,11040,0.0,0.0,0.0,26,10501,0.0,0.0,0.0,13,10241,0.0,0.0,0.0 +codedb_find,q004,List open pull requests,47,16,4347,0.0,0.0,0.0,10,4459,0.0,0.0,0.0,5,4359,0.0,0.0,0.0 +codedb_find,q005,Open a pull request on GitHub,36,12,3495,0.0,0.0,0.0,8,3776,0.0,0.0,0.0,4,3684,0.0,0.0,0.0 +codedb_find,q006,Run a SQL query against postgres,103,35,8997,0.0,0.0,0.0,21,8689,0.0,0.0,0.0,11,8745,0.0,0.0,0.0 +codedb_find,q007,Read a local file from disk,77,26,7264,0.0,0.0,0.0,16,7172,0.0,0.0,0.0,8,6996,0.0,0.0,0.0 +codedb_find,q008,What files are in this folder,82,28,7711,0.0,0.0,0.0,17,7530,0.0,0.0,0.0,9,7616,0.0,0.0,0.0 +codedb_find,q009,Remember this fact in the knowledge graph,89,30,8108,0.0,0.0,0.0,18,7832,0.0,0.0,0.0,9,7625,0.0,0.0,0.0 +codedb_find,q010,Search the web with Brave,124,42,10467,0.0,0.0,0.0,25,10040,0.0,0.0,0.0,13,10187,0.0,0.0,0.0 +codedb_find,q011,Fetch the contents of a URL,110,37,9458,0.0,0.0,0.0,22,9076,0.0,0.0,0.0,11,8834,0.0,0.0,0.0 +codedb_find,q012,Show git status,112,38,9466,0.0,0.0,0.0,23,9245,0.0,0.0,0.0,12,9359,0.0,0.0,0.0 +codedb_find,q013,Commit staged changes,110,37,9257,0.0,0.0,0.0,22,8907,0.0,0.0,0.0,11,8698,0.0,0.0,0.0 +codedb_find,q014,Create a Linear issue,1,1,303,1.0,1.0,1.0,1,461,1.0,1.0,1.0,1,820,1.0,1.0,1.0 +codedb_find,q015,Make a task in my project tracker Linear,136,46,11712,0.0,0.0,0.0,28,11391,0.0,0.0,0.0,14,11055,0.0,0.0,0.0 +codedb_find,q016,Post a comment on a GitHub issue,38,13,3791,0.0,0.0,0.0,8,3759,0.0,0.0,0.0,4,3663,0.0,0.0,0.0 +codedb_find,q017,Merge the pull request,1,1,365,1.0,1.0,1.0,1,560,1.0,1.0,1.0,1,988,1.0,1.0,1.0 +codedb_find,q018,Trigger a GitHub Actions workflow,62,21,6123,0.0,0.0,0.0,13,6050,0.0,0.0,0.0,7,6293,0.0,0.0,0.0 +codedb_find,q019,Get Dependabot alerts,1,1,240,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,671,1.0,1.0,1.0 +codedb_find,q020,List Sentry issues,1,1,260,1.0,1.0,1.0,1,363,1.0,1.0,1.0,1,716,1.0,1.0,1.0 +codedb_find,q021,Create a Stripe customer,1,1,293,1.0,1.0,1.0,1,450,1.0,1.0,1.0,1,892,1.0,1.0,1.0 +codedb_find,q022,Refund a Stripe payment,217,73,17304,0.0,0.0,0.0,44,16814,0.0,0.0,0.0,22,16374,0.0,0.0,0.0 +codedb_find,q023,Send an email with Gmail,245,82,19583,0.0,0.0,0.0,49,18819,0.0,0.0,0.0,25,18648,0.0,0.0,0.0 +codedb_find,q024,Search my Gmail inbox,245,82,19506,0.0,0.0,0.0,49,18792,0.0,0.0,0.0,25,18630,0.0,0.0,0.0 +codedb_find,q025,Upload a file to Google Drive,253,85,20366,0.0,0.0,0.0,51,19618,0.0,0.0,0.0,26,19523,0.0,0.0,0.0 +codedb_find,q026,Create a calendar event,1,1,324,1.0,1.0,1.0,1,484,1.0,1.0,1.0,1,984,1.0,1.0,1.0 +codedb_find,q027,List Kubernetes pods,1,1,198,1.0,1.0,1.0,1,299,1.0,1.0,1.0,1,630,1.0,1.0,1.0 +codedb_find,q028,Get logs from a k8s pod,193,65,15818,0.0,0.0,0.0,39,15194,0.0,0.0,0.0,20,15064,0.0,0.0,0.0 +codedb_find,q029,Run a docker container,194,65,15640,0.0,0.0,0.0,39,15120,0.0,0.0,0.0,20,15019,0.0,0.0,0.0 +codedb_find,q030,Show docker container logs,184,62,14888,0.0,0.0,0.0,37,14342,0.0,0.0,0.0,19,14250,0.0,0.0,0.0 +codedb_find,q031,Navigate the browser to a URL,171,57,13984,0.0,0.0,0.0,35,13740,0.0,0.0,0.0,18,13672,0.0,0.0,0.0 +codedb_find,q032,Take a screenshot of the page,171,57,14038,0.0,0.0,0.0,35,13740,0.0,0.0,0.0,18,13672,0.0,0.0,0.0 +codedb_find,q033,Create a GitLab merge request,1,1,397,1.0,1.0,1.0,1,592,1.0,1.0,1.0,1,930,1.0,1.0,1.0 +codedb_find,q034,Search Notion pages,165,55,13537,0.0,0.0,0.0,33,13097,0.0,0.0,0.0,17,13110,0.0,0.0,0.0 +codedb_find,q035,Create a Notion page,1,1,244,1.0,1.0,1.0,1,430,1.0,1.0,1.0,1,773,1.0,1.0,1.0 +codedb_find,q036,Query a Notion database,1,1,233,1.0,1.0,1.0,1,416,1.0,1.0,1.0,1,752,1.0,1.0,1.0 +codedb_find,q037,Create a Zendesk ticket,1,1,252,1.0,1.0,1.0,1,428,1.0,1.0,1.0,1,756,1.0,1.0,1.0 +codedb_find,q038,Create a HubSpot contact,1,1,301,1.0,1.0,1.0,1,508,1.0,1.0,1.0,1,832,1.0,1.0,1.0 +codedb_find,q039,Create a Salesforce lead,1,1,252,1.0,1.0,1.0,1,445,1.0,1.0,1.0,1,895,1.0,1.0,1.0 +codedb_find,q040,Run a SOQL query,2,1,242,1.0,1.0,1.0,1,398,1.0,1.0,1.0,1,837,1.0,1.0,1.0 +codedb_find,q041,List Shopify orders,1,1,210,1.0,1.0,1.0,1,333,1.0,1.0,1.0,1,692,1.0,1.0,1.0 +codedb_find,q042,Create a PagerDuty incident,1,1,267,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,823,1.0,1.0,1.0 +codedb_find,q043,Who is on call in PagerDuty,243,81,19511,0.0,0.0,0.0,49,18939,0.0,0.0,0.0,25,18705,0.0,0.0,0.0 +codedb_find,q044,Query Datadog metrics,1,1,277,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,835,1.0,1.0,1.0 +codedb_find,q045,Search Datadog logs,241,81,19283,0.0,0.0,0.0,49,18785,0.0,0.0,0.0,25,18646,0.0,0.0,0.0 +codedb_find,q046,List S3 buckets,1,1,153,1.0,1.0,1.0,1,311,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +codedb_find,q047,Upload an object to S3,296,99,23677,0.0,0.0,0.0,60,23014,0.0,0.0,0.0,30,22354,0.0,0.0,0.0 +codedb_find,q048,Purge Cloudflare cache,1,1,231,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,844,1.0,1.0,1.0 +codedb_find,q049,Create a Cloudflare DNS record,1,1,217,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,642,1.0,1.0,1.0 +codedb_find,q050,List Vercel deployments,1,1,208,1.0,1.0,1.0,1,336,1.0,1.0,1.0,1,647,1.0,1.0,1.0 +codedb_find,q051,Insert a row into Supabase,309,103,24489,0.0,0.0,0.0,62,23669,0.0,0.0,0.0,31,22987,0.0,0.0,0.0 +codedb_find,q052,Export images from Figma,316,106,25071,0.0,0.0,0.0,64,24328,0.0,0.0,0.0,32,23656,0.0,0.0,0.0 +codedb_find,q053,Send an SMS with Twilio,316,106,25217,0.0,0.0,0.0,64,24398,0.0,0.0,0.0,32,23694,0.0,0.0,0.0 +codedb_find,q054,Create an Okta user,325,109,25800,0.0,0.0,0.0,65,24768,0.0,0.0,0.0,33,24366,0.0,0.0,0.0 +codedb_find,q055,Instant query Prometheus,329,110,25880,0.0,0.0,0.0,66,25000,0.0,0.0,0.0,33,24340,0.0,0.0,0.0 +codedb_find,q056,Search Grafana dashboards,1,1,225,1.0,1.0,1.0,1,342,1.0,1.0,1.0,1,699,1.0,1.0,1.0 +codedb_find,q057,Trigger a CircleCI pipeline,332,111,26129,0.0,0.0,0.0,67,25314,0.0,0.0,0.0,34,25005,0.0,0.0,0.0 +codedb_find,q058,Run terraform plan,1,1,227,1.0,1.0,1.0,1,365,1.0,1.0,1.0,1,645,1.0,1.0,1.0 +codedb_find,q059,Create an Asana task,1,1,337,1.0,1.0,1.0,1,467,1.0,1.0,1.0,1,843,1.0,1.0,1.0 +codedb_find,q060,Create a Trello card,1,1,245,1.0,1.0,1.0,1,394,1.0,1.0,1.0,1,895,1.0,1.0,1.0 +codedb_find,q061,Create a ClickUp task,1,1,304,1.0,1.0,1.0,1,467,1.0,1.0,1.0,1,854,1.0,1.0,1.0 +codedb_find,q062,Create a Confluence page,1,1,260,1.0,1.0,1.0,1,450,1.0,1.0,1.0,1,889,1.0,1.0,1.0 +codedb_find,q063,Send a Discord message,1,1,218,1.0,1.0,1.0,1,371,1.0,1.0,1.0,1,752,1.0,1.0,1.0 +codedb_find,q064,Search Elasticsearch,1,1,239,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,700,1.0,1.0,1.0 +codedb_find,q065,Get a Redis key,291,97,22977,0.0,0.0,0.0,59,22447,0.0,0.0,0.0,30,22206,0.0,0.0,0.0 +codedb_find,q066,Find MongoDB documents,281,94,22397,0.0,0.0,0.0,57,21868,0.0,0.0,0.0,29,21583,0.0,0.0,0.0 +codedb_find,q067,List SQLite tables,3,1,151,1.0,1.0,1.0,1,302,1.0,1.0,1.0,1,651,1.0,1.0,1.0 +codedb_find,q068,Describe a postgres table,1,1,212,1.0,1.0,1.0,1,358,1.0,1.0,1.0,1,677,1.0,1.0,1.0 +codedb_find,q069,github_create_issue,1,1,324,1.0,1.0,1.0,1,520,1.0,1.0,1.0,1,998,1.0,1.0,1.0 +codedb_find,q070,createIssue,1,1,397,1.0,1.0,1.0,1,579,1.0,1.0,1.0,1,1039,1.0,1.0,1.0 +codedb_find,q071,list-directory,1,1,186,1.0,1.0,1.0,1,281,1.0,1.0,1.0,1,599,1.0,1.0,1.0 +codedb_find,q072,sequential thinking next thought,116,39,9799,0.0,0.0,0.0,24,9727,0.0,0.0,0.0,12,9475,0.0,0.0,0.0 +codedb_find,q073,What time is it in Tokyo,119,40,10114,0.0,0.0,0.0,24,9762,0.0,0.0,0.0,12,9498,0.0,0.0,0.0 +codedb_find,q074,Convert time between timezones,122,41,10273,0.0,0.0,0.0,25,10083,0.0,0.0,0.0,13,10227,0.0,0.0,0.0 +codedb_find,q075,Star a GitHub repo,1,1,227,1.0,1.0,1.0,1,349,1.0,1.0,1.0,1,796,1.0,1.0,1.0 +codedb_find,q076,Fork a repository,30,10,2448,0.0,0.0,0.0,6,2372,0.0,0.0,0.0,3,2315,0.0,0.0,1.0 +codedb_find,q077,Get the latest GitHub release,1,1,219,1.0,1.0,1.0,1,416,1.0,1.0,1.0,1,810,1.0,1.0,1.0 +codedb_find,q078,Create a gist,1,1,290,1.0,1.0,1.0,1,496,1.0,1.0,1.0,1,1021,1.0,1.0,1.0 +codedb_find,q079,Search GitHub code,1,1,277,1.0,1.0,1.0,1,369,1.0,1.0,1.0,1,783,1.0,1.0,1.0 +codedb_find,q080,List GitHub notifications,1,1,175,1.0,1.0,1.0,1,325,1.0,1.0,1.0,1,643,1.0,1.0,1.0 +codedb_find,q081,Assign Copilot to an issue,1,1,312,1.0,1.0,1.0,1,418,1.0,1.0,1.0,1,824,1.0,1.0,1.0 +codedb_find,q082,Get secret scanning alerts,1,1,295,1.0,1.0,1.0,1,465,1.0,1.0,1.0,1,884,1.0,1.0,1.0 +codedb_find,q083,Create or update a file on GitHub,20,7,1850,0.0,0.0,0.0,4,1698,0.0,0.0,0.0,2,1650,0.0,1.0,1.0 +codedb_find,q084,List collaborators on a repo,33,11,2773,0.0,0.0,0.0,7,2927,0.0,0.0,0.0,4,3392,0.0,0.0,0.0 +codedb_find,q085,Reply in a Slack thread,130,44,11132,0.0,0.0,0.0,26,10591,0.0,0.0,0.0,13,10318,0.0,0.0,0.0 +codedb_find,q086,Add an emoji reaction on Slack,125,42,10605,0.0,0.0,0.0,25,10158,0.0,0.0,0.0,13,10233,0.0,0.0,0.0 +codedb_find,q087,Search Slack messages,1,1,211,1.0,1.0,1.0,1,349,1.0,1.0,1.0,1,742,1.0,1.0,1.0 +codedb_find,q088,Create a Slack channel,1,1,211,1.0,1.0,1.0,1,357,1.0,1.0,1.0,1,686,1.0,1.0,1.0 +codedb_find,q089,Transition a Jira issue,1,1,240,1.0,1.0,1.0,1,355,1.0,1.0,1.0,1,740,1.0,1.0,1.0 +codedb_find,q090,Search Jira with JQL,151,51,12663,0.0,0.0,0.0,31,12344,0.0,0.0,0.0,16,12438,0.0,0.0,0.0 +codedb_find,q091,Add a worklog to Jira,155,52,12978,0.0,0.0,0.0,31,12447,0.0,0.0,0.0,16,12448,0.0,0.0,0.0 +codedb_find,q092,List Linear teams,1,1,199,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,684,1.0,1.0,1.0 +codedb_find,q093,Comment on a Linear issue,146,49,12189,0.0,0.0,0.0,30,11992,0.0,0.0,0.0,15,11677,0.0,0.0,0.0 +codedb_find,q094,Move a local file,95,32,8648,0.0,0.0,0.0,19,8320,0.0,0.0,0.0,10,8418,0.0,0.0,0.0 +codedb_find,q095,Recursively search files by pattern,88,30,8094,0.0,0.0,0.0,18,7830,0.0,0.0,0.0,9,7632,0.0,0.0,0.0 +codedb_find,q096,Edit a file in place with old and new string,81,27,7623,0.0,0.0,0.0,17,7586,0.0,0.0,0.0,9,7650,0.0,0.0,0.0 +codedb_find,q097,Read the whole knowledge graph,105,35,9234,0.0,0.0,0.0,21,8940,0.0,0.0,0.0,11,9003,0.0,0.0,0.0 +codedb_find,q098,Search memory nodes,1,1,221,1.0,1.0,1.0,1,348,1.0,1.0,1.0,1,728,1.0,1.0,1.0 +codedb_find,q099,Show unstaged git diff,105,35,8947,0.0,0.0,0.0,21,8653,0.0,0.0,0.0,11,8735,0.0,0.0,0.0 +codedb_find,q100,Stage files with git add,111,37,9333,0.0,0.0,0.0,23,9295,0.0,0.0,0.0,12,9410,0.0,0.0,0.0 +codedb_find,q101,Create a Shopify product,1,1,286,1.0,1.0,1.0,1,414,1.0,1.0,1.0,1,820,1.0,1.0,1.0 +codedb_find,q102,Cancel a Stripe subscription,1,1,208,1.0,1.0,1.0,1,400,1.0,1.0,1.0,1,809,1.0,1.0,1.0 +codedb_find,q103,Create a payment intent,3,1,299,1.0,1.0,1.0,1,489,1.0,1.0,1.0,1,779,1.0,1.0,1.0 +codedb_find,q104,Acknowledge a PagerDuty incident,1,1,185,1.0,1.0,1.0,1,343,1.0,1.0,1.0,1,758,1.0,1.0,1.0 +codedb_find,q105,Create a Datadog monitor,1,1,296,1.0,1.0,1.0,1,423,1.0,1.0,1.0,1,774,1.0,1.0,1.0 +codedb_find,q106,Resolve a Sentry issue,232,78,18609,0.0,0.0,0.0,47,18021,0.0,0.0,0.0,24,17953,0.0,0.0,0.0 +codedb_find,q107,Scale a Kubernetes deployment,1,1,269,1.0,1.0,1.0,1,395,1.0,1.0,1.0,1,804,1.0,1.0,1.0 +codedb_find,q108,Apply a Kubernetes manifest,200,67,16099,0.0,0.0,0.0,40,15471,0.0,0.0,0.0,20,15071,0.0,0.0,0.0 +codedb_find,q109,Stop a docker container,196,66,15806,0.0,0.0,0.0,40,15415,0.0,0.0,0.0,20,15015,0.0,0.0,0.0 +codedb_find,q110,Build a docker image,195,65,15591,0.0,0.0,0.0,39,15071,0.0,0.0,0.0,20,15055,0.0,0.0,0.0 +codedb_find,q111,Click an element in Playwright,175,59,14528,0.0,0.0,0.0,35,13874,0.0,0.0,0.0,18,13763,0.0,0.0,0.0 +codedb_find,q112,Type text into a browser input,169,57,14038,0.0,0.0,0.0,34,13458,0.0,0.0,0.0,17,13084,0.0,0.0,0.0 +codedb_find,q113,List GitLab projects,1,1,204,1.0,1.0,1.0,1,337,1.0,1.0,1.0,1,653,1.0,1.0,1.0 +codedb_find,q114,Create a Bitbucket pull request,272,91,21988,0.0,0.0,0.0,55,21340,0.0,0.0,0.0,28,20983,0.0,0.0,0.0 +codedb_find,q115,Create a Gmail draft,1,1,294,1.0,1.0,1.0,1,494,1.0,1.0,1.0,1,1037,1.0,1.0,1.0 +codedb_find,q116,Share a Google Drive file,1,1,316,1.0,1.0,1.0,1,409,1.0,1.0,1.0,1,733,1.0,1.0,1.0 +codedb_find,q117,List Google Calendar events,1,1,202,1.0,1.0,1.0,1,366,1.0,1.0,1.0,1,816,1.0,1.0,1.0 +codedb_find,q118,Delete a calendar event,1,1,301,1.0,1.0,1.0,1,440,1.0,1.0,1.0,1,742,1.0,1.0,1.0 +codedb_find,q119,List Vercel projects,1,1,156,1.0,1.0,1.0,1,296,1.0,1.0,1.0,1,614,1.0,1.0,1.0 +codedb_find,q120,Select rows from Supabase,307,103,24397,0.0,0.0,0.0,62,23613,0.0,0.0,0.0,31,22962,0.0,0.0,0.0 +codedb_find,q121,Post a Figma comment,1,1,289,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,817,1.0,1.0,1.0 +codedb_find,q122,List Okta users,1,1,233,1.0,1.0,1.0,1,361,1.0,1.0,1.0,1,656,1.0,1.0,1.0 +codedb_find,q123,Range query Prometheus,1,1,266,1.0,1.0,1.0,1,472,1.0,1.0,1.0,1,835,1.0,1.0,1.0 +codedb_find,q124,Create a Grafana annotation,1,1,197,1.0,1.0,1.0,1,371,1.0,1.0,1.0,1,728,1.0,1.0,1.0 +codedb_find,q125,Apply terraform,1,1,171,1.0,1.0,1.0,1,287,1.0,1.0,1.0,1,681,1.0,1.0,1.0 +codedb_find,q126,List Asana tasks,1,1,250,1.0,1.0,1.0,1,395,1.0,1.0,1.0,1,767,1.0,1.0,1.0 +codedb_find,q127,Comment on a Trello card,345,115,27183,0.0,0.0,0.0,69,26171,0.0,0.0,0.0,35,25769,0.0,0.0,0.0 +codedb_find,q128,Update a ClickUp task,1,1,277,1.0,1.0,1.0,1,416,1.0,1.0,1.0,1,840,1.0,1.0,1.0 +codedb_find,q129,Search Confluence,1,1,223,1.0,1.0,1.0,1,350,1.0,1.0,1.0,1,757,1.0,1.0,1.0 +codedb_find,q130,Create a HubSpot deal,1,1,295,1.0,1.0,1.0,1,536,1.0,1.0,1.0,1,982,1.0,1.0,1.0 +codedb_find,q131,List Zendesk tickets,1,1,204,1.0,1.0,1.0,1,316,1.0,1.0,1.0,1,644,1.0,1.0,1.0 +codedb_find,q132,List Discord channels,1,1,215,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,596,1.0,1.0,1.0 +codedb_find,q133,Set a Redis key,293,98,23335,0.0,0.0,0.0,59,22602,0.0,0.0,0.0,30,22356,0.0,0.0,0.0 +codedb_find,q134,Insert MongoDB documents,283,95,22628,0.0,0.0,0.0,57,21868,0.0,0.0,0.0,29,21583,0.0,0.0,0.0 +codedb_find,q135,Run a MySQL query,280,94,22516,0.0,0.0,0.0,56,21595,0.0,0.0,0.0,28,21007,0.0,0.0,0.0 +codedb_find,q136,Index a document in Elasticsearch,291,97,23125,0.0,0.0,0.0,59,22581,0.0,0.0,0.0,30,22324,0.0,0.0,0.0 +codedb_find,q137,Get an S3 object,295,99,23578,0.0,0.0,0.0,59,22581,0.0,0.0,0.0,30,22324,0.0,0.0,0.0 +codedb_find,q138,List Cloudflare zones,1,1,161,1.0,1.0,1.0,1,321,1.0,1.0,1.0,1,613,1.0,1.0,1.0 +codedb_find,q139,Create a Stripe invoice,1,1,250,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,770,1.0,1.0,1.0 +codedb_find,q140,List GitHub Actions workflows,60,20,5782,0.0,0.0,0.0,12,5614,0.0,0.0,0.0,6,5488,0.0,0.0,0.0 +codedb_find,q141,Get GitHub Actions job logs,1,1,208,1.0,1.0,1.0,1,366,1.0,1.0,1.0,1,781,1.0,1.0,1.0 +codedb_find,q142,Create a GitHub repository,1,1,297,1.0,1.0,1.0,1,502,1.0,1.0,1.0,1,1026,1.0,1.0,1.0 +codedb_find,q143,Checkout a git branch,117,39,9806,0.0,0.0,0.0,24,9724,0.0,0.0,0.0,12,9484,0.0,0.0,0.0 +codedb_find,q144,Show git log,119,40,9928,0.0,0.0,0.0,24,9624,0.0,0.0,0.0,12,9396,0.0,0.0,0.0 +codedb_find,q145,List postgres schemas,1,1,207,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,668,1.0,1.0,1.0 +codedb_find,q146,Open nodes in the knowledge graph,100,34,8817,0.0,0.0,0.0,20,8407,0.0,0.0,0.0,10,8187,0.0,0.0,0.0 +codedb_find,q147,Get file metadata from the filesystem,90,30,8177,0.0,0.0,0.0,18,7913,0.0,0.0,0.0,9,7715,0.0,0.0,0.0 +codedb_find,q148,Write a new local file,88,30,8232,0.0,0.0,0.0,18,7980,0.0,0.0,0.0,9,7791,0.0,0.0,0.0 +codedb_find,q149,Search GitHub issues,1,1,208,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,712,1.0,1.0,1.0 +codedb_find,q150,Get the authenticated GitHub user,10,4,1056,0.0,0.0,0.0,2,907,0.0,1.0,1.0,1,885,1.0,1.0,1.0 +codedb_search,q001,Create a GitHub issue,1,1,335,1.0,1.0,1.0,1,582,1.0,1.0,1.0,1,1111,1.0,1.0,1.0 +codedb_search,q002,Create a Jira ticket,1,1,333,1.0,1.0,1.0,1,481,1.0,1.0,1.0,1,966,1.0,1.0,1.0 +codedb_search,q003,Send a Slack message,2,1,247,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +codedb_search,q004,List open pull requests,1,1,234,1.0,1.0,1.0,1,460,1.0,1.0,1.0,1,836,1.0,1.0,1.0 +codedb_search,q005,Open a pull request on GitHub,1,1,368,1.0,1.0,1.0,1,653,1.0,1.0,1.0,1,1144,1.0,1.0,1.0 +codedb_search,q006,Run a SQL query against postgres,1,1,180,1.0,1.0,1.0,1,295,1.0,1.0,1.0,1,553,1.0,1.0,1.0 +codedb_search,q007,Read a local file from disk,2,1,223,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,759,1.0,1.0,1.0 +codedb_search,q008,What files are in this folder,6,2,482,0.0,1.0,1.0,2,712,0.0,1.0,1.0,1,690,1.0,1.0,1.0 +codedb_search,q009,Remember this fact in the knowledge graph,2,1,156,1.0,1.0,1.0,1,262,1.0,1.0,1.0,1,493,1.0,1.0,1.0 +codedb_search,q010,Search the web with Brave,1,1,258,1.0,1.0,1.0,1,374,1.0,1.0,1.0,1,687,1.0,1.0,1.0 +codedb_search,q011,Fetch the contents of a URL,1,1,264,1.0,1.0,1.0,1,380,1.0,1.0,1.0,1,672,1.0,1.0,1.0 +codedb_search,q012,Show git status,1,1,186,1.0,1.0,1.0,1,299,1.0,1.0,1.0,1,684,1.0,1.0,1.0 +codedb_search,q013,Commit staged changes,1,1,182,1.0,1.0,1.0,1,327,1.0,1.0,1.0,1,596,1.0,1.0,1.0 +codedb_search,q014,Create a Linear issue,1,1,293,1.0,1.0,1.0,1,463,1.0,1.0,1.0,1,974,1.0,1.0,1.0 +codedb_search,q015,Make a task in my project tracker Linear,146,49,12453,0.0,0.0,0.0,30,12193,0.0,0.0,0.0,15,11833,0.0,0.0,0.0 +codedb_search,q016,Post a comment on a GitHub issue,3,1,307,1.0,1.0,1.0,1,510,1.0,1.0,1.0,1,1103,1.0,1.0,1.0 +codedb_search,q017,Merge the pull request,1,1,299,1.0,1.0,1.0,1,495,1.0,1.0,1.0,1,1121,1.0,1.0,1.0 +codedb_search,q018,Trigger a GitHub Actions workflow,1,1,294,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,733,1.0,1.0,1.0 +codedb_search,q019,Get Dependabot alerts,1,1,269,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,642,1.0,1.0,1.0 +codedb_search,q020,List Sentry issues,1,1,238,1.0,1.0,1.0,1,395,1.0,1.0,1.0,1,796,1.0,1.0,1.0 +codedb_search,q021,Create a Stripe customer,1,1,223,1.0,1.0,1.0,1,341,1.0,1.0,1.0,1,662,1.0,1.0,1.0 +codedb_search,q022,Refund a Stripe payment,1,1,234,1.0,1.0,1.0,1,332,1.0,1.0,1.0,1,629,1.0,1.0,1.0 +codedb_search,q023,Send an email with Gmail,1,1,224,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,716,1.0,1.0,1.0 +codedb_search,q024,Search my Gmail inbox,1,1,180,1.0,1.0,1.0,1,315,1.0,1.0,1.0,1,668,1.0,1.0,1.0 +codedb_search,q025,Upload a file to Google Drive,5,2,523,0.0,1.0,1.0,1,446,1.0,1.0,1.0,1,774,1.0,1.0,1.0 +codedb_search,q026,Create a calendar event,1,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,733,1.0,1.0,1.0 +codedb_search,q027,List Kubernetes pods,1,1,184,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,665,1.0,1.0,1.0 +codedb_search,q028,Get logs from a k8s pod,1,1,294,1.0,1.0,1.0,1,455,1.0,1.0,1.0,1,836,1.0,1.0,1.0 +codedb_search,q029,Run a docker container,1,1,223,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,597,1.0,1.0,1.0 +codedb_search,q030,Show docker container logs,1,1,188,1.0,1.0,1.0,1,350,1.0,1.0,1.0,1,759,1.0,1.0,1.0 +codedb_search,q031,Navigate the browser to a URL,1,1,180,1.0,1.0,1.0,1,282,1.0,1.0,1.0,1,603,1.0,1.0,1.0 +codedb_search,q032,Take a screenshot of the page,1,1,212,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,711,1.0,1.0,1.0 +codedb_search,q033,Create a GitLab merge request,1,1,297,1.0,1.0,1.0,1,475,1.0,1.0,1.0,1,922,1.0,1.0,1.0 +codedb_search,q034,Search Notion pages,1,1,180,1.0,1.0,1.0,1,292,1.0,1.0,1.0,1,558,1.0,1.0,1.0 +codedb_search,q035,Create a Notion page,1,1,230,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,822,1.0,1.0,1.0 +codedb_search,q036,Query a Notion database,1,1,279,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +codedb_search,q037,Create a Zendesk ticket,1,1,257,1.0,1.0,1.0,1,426,1.0,1.0,1.0,1,646,1.0,1.0,1.0 +codedb_search,q038,Create a HubSpot contact,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,899,1.0,1.0,1.0 +codedb_search,q039,Create a Salesforce lead,1,1,252,1.0,1.0,1.0,1,378,1.0,1.0,1.0,1,661,1.0,1.0,1.0 +codedb_search,q040,Run a SOQL query,1,1,187,1.0,1.0,1.0,1,293,1.0,1.0,1.0,1,729,1.0,1.0,1.0 +codedb_search,q041,List Shopify orders,1,1,208,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,606,1.0,1.0,1.0 +codedb_search,q042,Create a PagerDuty incident,1,1,259,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,628,1.0,1.0,1.0 +codedb_search,q043,Who is on call in PagerDuty,1,1,219,1.0,1.0,1.0,1,363,1.0,1.0,1.0,1,724,1.0,1.0,1.0 +codedb_search,q044,Query Datadog metrics,1,1,284,1.0,1.0,1.0,1,449,1.0,1.0,1.0,1,767,1.0,1.0,1.0 +codedb_search,q045,Search Datadog logs,1,1,242,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,821,1.0,1.0,1.0 +codedb_search,q046,List S3 buckets,1,1,203,1.0,1.0,1.0,1,361,1.0,1.0,1.0,1,523,1.0,1.0,1.0 +codedb_search,q047,Upload an object to S3,1,1,250,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,820,1.0,1.0,1.0 +codedb_search,q048,Purge Cloudflare cache,1,1,183,1.0,1.0,1.0,1,286,1.0,1.0,1.0,1,643,1.0,1.0,1.0 +codedb_search,q049,Create a Cloudflare DNS record,1,1,249,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,660,1.0,1.0,1.0 +codedb_search,q050,List Vercel deployments,1,1,173,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,449,1.0,1.0,1.0 +codedb_search,q051,Insert a row into Supabase,1,1,203,1.0,1.0,1.0,1,347,1.0,1.0,1.0,1,598,1.0,1.0,1.0 +codedb_search,q052,Export images from Figma,1,1,218,1.0,1.0,1.0,1,328,1.0,1.0,1.0,1,673,1.0,1.0,1.0 +codedb_search,q053,Send an SMS with Twilio,1,1,254,1.0,1.0,1.0,1,409,1.0,1.0,1.0,1,647,1.0,1.0,1.0 +codedb_search,q054,Create an Okta user,1,1,183,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,743,1.0,1.0,1.0 +codedb_search,q055,Instant query Prometheus,1,1,221,1.0,1.0,1.0,1,375,1.0,1.0,1.0,1,680,1.0,1.0,1.0 +codedb_search,q056,Search Grafana dashboards,1,1,217,1.0,1.0,1.0,1,324,1.0,1.0,1.0,1,605,1.0,1.0,1.0 +codedb_search,q057,Trigger a CircleCI pipeline,1,1,200,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,664,1.0,1.0,1.0 +codedb_search,q058,Run terraform plan,1,1,172,1.0,1.0,1.0,1,280,1.0,1.0,1.0,1,717,1.0,1.0,1.0 +codedb_search,q059,Create an Asana task,1,1,267,1.0,1.0,1.0,1,429,1.0,1.0,1.0,1,825,1.0,1.0,1.0 +codedb_search,q060,Create a Trello card,1,1,240,1.0,1.0,1.0,1,316,1.0,1.0,1.0,1,615,1.0,1.0,1.0 +codedb_search,q061,Create a ClickUp task,1,1,257,1.0,1.0,1.0,1,415,1.0,1.0,1.0,1,673,1.0,1.0,1.0 +codedb_search,q062,Create a Confluence page,1,1,260,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,654,1.0,1.0,1.0 +codedb_search,q063,Send a Discord message,1,1,262,1.0,1.0,1.0,1,404,1.0,1.0,1.0,1,628,1.0,1.0,1.0 +codedb_search,q064,Search Elasticsearch,1,1,239,1.0,1.0,1.0,1,348,1.0,1.0,1.0,1,617,1.0,1.0,1.0 +codedb_search,q065,Get a Redis key,1,1,190,1.0,1.0,1.0,1,281,1.0,1.0,1.0,1,626,1.0,1.0,1.0 +codedb_search,q066,Find MongoDB documents,1,1,244,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,631,1.0,1.0,1.0 +codedb_search,q067,List SQLite tables,1,1,142,1.0,1.0,1.0,1,240,1.0,1.0,1.0,1,417,1.0,1.0,1.0 +codedb_search,q068,Describe a postgres table,1,1,188,1.0,1.0,1.0,1,274,1.0,1.0,1.0,1,543,1.0,1.0,1.0 +codedb_search,q069,github_create_issue,29,10,2762,0.0,0.0,0.0,6,2686,0.0,0.0,0.0,3,2629,0.0,0.0,1.0 +codedb_search,q070,createIssue,29,10,2752,0.0,0.0,0.0,6,2680,0.0,0.0,0.0,3,2626,0.0,0.0,1.0 +codedb_search,q071,list-directory,82,28,7599,0.0,0.0,0.0,17,7462,0.0,0.0,0.0,9,7580,0.0,0.0,0.0 +codedb_search,q072,sequential thinking next thought,1,1,186,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,713,1.0,1.0,1.0 +codedb_search,q073,What time is it in Tokyo,1,1,222,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,633,1.0,1.0,1.0 +codedb_search,q074,Convert time between timezones,1,1,204,1.0,1.0,1.0,1,319,1.0,1.0,1.0,1,650,1.0,1.0,1.0 +codedb_search,q075,Star a GitHub repo,1,1,226,1.0,1.0,1.0,1,376,1.0,1.0,1.0,1,772,1.0,1.0,1.0 +codedb_search,q076,Fork a repository,1,1,239,1.0,1.0,1.0,1,384,1.0,1.0,1.0,1,783,1.0,1.0,1.0 +codedb_search,q077,Get the latest GitHub release,1,1,239,1.0,1.0,1.0,1,331,1.0,1.0,1.0,1,704,1.0,1.0,1.0 +codedb_search,q078,Create a gist,1,1,229,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,653,1.0,1.0,1.0 +codedb_search,q079,Search GitHub code,1,1,283,1.0,1.0,1.0,1,385,1.0,1.0,1.0,1,725,1.0,1.0,1.0 +codedb_search,q080,List GitHub notifications,1,1,189,1.0,1.0,1.0,1,307,1.0,1.0,1.0,1,655,1.0,1.0,1.0 +codedb_search,q081,Assign Copilot to an issue,1,1,267,1.0,1.0,1.0,1,402,1.0,1.0,1.0,1,875,1.0,1.0,1.0 +codedb_search,q082,Get secret scanning alerts,1,1,295,1.0,1.0,1.0,1,470,1.0,1.0,1.0,1,681,1.0,1.0,1.0 +codedb_search,q083,Create or update a file on GitHub,1,1,327,1.0,1.0,1.0,1,532,1.0,1.0,1.0,1,1080,1.0,1.0,1.0 +codedb_search,q084,List collaborators on a repo,1,1,264,1.0,1.0,1.0,1,416,1.0,1.0,1.0,1,776,1.0,1.0,1.0 +codedb_search,q085,Reply in a Slack thread,1,1,254,1.0,1.0,1.0,1,365,1.0,1.0,1.0,1,642,1.0,1.0,1.0 +codedb_search,q086,Add an emoji reaction on Slack,1,1,271,1.0,1.0,1.0,1,433,1.0,1.0,1.0,1,903,1.0,1.0,1.0 +codedb_search,q087,Search Slack messages,1,1,232,1.0,1.0,1.0,1,343,1.0,1.0,1.0,1,609,1.0,1.0,1.0 +codedb_search,q088,Create a Slack channel,1,1,196,1.0,1.0,1.0,1,342,1.0,1.0,1.0,1,751,1.0,1.0,1.0 +codedb_search,q089,Transition a Jira issue,1,1,208,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,778,1.0,1.0,1.0 +codedb_search,q090,Search Jira with JQL,1,1,211,1.0,1.0,1.0,1,318,1.0,1.0,1.0,1,599,1.0,1.0,1.0 +codedb_search,q091,Add a worklog to Jira,1,1,334,1.0,1.0,1.0,1,478,1.0,1.0,1.0,1,878,1.0,1.0,1.0 +codedb_search,q092,List Linear teams,1,1,184,1.0,1.0,1.0,1,295,1.0,1.0,1.0,1,646,1.0,1.0,1.0 +codedb_search,q093,Comment on a Linear issue,2,1,298,1.0,1.0,1.0,1,428,1.0,1.0,1.0,1,886,1.0,1.0,1.0 +codedb_search,q094,Move a local file,1,1,196,1.0,1.0,1.0,1,358,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +codedb_search,q095,Recursively search files by pattern,1,1,200,1.0,1.0,1.0,1,366,1.0,1.0,1.0,1,694,1.0,1.0,1.0 +codedb_search,q096,Edit a file in place with old and new string,1,1,247,1.0,1.0,1.0,1,358,1.0,1.0,1.0,1,736,1.0,1.0,1.0 +codedb_search,q097,Read the whole knowledge graph,1,1,170,1.0,1.0,1.0,1,305,1.0,1.0,1.0,1,562,1.0,1.0,1.0 +codedb_search,q098,Search memory nodes,1,1,202,1.0,1.0,1.0,1,290,1.0,1.0,1.0,1,546,1.0,1.0,1.0 +codedb_search,q099,Show unstaged git diff,1,1,190,1.0,1.0,1.0,1,313,1.0,1.0,1.0,1,557,1.0,1.0,1.0 +codedb_search,q100,Stage files with git add,1,1,250,1.0,1.0,1.0,1,519,1.0,1.0,1.0,1,923,1.0,1.0,1.0 +codedb_search,q101,Create a Shopify product,1,1,240,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +codedb_search,q102,Cancel a Stripe subscription,1,1,176,1.0,1.0,1.0,1,275,1.0,1.0,1.0,1,610,1.0,1.0,1.0 +codedb_search,q103,Create a payment intent,1,1,234,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,590,1.0,1.0,1.0 +codedb_search,q104,Acknowledge a PagerDuty incident,1,1,223,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,626,1.0,1.0,1.0 +codedb_search,q105,Create a Datadog monitor,1,1,259,1.0,1.0,1.0,1,459,1.0,1.0,1.0,1,674,1.0,1.0,1.0 +codedb_search,q106,Resolve a Sentry issue,2,1,251,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,721,1.0,1.0,1.0 +codedb_search,q107,Scale a Kubernetes deployment,1,1,234,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +codedb_search,q108,Apply a Kubernetes manifest,1,1,204,1.0,1.0,1.0,1,308,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +codedb_search,q109,Stop a docker container,1,1,188,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,587,1.0,1.0,1.0 +codedb_search,q110,Build a docker image,1,1,211,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,547,1.0,1.0,1.0 +codedb_search,q111,Click an element in Playwright,1,1,216,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,571,1.0,1.0,1.0 +codedb_search,q112,Type text into a browser input,1,1,252,1.0,1.0,1.0,1,337,1.0,1.0,1.0,1,589,1.0,1.0,1.0 +codedb_search,q113,List GitLab projects,1,1,216,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,595,1.0,1.0,1.0 +codedb_search,q114,Create a Bitbucket pull request,1,1,439,1.0,1.0,1.0,1,634,1.0,1.0,1.0,1,1155,1.0,1.0,1.0 +codedb_search,q115,Create a Gmail draft,1,1,284,1.0,1.0,1.0,1,411,1.0,1.0,1.0,1,709,1.0,1.0,1.0 +codedb_search,q116,Share a Google Drive file,1,1,254,1.0,1.0,1.0,1,417,1.0,1.0,1.0,1,760,1.0,1.0,1.0 +codedb_search,q117,List Google Calendar events,1,1,219,1.0,1.0,1.0,1,379,1.0,1.0,1.0,1,718,1.0,1.0,1.0 +codedb_search,q118,Delete a calendar event,1,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,675,1.0,1.0,1.0 +codedb_search,q119,List Vercel projects,1,1,151,1.0,1.0,1.0,1,253,1.0,1.0,1.0,1,550,1.0,1.0,1.0 +codedb_search,q120,Select rows from Supabase,1,1,237,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,697,1.0,1.0,1.0 +codedb_search,q121,Post a Figma comment,1,1,218,1.0,1.0,1.0,1,350,1.0,1.0,1.0,1,646,1.0,1.0,1.0 +codedb_search,q122,List Okta users,1,1,174,1.0,1.0,1.0,1,327,1.0,1.0,1.0,1,618,1.0,1.0,1.0 +codedb_search,q123,Range query Prometheus,1,1,221,1.0,1.0,1.0,1,375,1.0,1.0,1.0,1,680,1.0,1.0,1.0 +codedb_search,q124,Create a Grafana annotation,1,1,218,1.0,1.0,1.0,1,275,1.0,1.0,1.0,1,574,1.0,1.0,1.0 +codedb_search,q125,Apply terraform,1,1,190,1.0,1.0,1.0,1,238,1.0,1.0,1.0,1,595,1.0,1.0,1.0 +codedb_search,q126,List Asana tasks,1,1,192,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,566,1.0,1.0,1.0 +codedb_search,q127,Comment on a Trello card,1,1,260,1.0,1.0,1.0,1,438,1.0,1.0,1.0,1,703,1.0,1.0,1.0 +codedb_search,q128,Update a ClickUp task,1,1,257,1.0,1.0,1.0,1,399,1.0,1.0,1.0,1,673,1.0,1.0,1.0 +codedb_search,q129,Search Confluence,1,1,221,1.0,1.0,1.0,1,371,1.0,1.0,1.0,1,636,1.0,1.0,1.0 +codedb_search,q130,Create a HubSpot deal,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,899,1.0,1.0,1.0 +codedb_search,q131,List Zendesk tickets,1,1,189,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,529,1.0,1.0,1.0 +codedb_search,q132,List Discord channels,1,1,184,1.0,1.0,1.0,1,300,1.0,1.0,1.0,1,513,1.0,1.0,1.0 +codedb_search,q133,Set a Redis key,1,1,211,1.0,1.0,1.0,1,302,1.0,1.0,1.0,1,526,1.0,1.0,1.0 +codedb_search,q134,Insert MongoDB documents,1,1,244,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,607,1.0,1.0,1.0 +codedb_search,q135,Run a MySQL query,1,1,174,1.0,1.0,1.0,1,281,1.0,1.0,1.0,1,611,1.0,1.0,1.0 +codedb_search,q136,Index a document in Elasticsearch,1,1,242,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,538,1.0,1.0,1.0 +codedb_search,q137,Get an S3 object,1,1,249,1.0,1.0,1.0,1,373,1.0,1.0,1.0,1,634,1.0,1.0,1.0 +codedb_search,q138,List Cloudflare zones,1,1,182,1.0,1.0,1.0,1,285,1.0,1.0,1.0,1,482,1.0,1.0,1.0 +codedb_search,q139,Create a Stripe invoice,1,1,205,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,647,1.0,1.0,1.0 +codedb_search,q140,List GitHub Actions workflows,1,1,268,1.0,1.0,1.0,1,453,1.0,1.0,1.0,1,772,1.0,1.0,1.0 +codedb_search,q141,Get GitHub Actions job logs,1,1,256,1.0,1.0,1.0,1,442,1.0,1.0,1.0,1,769,1.0,1.0,1.0 +codedb_search,q142,Create a GitHub repository,1,1,333,1.0,1.0,1.0,1,562,1.0,1.0,1.0,1,927,1.0,1.0,1.0 +codedb_search,q143,Checkout a git branch,1,1,212,1.0,1.0,1.0,1,429,1.0,1.0,1.0,1,1001,1.0,1.0,1.0 +codedb_search,q144,Show git log,1,1,196,1.0,1.0,1.0,1,299,1.0,1.0,1.0,1,555,1.0,1.0,1.0 +codedb_search,q145,List postgres schemas,1,1,160,1.0,1.0,1.0,1,277,1.0,1.0,1.0,1,439,1.0,1.0,1.0 +codedb_search,q146,Open nodes in the knowledge graph,1,1,163,1.0,1.0,1.0,1,262,1.0,1.0,1.0,1,531,1.0,1.0,1.0 +codedb_search,q147,Get file metadata from the filesystem,1,1,209,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,715,1.0,1.0,1.0 +codedb_search,q148,Write a new local file,1,1,206,1.0,1.0,1.0,1,384,1.0,1.0,1.0,1,846,1.0,1.0,1.0 +codedb_search,q149,Search GitHub issues,1,1,233,1.0,1.0,1.0,1,385,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +codedb_search,q150,Get the authenticated GitHub user,2,1,179,1.0,1.0,1.0,1,288,1.0,1.0,1.0,1,622,1.0,1.0,1.0 +codedb_symbol,q001,Create a GitHub issue,3,1,341,1.0,1.0,1.0,1,466,1.0,1.0,1.0,1,802,1.0,1.0,1.0 +codedb_symbol,q002,Create a Jira ticket,8,3,719,0.0,0.0,1.0,2,809,0.0,1.0,1.0,1,789,1.0,1.0,1.0 +codedb_symbol,q003,Send a Slack message,1,1,226,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,694,1.0,1.0,1.0 +codedb_symbol,q004,List open pull requests,1,1,251,1.0,1.0,1.0,1,506,1.0,1.0,1.0,1,1038,1.0,1.0,1.0 +codedb_symbol,q005,Open a pull request on GitHub,46,16,4346,0.0,0.0,0.0,10,4440,0.0,0.0,0.0,5,4325,0.0,0.0,0.0 +codedb_symbol,q006,Run a SQL query against postgres,104,35,9005,0.0,0.0,0.0,21,8697,0.0,0.0,0.0,11,8763,0.0,0.0,0.0 +codedb_symbol,q007,Read a local file from disk,85,29,7857,0.0,0.0,0.0,17,7449,0.0,0.0,0.0,9,7634,0.0,0.0,0.0 +codedb_symbol,q008,What files are in this folder,7,3,648,0.0,0.0,1.0,2,691,0.0,1.0,1.0,1,669,1.0,1.0,1.0 +codedb_symbol,q009,Remember this fact in the knowledge graph,99,33,8732,0.0,0.0,0.0,20,8484,0.0,0.0,0.0,10,8254,0.0,0.0,0.0 +codedb_symbol,q010,Search the web with Brave,4,2,500,0.0,1.0,1.0,1,428,1.0,1.0,1.0,1,810,1.0,1.0,1.0 +codedb_symbol,q011,Fetch the contents of a URL,111,37,9389,0.0,0.0,0.0,23,9302,0.0,0.0,0.0,12,9383,0.0,0.0,0.0 +codedb_symbol,q012,Show git status,1,1,189,1.0,1.0,1.0,1,315,1.0,1.0,1.0,1,602,1.0,1.0,1.0 +codedb_symbol,q013,Commit staged changes,117,39,9686,0.0,0.0,0.0,24,9576,0.0,0.0,0.0,12,9348,0.0,0.0,0.0 +codedb_symbol,q014,Create a Linear issue,143,48,12006,0.0,0.0,0.0,29,11681,0.0,0.0,0.0,15,11704,0.0,0.0,0.0 +codedb_symbol,q015,Make a task in my project tracker Linear,9,3,741,0.0,0.0,1.0,2,831,0.0,1.0,1.0,1,807,1.0,1.0,1.0 +codedb_symbol,q016,Post a comment on a GitHub issue,41,14,3866,0.0,0.0,0.0,9,4037,0.0,0.0,0.0,5,4516,0.0,0.0,0.0 +codedb_symbol,q017,Merge the pull request,3,1,303,1.0,1.0,1.0,1,595,1.0,1.0,1.0,1,1038,1.0,1.0,1.0 +codedb_symbol,q018,Trigger a GitHub Actions workflow,65,22,6335,0.0,0.0,0.0,13,6086,0.0,0.0,0.0,7,6281,0.0,0.0,0.0 +codedb_symbol,q019,Get Dependabot alerts,1,1,262,1.0,1.0,1.0,1,437,1.0,1.0,1.0,1,747,1.0,1.0,1.0 +codedb_symbol,q020,List Sentry issues,3,1,303,1.0,1.0,1.0,1,507,1.0,1.0,1.0,1,841,1.0,1.0,1.0 +codedb_symbol,q021,Create a Stripe customer,3,1,203,1.0,1.0,1.0,1,324,1.0,1.0,1.0,1,720,1.0,1.0,1.0 +codedb_symbol,q022,Refund a Stripe payment,9,3,678,0.0,0.0,1.0,2,706,0.0,1.0,1.0,1,686,1.0,1.0,1.0 +codedb_symbol,q023,Send an email with Gmail,3,1,224,1.0,1.0,1.0,1,349,1.0,1.0,1.0,1,713,1.0,1.0,1.0 +codedb_symbol,q024,Search my Gmail inbox,251,84,19977,0.0,0.0,0.0,51,19522,0.0,0.0,0.0,26,19419,0.0,0.0,0.0 +codedb_symbol,q025,Upload a file to Google Drive,4,2,496,0.0,1.0,1.0,1,383,1.0,1.0,1.0,1,671,1.0,1.0,1.0 +codedb_symbol,q026,Create a calendar event,1,1,317,1.0,1.0,1.0,1,461,1.0,1.0,1.0,1,834,1.0,1.0,1.0 +codedb_symbol,q027,List Kubernetes pods,1,1,242,1.0,1.0,1.0,1,370,1.0,1.0,1.0,1,717,1.0,1.0,1.0 +codedb_symbol,q028,Get logs from a k8s pod,9,3,758,0.0,0.0,1.0,2,807,0.0,1.0,1.0,1,783,1.0,1.0,1.0 +codedb_symbol,q029,Run a docker container,189,63,15143,0.0,0.0,0.0,38,14689,0.0,0.0,0.0,19,14309,0.0,0.0,0.0 +codedb_symbol,q030,Show docker container logs,2,1,274,1.0,1.0,1.0,1,387,1.0,1.0,1.0,1,757,1.0,1.0,1.0 +codedb_symbol,q031,Navigate the browser to a URL,174,58,14269,0.0,0.0,0.0,35,13802,0.0,0.0,0.0,18,13742,0.0,0.0,0.0 +codedb_symbol,q032,Take a screenshot of the page,178,60,14674,0.0,0.0,0.0,36,14146,0.0,0.0,0.0,18,13750,0.0,0.0,0.0 +codedb_symbol,q033,Create a GitLab merge request,10,4,1156,0.0,0.0,0.0,2,1020,0.0,1.0,1.0,1,998,1.0,1.0,1.0 +codedb_symbol,q034,Search Notion pages,10,4,782,0.0,0.0,0.0,2,650,0.0,1.0,1.0,1,630,1.0,1.0,1.0 +codedb_symbol,q035,Create a Notion page,2,1,230,1.0,1.0,1.0,1,373,1.0,1.0,1.0,1,774,1.0,1.0,1.0 +codedb_symbol,q036,Query a Notion database,1,1,266,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,733,1.0,1.0,1.0 +codedb_symbol,q037,Create a Zendesk ticket,3,1,221,1.0,1.0,1.0,1,365,1.0,1.0,1.0,1,645,1.0,1.0,1.0 +codedb_symbol,q038,Create a HubSpot contact,3,1,216,1.0,1.0,1.0,1,328,1.0,1.0,1.0,1,722,1.0,1.0,1.0 +codedb_symbol,q039,Create a Salesforce lead,1,1,234,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,741,1.0,1.0,1.0 +codedb_symbol,q040,Run a SOQL query,5,2,413,0.0,1.0,1.0,1,293,1.0,1.0,1.0,1,721,1.0,1.0,1.0 +codedb_symbol,q041,List Shopify orders,1,1,210,1.0,1.0,1.0,1,335,1.0,1.0,1.0,1,644,1.0,1.0,1.0 +codedb_symbol,q042,Create a PagerDuty incident,3,1,267,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,802,1.0,1.0,1.0 +codedb_symbol,q043,Who is on call in PagerDuty,2,1,227,1.0,1.0,1.0,1,397,1.0,1.0,1.0,1,714,1.0,1.0,1.0 +codedb_symbol,q044,Query Datadog metrics,1,1,235,1.0,1.0,1.0,1,347,1.0,1.0,1.0,1,801,1.0,1.0,1.0 +codedb_symbol,q045,Search Datadog logs,1,1,276,1.0,1.0,1.0,1,437,1.0,1.0,1.0,1,780,1.0,1.0,1.0 +codedb_symbol,q046,List S3 buckets,1,1,191,1.0,1.0,1.0,1,317,1.0,1.0,1.0,1,752,1.0,1.0,1.0 +codedb_symbol,q047,Upload an object to S3,2,1,234,1.0,1.0,1.0,1,363,1.0,1.0,1.0,1,676,1.0,1.0,1.0 +codedb_symbol,q048,Purge Cloudflare cache,1,1,287,1.0,1.0,1.0,1,477,1.0,1.0,1.0,1,917,1.0,1.0,1.0 +codedb_symbol,q049,Create a Cloudflare DNS record,301,101,24119,0.0,0.0,0.0,61,23357,0.0,0.0,0.0,31,23011,0.0,0.0,0.0 +codedb_symbol,q050,List Vercel deployments,1,1,195,1.0,1.0,1.0,1,344,1.0,1.0,1.0,1,618,1.0,1.0,1.0 +codedb_symbol,q051,Insert a row into Supabase,2,1,238,1.0,1.0,1.0,1,355,1.0,1.0,1.0,1,739,1.0,1.0,1.0 +codedb_symbol,q052,Export images from Figma,1,1,214,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,627,1.0,1.0,1.0 +codedb_symbol,q053,Send an SMS with Twilio,1,1,262,1.0,1.0,1.0,1,385,1.0,1.0,1.0,1,739,1.0,1.0,1.0 +codedb_symbol,q054,Create an Okta user,8,3,605,0.0,0.0,1.0,2,640,0.0,1.0,1.0,1,619,1.0,1.0,1.0 +codedb_symbol,q055,Instant query Prometheus,327,109,25677,0.0,0.0,0.0,66,25014,0.0,0.0,0.0,33,24354,0.0,0.0,0.0 +codedb_symbol,q056,Search Grafana dashboards,2,1,209,1.0,1.0,1.0,1,359,1.0,1.0,1.0,1,732,1.0,1.0,1.0 +codedb_symbol,q057,Trigger a CircleCI pipeline,9,3,667,0.0,0.0,1.0,2,695,0.0,1.0,1.0,1,674,1.0,1.0,1.0 +codedb_symbol,q058,Run terraform plan,1,1,202,1.0,1.0,1.0,1,342,1.0,1.0,1.0,1,682,1.0,1.0,1.0 +codedb_symbol,q059,Create an Asana task,3,1,245,1.0,1.0,1.0,1,387,1.0,1.0,1.0,1,747,1.0,1.0,1.0 +codedb_symbol,q060,Create a Trello card,2,1,245,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,796,1.0,1.0,1.0 +codedb_symbol,q061,Create a ClickUp task,7,3,736,0.0,0.0,1.0,2,785,0.0,1.0,1.0,1,764,1.0,1.0,1.0 +codedb_symbol,q062,Create a Confluence page,4,2,488,0.0,1.0,1.0,1,416,1.0,1.0,1.0,1,742,1.0,1.0,1.0 +codedb_symbol,q063,Send a Discord message,2,1,226,1.0,1.0,1.0,1,376,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +codedb_symbol,q064,Search Elasticsearch,1,1,244,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,706,1.0,1.0,1.0 +codedb_symbol,q065,Get a Redis key,4,2,397,0.0,1.0,1.0,1,289,1.0,1.0,1.0,1,602,1.0,1.0,1.0 +codedb_symbol,q066,Find MongoDB documents,286,96,22835,0.0,0.0,0.0,58,22198,0.0,0.0,0.0,29,21618,0.0,0.0,0.0 +codedb_symbol,q067,List SQLite tables,3,1,151,1.0,1.0,1.0,1,260,1.0,1.0,1.0,1,626,1.0,1.0,1.0 +codedb_symbol,q068,Describe a postgres table,1,1,160,1.0,1.0,1.0,1,258,1.0,1.0,1.0,1,577,1.0,1.0,1.0 +codedb_symbol,q069,github_create_issue,1,1,357,1.0,1.0,1.0,1,522,1.0,1.0,1.0,1,1073,1.0,1.0,1.0 +codedb_symbol,q070,createIssue,1,1,356,1.0,1.0,1.0,1,521,1.0,1.0,1.0,1,1072,1.0,1.0,1.0 +codedb_symbol,q071,list-directory,1,1,186,1.0,1.0,1.0,1,281,1.0,1.0,1.0,1,629,1.0,1.0,1.0 +codedb_symbol,q072,sequential thinking next thought,116,39,9799,0.0,0.0,0.0,24,9727,0.0,0.0,0.0,12,9475,0.0,0.0,0.0 +codedb_symbol,q073,What time is it in Tokyo,125,42,10496,0.0,0.0,0.0,25,10044,0.0,0.0,0.0,13,10182,0.0,0.0,0.0 +codedb_symbol,q074,Convert time between timezones,127,43,10789,0.0,0.0,0.0,26,10502,0.0,0.0,0.0,13,10229,0.0,0.0,0.0 +codedb_symbol,q075,Star a GitHub repo,3,1,241,1.0,1.0,1.0,1,412,1.0,1.0,1.0,1,766,1.0,1.0,1.0 +codedb_symbol,q076,Fork a repository,1,1,248,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,831,1.0,1.0,1.0 +codedb_symbol,q077,Get the latest GitHub release,7,3,731,0.0,0.0,1.0,2,767,0.0,1.0,1.0,1,745,1.0,1.0,1.0 +codedb_symbol,q078,Create a gist,1,1,229,1.0,1.0,1.0,1,413,1.0,1.0,1.0,1,814,1.0,1.0,1.0 +codedb_symbol,q079,Search GitHub code,1,1,214,1.0,1.0,1.0,1,409,1.0,1.0,1.0,1,751,1.0,1.0,1.0 +codedb_symbol,q080,List GitHub notifications,1,1,190,1.0,1.0,1.0,1,283,1.0,1.0,1.0,1,620,1.0,1.0,1.0 +codedb_symbol,q081,Assign Copilot to an issue,49,17,5013,0.0,0.0,0.0,10,4762,0.0,0.0,0.0,5,4652,0.0,0.0,0.0 +codedb_symbol,q082,Get secret scanning alerts,3,1,276,1.0,1.0,1.0,1,446,1.0,1.0,1.0,1,789,1.0,1.0,1.0 +codedb_symbol,q083,Create or update a file on GitHub,27,9,2203,0.0,0.0,0.0,6,2360,0.0,0.0,0.0,3,2288,0.0,0.0,1.0 +codedb_symbol,q084,List collaborators on a repo,31,11,3009,0.0,0.0,0.0,7,3110,0.0,0.0,0.0,4,3597,0.0,0.0,0.0 +codedb_symbol,q085,Reply in a Slack thread,1,1,244,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,747,1.0,1.0,1.0 +codedb_symbol,q086,Add an emoji reaction on Slack,4,2,471,0.0,1.0,1.0,1,379,1.0,1.0,1.0,1,729,1.0,1.0,1.0 +codedb_symbol,q087,Search Slack messages,2,1,210,1.0,1.0,1.0,1,344,1.0,1.0,1.0,1,666,1.0,1.0,1.0 +codedb_symbol,q088,Create a Slack channel,5,2,405,0.0,1.0,1.0,1,312,1.0,1.0,1.0,1,681,1.0,1.0,1.0 +codedb_symbol,q089,Transition a Jira issue,153,51,12624,0.0,0.0,0.0,31,12368,0.0,0.0,0.0,16,12436,0.0,0.0,0.0 +codedb_symbol,q090,Search Jira with JQL,155,52,12765,0.0,0.0,0.0,31,12256,0.0,0.0,0.0,16,12345,0.0,0.0,0.0 +codedb_symbol,q091,Add a worklog to Jira,5,2,478,0.0,1.0,1.0,1,399,1.0,1.0,1.0,1,814,1.0,1.0,1.0 +codedb_symbol,q092,List Linear teams,1,1,158,1.0,1.0,1.0,1,375,1.0,1.0,1.0,1,708,1.0,1.0,1.0 +codedb_symbol,q093,Comment on a Linear issue,146,49,12231,0.0,0.0,0.0,30,12034,0.0,0.0,0.0,15,11719,0.0,0.0,0.0 +codedb_symbol,q094,Move a local file,2,1,213,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,780,1.0,1.0,1.0 +codedb_symbol,q095,Recursively search files by pattern,96,32,8562,0.0,0.0,0.0,20,8492,0.0,0.0,0.0,10,8272,0.0,0.0,0.0 +codedb_symbol,q096,Edit a file in place with old and new string,89,30,8310,0.0,0.0,0.0,18,7998,0.0,0.0,0.0,9,7764,0.0,0.0,0.0 +codedb_symbol,q097,Read the whole knowledge graph,1,1,171,1.0,1.0,1.0,1,280,1.0,1.0,1.0,1,627,1.0,1.0,1.0 +codedb_symbol,q098,Search memory nodes,1,1,202,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,652,1.0,1.0,1.0 +codedb_symbol,q099,Show unstaged git diff,3,1,190,1.0,1.0,1.0,1,340,1.0,1.0,1.0,1,719,1.0,1.0,1.0 +codedb_symbol,q100,Stage files with git add,1,1,244,1.0,1.0,1.0,1,355,1.0,1.0,1.0,1,678,1.0,1.0,1.0 +codedb_symbol,q101,Create a Shopify product,3,1,237,1.0,1.0,1.0,1,379,1.0,1.0,1.0,1,703,1.0,1.0,1.0 +codedb_symbol,q102,Cancel a Stripe subscription,2,1,187,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,631,1.0,1.0,1.0 +codedb_symbol,q103,Create a payment intent,3,1,272,1.0,1.0,1.0,1,409,1.0,1.0,1.0,1,780,1.0,1.0,1.0 +codedb_symbol,q104,Acknowledge a PagerDuty incident,5,2,481,0.0,1.0,1.0,1,405,1.0,1.0,1.0,1,804,1.0,1.0,1.0 +codedb_symbol,q105,Create a Datadog monitor,3,1,259,1.0,1.0,1.0,1,388,1.0,1.0,1.0,1,786,1.0,1.0,1.0 +codedb_symbol,q106,Resolve a Sentry issue,232,78,18579,0.0,0.0,0.0,47,17991,0.0,0.0,0.0,24,17923,0.0,0.0,0.0 +codedb_symbol,q107,Scale a Kubernetes deployment,3,1,234,1.0,1.0,1.0,1,343,1.0,1.0,1.0,1,652,1.0,1.0,1.0 +codedb_symbol,q108,Apply a Kubernetes manifest,203,68,16305,0.0,0.0,0.0,41,15821,0.0,0.0,0.0,21,15741,0.0,0.0,0.0 +codedb_symbol,q109,Stop a docker container,190,64,15302,0.0,0.0,0.0,38,14689,0.0,0.0,0.0,19,14309,0.0,0.0,0.0 +codedb_symbol,q110,Build a docker image,10,4,797,0.0,0.0,0.0,2,665,0.0,1.0,1.0,1,645,1.0,1.0,1.0 +codedb_symbol,q111,Click an element in Playwright,171,57,14078,0.0,0.0,0.0,35,13845,0.0,0.0,0.0,18,13724,0.0,0.0,0.0 +codedb_symbol,q112,Type text into a browser input,2,1,266,1.0,1.0,1.0,1,396,1.0,1.0,1.0,1,699,1.0,1.0,1.0 +codedb_symbol,q113,List GitLab projects,2,1,204,1.0,1.0,1.0,1,284,1.0,1.0,1.0,1,610,1.0,1.0,1.0 +codedb_symbol,q114,Create a Bitbucket pull request,272,91,21988,0.0,0.0,0.0,55,21340,0.0,0.0,0.0,28,20983,0.0,0.0,0.0 +codedb_symbol,q115,Create a Gmail draft,1,1,214,1.0,1.0,1.0,1,402,1.0,1.0,1.0,1,717,1.0,1.0,1.0 +codedb_symbol,q116,Share a Google Drive file,261,87,20738,0.0,0.0,0.0,53,20393,0.0,0.0,0.0,27,20238,0.0,0.0,0.0 +codedb_symbol,q117,List Google Calendar events,1,1,301,1.0,1.0,1.0,1,461,1.0,1.0,1.0,1,771,1.0,1.0,1.0 +codedb_symbol,q118,Delete a calendar event,4,2,576,0.0,1.0,1.0,1,461,1.0,1.0,1.0,1,834,1.0,1.0,1.0 +codedb_symbol,q119,List Vercel projects,1,1,139,1.0,1.0,1.0,1,241,1.0,1.0,1.0,1,585,1.0,1.0,1.0 +codedb_symbol,q120,Select rows from Supabase,1,1,237,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,711,1.0,1.0,1.0 +codedb_symbol,q121,Post a Figma comment,8,3,704,0.0,0.0,1.0,2,748,0.0,1.0,1.0,1,727,1.0,1.0,1.0 +codedb_symbol,q122,List Okta users,2,1,195,1.0,1.0,1.0,1,312,1.0,1.0,1.0,1,646,1.0,1.0,1.0 +codedb_symbol,q123,Range query Prometheus,10,4,940,0.0,0.0,0.0,2,808,0.0,1.0,1.0,1,788,1.0,1.0,1.0 +codedb_symbol,q124,Create a Grafana annotation,1,1,209,1.0,1.0,1.0,1,314,1.0,1.0,1.0,1,602,1.0,1.0,1.0 +codedb_symbol,q125,Apply terraform,334,112,26006,0.0,0.0,0.0,67,25091,0.0,0.0,0.0,34,24881,0.0,0.0,0.0 +codedb_symbol,q126,List Asana tasks,1,1,192,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,718,1.0,1.0,1.0 +codedb_symbol,q127,Comment on a Trello card,8,3,746,0.0,0.0,1.0,2,819,0.0,1.0,1.0,1,797,1.0,1.0,1.0 +codedb_symbol,q128,Update a ClickUp task,6,2,448,0.0,1.0,1.0,2,785,0.0,1.0,1.0,1,764,1.0,1.0,1.0 +codedb_symbol,q129,Search Confluence,1,1,187,1.0,1.0,1.0,1,369,1.0,1.0,1.0,1,848,1.0,1.0,1.0 +codedb_symbol,q130,Create a HubSpot deal,1,1,257,1.0,1.0,1.0,1,378,1.0,1.0,1.0,1,676,1.0,1.0,1.0 +codedb_symbol,q131,List Zendesk tickets,1,1,220,1.0,1.0,1.0,1,388,1.0,1.0,1.0,1,713,1.0,1.0,1.0 +codedb_symbol,q132,List Discord channels,1,1,184,1.0,1.0,1.0,1,308,1.0,1.0,1.0,1,622,1.0,1.0,1.0 +codedb_symbol,q133,Set a Redis key,5,2,397,0.0,1.0,1.0,1,289,1.0,1.0,1.0,1,602,1.0,1.0,1.0 +codedb_symbol,q134,Insert MongoDB documents,287,96,22835,0.0,0.0,0.0,58,22198,0.0,0.0,0.0,29,21618,0.0,0.0,0.0 +codedb_symbol,q135,Run a MySQL query,1,1,178,1.0,1.0,1.0,1,293,1.0,1.0,1.0,1,721,1.0,1.0,1.0 +codedb_symbol,q136,Index a document in Elasticsearch,293,98,23327,0.0,0.0,0.0,59,22549,0.0,0.0,0.0,30,22277,0.0,0.0,0.0 +codedb_symbol,q137,Get an S3 object,1,1,251,1.0,1.0,1.0,1,392,1.0,1.0,1.0,1,630,1.0,1.0,1.0 +codedb_symbol,q138,List Cloudflare zones,1,1,164,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,673,1.0,1.0,1.0 +codedb_symbol,q139,Create a Stripe invoice,2,1,194,1.0,1.0,1.0,1,361,1.0,1.0,1.0,1,701,1.0,1.0,1.0 +codedb_symbol,q140,List GitHub Actions workflows,62,21,6073,0.0,0.0,0.0,13,6075,0.0,0.0,0.0,7,6292,0.0,0.0,0.0 +codedb_symbol,q141,Get GitHub Actions job logs,3,1,276,1.0,1.0,1.0,1,437,1.0,1.0,1.0,1,759,1.0,1.0,1.0 +codedb_symbol,q142,Create a GitHub repository,4,2,515,0.0,1.0,1.0,1,388,1.0,1.0,1.0,1,773,1.0,1.0,1.0 +codedb_symbol,q143,Checkout a git branch,7,3,714,0.0,0.0,1.0,2,770,0.0,1.0,1.0,1,750,1.0,1.0,1.0 +codedb_symbol,q144,Show git log,1,1,274,1.0,1.0,1.0,1,397,1.0,1.0,1.0,1,762,1.0,1.0,1.0 +codedb_symbol,q145,List postgres schemas,1,1,195,1.0,1.0,1.0,1,302,1.0,1.0,1.0,1,615,1.0,1.0,1.0 +codedb_symbol,q146,Open nodes in the knowledge graph,106,36,9236,0.0,0.0,0.0,22,9029,0.0,0.0,0.0,11,8787,0.0,0.0,0.0 +codedb_symbol,q147,Get file metadata from the filesystem,97,33,8765,0.0,0.0,0.0,20,8530,0.0,0.0,0.0,10,8310,0.0,0.0,0.0 +codedb_symbol,q148,Write a new local file,87,29,7978,0.0,0.0,0.0,18,7948,0.0,0.0,0.0,9,7759,0.0,0.0,0.0 +codedb_symbol,q149,Search GitHub issues,2,1,280,1.0,1.0,1.0,1,484,1.0,1.0,1.0,1,844,1.0,1.0,1.0 +codedb_symbol,q150,Get the authenticated GitHub user,11,4,802,0.0,0.0,0.0,3,1045,0.0,0.0,1.0,2,1428,0.0,1.0,1.0 +codedb_trigram_tokens,q001,Create a GitHub issue,1,1,270,1.0,1.0,1.0,1,370,1.0,1.0,1.0,1,704,1.0,1.0,1.0 +codedb_trigram_tokens,q002,Create a Jira ticket,10,4,839,0.0,0.0,0.0,2,707,0.0,1.0,1.0,1,687,1.0,1.0,1.0 +codedb_trigram_tokens,q003,Send a Slack message,1,1,249,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,839,1.0,1.0,1.0 +codedb_trigram_tokens,q004,List open pull requests,2,1,393,1.0,1.0,1.0,1,536,1.0,1.0,1.0,1,885,1.0,1.0,1.0 +codedb_trigram_tokens,q005,Open a pull request on GitHub,2,1,395,1.0,1.0,1.0,1,598,1.0,1.0,1.0,1,962,1.0,1.0,1.0 +codedb_trigram_tokens,q006,Run a SQL query against postgres,1,1,183,1.0,1.0,1.0,1,272,1.0,1.0,1.0,1,546,1.0,1.0,1.0 +codedb_trigram_tokens,q007,Read a local file from disk,10,4,937,0.0,0.0,0.0,2,801,0.0,1.0,1.0,1,779,1.0,1.0,1.0 +codedb_trigram_tokens,q008,What files are in this folder,91,31,8425,0.0,0.0,0.0,19,8277,0.0,0.0,0.0,10,8326,0.0,0.0,0.0 +codedb_trigram_tokens,q009,Remember this fact in the knowledge graph,99,33,8757,0.0,0.0,0.0,20,8509,0.0,0.0,0.0,10,8279,0.0,0.0,0.0 +codedb_trigram_tokens,q010,Search the web with Brave,1,1,169,1.0,1.0,1.0,1,298,1.0,1.0,1.0,1,600,1.0,1.0,1.0 +codedb_trigram_tokens,q011,Fetch the contents of a URL,1,1,258,1.0,1.0,1.0,1,370,1.0,1.0,1.0,1,671,1.0,1.0,1.0 +codedb_trigram_tokens,q012,Show git status,1,1,178,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,681,1.0,1.0,1.0 +codedb_trigram_tokens,q013,Commit staged changes,1,1,134,1.0,1.0,1.0,1,236,1.0,1.0,1.0,1,611,1.0,1.0,1.0 +codedb_trigram_tokens,q014,Create a Linear issue,7,3,860,0.0,0.0,1.0,2,910,0.0,1.0,1.0,1,890,1.0,1.0,1.0 +codedb_trigram_tokens,q015,Make a task in my project tracker Linear,145,49,12381,0.0,0.0,0.0,29,11702,0.0,0.0,0.0,15,11743,0.0,0.0,0.0 +codedb_trigram_tokens,q016,Post a comment on a GitHub issue,44,15,4304,0.0,0.0,0.0,9,4160,0.0,0.0,0.0,5,4542,0.0,0.0,0.0 +codedb_trigram_tokens,q017,Merge the pull request,3,1,321,1.0,1.0,1.0,1,543,1.0,1.0,1.0,1,1093,1.0,1.0,1.0 +codedb_trigram_tokens,q018,Trigger a GitHub Actions workflow,1,1,272,1.0,1.0,1.0,1,412,1.0,1.0,1.0,1,722,1.0,1.0,1.0 +codedb_trigram_tokens,q019,Get Dependabot alerts,1,1,189,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,724,1.0,1.0,1.0 +codedb_trigram_tokens,q020,List Sentry issues,1,1,238,1.0,1.0,1.0,1,428,1.0,1.0,1.0,1,795,1.0,1.0,1.0 +codedb_trigram_tokens,q021,Create a Stripe customer,3,1,201,1.0,1.0,1.0,1,309,1.0,1.0,1.0,1,593,1.0,1.0,1.0 +codedb_trigram_tokens,q022,Refund a Stripe payment,2,1,212,1.0,1.0,1.0,1,379,1.0,1.0,1.0,1,587,1.0,1.0,1.0 +codedb_trigram_tokens,q023,Send an email with Gmail,1,1,264,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,710,1.0,1.0,1.0 +codedb_trigram_tokens,q024,Search my Gmail inbox,251,84,19910,0.0,0.0,0.0,51,19469,0.0,0.0,0.0,26,19411,0.0,0.0,0.0 +codedb_trigram_tokens,q025,Upload a file to Google Drive,10,4,1083,0.0,0.0,0.0,2,947,0.0,1.0,1.0,1,925,1.0,1.0,1.0 +codedb_trigram_tokens,q026,Create a calendar event,2,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,748,1.0,1.0,1.0 +codedb_trigram_tokens,q027,List Kubernetes pods,2,1,262,1.0,1.0,1.0,1,416,1.0,1.0,1.0,1,627,1.0,1.0,1.0 +codedb_trigram_tokens,q028,Get logs from a k8s pod,1,1,273,1.0,1.0,1.0,1,450,1.0,1.0,1.0,1,801,1.0,1.0,1.0 +codedb_trigram_tokens,q029,Run a docker container,3,1,219,1.0,1.0,1.0,1,348,1.0,1.0,1.0,1,646,1.0,1.0,1.0 +codedb_trigram_tokens,q030,Show docker container logs,3,1,188,1.0,1.0,1.0,1,379,1.0,1.0,1.0,1,747,1.0,1.0,1.0 +codedb_trigram_tokens,q031,Navigate the browser to a URL,1,1,180,1.0,1.0,1.0,1,423,1.0,1.0,1.0,1,731,1.0,1.0,1.0 +codedb_trigram_tokens,q032,Take a screenshot of the page,1,1,246,1.0,1.0,1.0,1,285,1.0,1.0,1.0,1,635,1.0,1.0,1.0 +codedb_trigram_tokens,q033,Create a GitLab merge request,3,1,297,1.0,1.0,1.0,1,431,1.0,1.0,1.0,1,831,1.0,1.0,1.0 +codedb_trigram_tokens,q034,Search Notion pages,1,1,215,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,743,1.0,1.0,1.0 +codedb_trigram_tokens,q035,Create a Notion page,166,56,13846,0.0,0.0,0.0,34,13474,0.0,0.0,0.0,17,13117,0.0,0.0,0.0 +codedb_trigram_tokens,q036,Query a Notion database,1,1,263,1.0,1.0,1.0,1,387,1.0,1.0,1.0,1,711,1.0,1.0,1.0 +codedb_trigram_tokens,q037,Create a Zendesk ticket,2,1,257,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,642,1.0,1.0,1.0 +codedb_trigram_tokens,q038,Create a HubSpot contact,1,1,216,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,801,1.0,1.0,1.0 +codedb_trigram_tokens,q039,Create a Salesforce lead,1,1,259,1.0,1.0,1.0,1,423,1.0,1.0,1.0,1,634,1.0,1.0,1.0 +codedb_trigram_tokens,q040,Run a SOQL query,1,1,173,1.0,1.0,1.0,1,331,1.0,1.0,1.0,1,746,1.0,1.0,1.0 +codedb_trigram_tokens,q041,List Shopify orders,1,1,230,1.0,1.0,1.0,1,376,1.0,1.0,1.0,1,626,1.0,1.0,1.0 +codedb_trigram_tokens,q042,Create a PagerDuty incident,6,2,486,0.0,1.0,1.0,2,736,0.0,1.0,1.0,1,715,1.0,1.0,1.0 +codedb_trigram_tokens,q043,Who is on call in PagerDuty,1,1,213,1.0,1.0,1.0,1,378,1.0,1.0,1.0,1,712,1.0,1.0,1.0 +codedb_trigram_tokens,q044,Query Datadog metrics,1,1,287,1.0,1.0,1.0,1,449,1.0,1.0,1.0,1,667,1.0,1.0,1.0 +codedb_trigram_tokens,q045,Search Datadog logs,1,1,315,1.0,1.0,1.0,1,528,1.0,1.0,1.0,1,828,1.0,1.0,1.0 +codedb_trigram_tokens,q046,List S3 buckets,1,1,115,1.0,1.0,1.0,1,220,1.0,1.0,1.0,1,642,1.0,1.0,1.0 +codedb_trigram_tokens,q047,Upload an object to S3,1,1,346,1.0,1.0,1.0,1,505,1.0,1.0,1.0,1,822,1.0,1.0,1.0 +codedb_trigram_tokens,q048,Purge Cloudflare cache,1,1,231,1.0,1.0,1.0,1,286,1.0,1.0,1.0,1,643,1.0,1.0,1.0 +codedb_trigram_tokens,q049,Create a Cloudflare DNS record,1,1,217,1.0,1.0,1.0,1,363,1.0,1.0,1.0,1,660,1.0,1.0,1.0 +codedb_trigram_tokens,q050,List Vercel deployments,1,1,173,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,541,1.0,1.0,1.0 +codedb_trigram_tokens,q051,Insert a row into Supabase,1,1,245,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,620,1.0,1.0,1.0 +codedb_trigram_tokens,q052,Export images from Figma,1,1,215,1.0,1.0,1.0,1,325,1.0,1.0,1.0,1,797,1.0,1.0,1.0 +codedb_trigram_tokens,q053,Send an SMS with Twilio,1,1,161,1.0,1.0,1.0,1,308,1.0,1.0,1.0,1,703,1.0,1.0,1.0 +codedb_trigram_tokens,q054,Create an Okta user,2,1,183,1.0,1.0,1.0,1,289,1.0,1.0,1.0,1,657,1.0,1.0,1.0 +codedb_trigram_tokens,q055,Instant query Prometheus,1,1,221,1.0,1.0,1.0,1,275,1.0,1.0,1.0,1,681,1.0,1.0,1.0 +codedb_trigram_tokens,q056,Search Grafana dashboards,1,1,217,1.0,1.0,1.0,1,271,1.0,1.0,1.0,1,677,1.0,1.0,1.0 +codedb_trigram_tokens,q057,Trigger a CircleCI pipeline,1,1,200,1.0,1.0,1.0,1,271,1.0,1.0,1.0,1,634,1.0,1.0,1.0 +codedb_trigram_tokens,q058,Run terraform plan,1,1,172,1.0,1.0,1.0,1,227,1.0,1.0,1.0,1,584,1.0,1.0,1.0 +codedb_trigram_tokens,q059,Create an Asana task,3,1,267,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,824,1.0,1.0,1.0 +codedb_trigram_tokens,q060,Create a Trello card,2,1,259,1.0,1.0,1.0,1,316,1.0,1.0,1.0,1,673,1.0,1.0,1.0 +codedb_trigram_tokens,q061,Create a ClickUp task,2,1,257,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,660,1.0,1.0,1.0 +codedb_trigram_tokens,q062,Create a Confluence page,2,1,260,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,686,1.0,1.0,1.0 +codedb_trigram_tokens,q063,Send a Discord message,2,1,245,1.0,1.0,1.0,1,343,1.0,1.0,1.0,1,763,1.0,1.0,1.0 +codedb_trigram_tokens,q064,Search Elasticsearch,3,1,239,1.0,1.0,1.0,1,293,1.0,1.0,1.0,1,699,1.0,1.0,1.0 +codedb_trigram_tokens,q065,Get a Redis key,1,1,185,1.0,1.0,1.0,1,288,1.0,1.0,1.0,1,553,1.0,1.0,1.0 +codedb_trigram_tokens,q066,Find MongoDB documents,3,1,244,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,738,1.0,1.0,1.0 +codedb_trigram_tokens,q067,List SQLite tables,1,1,156,1.0,1.0,1.0,1,252,1.0,1.0,1.0,1,451,1.0,1.0,1.0 +codedb_trigram_tokens,q068,Describe a postgres table,1,1,170,1.0,1.0,1.0,1,286,1.0,1.0,1.0,1,543,1.0,1.0,1.0 +codedb_trigram_tokens,q069,github_create_issue,4,2,634,0.0,1.0,1.0,1,542,1.0,1.0,1.0,1,920,1.0,1.0,1.0 +codedb_trigram_tokens,q070,createIssue,34,12,3256,0.0,0.0,0.0,7,3080,0.0,0.0,0.0,4,3574,0.0,0.0,0.0 +codedb_trigram_tokens,q071,list-directory,2,1,182,1.0,1.0,1.0,1,348,1.0,1.0,1.0,1,653,1.0,1.0,1.0 +codedb_trigram_tokens,q072,sequential thinking next thought,1,1,186,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,713,1.0,1.0,1.0 +codedb_trigram_tokens,q073,What time is it in Tokyo,4,2,452,0.0,1.0,1.0,1,380,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +codedb_trigram_tokens,q074,Convert time between timezones,1,1,204,1.0,1.0,1.0,1,335,1.0,1.0,1.0,1,650,1.0,1.0,1.0 +codedb_trigram_tokens,q075,Star a GitHub repo,1,1,220,1.0,1.0,1.0,1,380,1.0,1.0,1.0,1,773,1.0,1.0,1.0 +codedb_trigram_tokens,q076,Fork a repository,1,1,239,1.0,1.0,1.0,1,425,1.0,1.0,1.0,1,675,1.0,1.0,1.0 +codedb_trigram_tokens,q077,Get the latest GitHub release,1,1,200,1.0,1.0,1.0,1,306,1.0,1.0,1.0,1,717,1.0,1.0,1.0 +codedb_trigram_tokens,q078,Create a gist,1,1,200,1.0,1.0,1.0,1,280,1.0,1.0,1.0,1,637,1.0,1.0,1.0 +codedb_trigram_tokens,q079,Search GitHub code,6,2,464,0.0,1.0,1.0,2,681,0.0,1.0,1.0,1,661,1.0,1.0,1.0 +codedb_trigram_tokens,q080,List GitHub notifications,2,1,192,1.0,1.0,1.0,1,284,1.0,1.0,1.0,1,737,1.0,1.0,1.0 +codedb_trigram_tokens,q081,Assign Copilot to an issue,1,1,347,1.0,1.0,1.0,1,537,1.0,1.0,1.0,1,858,1.0,1.0,1.0 +codedb_trigram_tokens,q082,Get secret scanning alerts,1,1,295,1.0,1.0,1.0,1,373,1.0,1.0,1.0,1,720,1.0,1.0,1.0 +codedb_trigram_tokens,q083,Create or update a file on GitHub,1,1,286,1.0,1.0,1.0,1,423,1.0,1.0,1.0,1,768,1.0,1.0,1.0 +codedb_trigram_tokens,q084,List collaborators on a repo,1,1,259,1.0,1.0,1.0,1,392,1.0,1.0,1.0,1,764,1.0,1.0,1.0 +codedb_trigram_tokens,q085,Reply in a Slack thread,1,1,256,1.0,1.0,1.0,1,411,1.0,1.0,1.0,1,675,1.0,1.0,1.0 +codedb_trigram_tokens,q086,Add an emoji reaction on Slack,1,1,252,1.0,1.0,1.0,1,434,1.0,1.0,1.0,1,800,1.0,1.0,1.0 +codedb_trigram_tokens,q087,Search Slack messages,1,1,230,1.0,1.0,1.0,1,384,1.0,1.0,1.0,1,652,1.0,1.0,1.0 +codedb_trigram_tokens,q088,Create a Slack channel,135,45,11143,0.0,0.0,0.0,27,10783,0.0,0.0,0.0,14,10993,0.0,0.0,0.0 +codedb_trigram_tokens,q089,Transition a Jira issue,2,1,213,1.0,1.0,1.0,1,383,1.0,1.0,1.0,1,772,1.0,1.0,1.0 +codedb_trigram_tokens,q090,Search Jira with JQL,1,1,204,1.0,1.0,1.0,1,328,1.0,1.0,1.0,1,670,1.0,1.0,1.0 +codedb_trigram_tokens,q091,Add a worklog to Jira,1,1,338,1.0,1.0,1.0,1,479,1.0,1.0,1.0,1,815,1.0,1.0,1.0 +codedb_trigram_tokens,q092,List Linear teams,4,2,494,0.0,1.0,1.0,1,343,1.0,1.0,1.0,1,663,1.0,1.0,1.0 +codedb_trigram_tokens,q093,Comment on a Linear issue,147,49,12287,0.0,0.0,0.0,30,12042,0.0,0.0,0.0,15,11727,0.0,0.0,0.0 +codedb_trigram_tokens,q094,Move a local file,1,1,213,1.0,1.0,1.0,1,395,1.0,1.0,1.0,1,746,1.0,1.0,1.0 +codedb_trigram_tokens,q095,Recursively search files by pattern,1,1,297,1.0,1.0,1.0,1,419,1.0,1.0,1.0,1,748,1.0,1.0,1.0 +codedb_trigram_tokens,q096,Edit a file in place with old and new string,1,1,265,1.0,1.0,1.0,1,410,1.0,1.0,1.0,1,801,1.0,1.0,1.0 +codedb_trigram_tokens,q097,Read the whole knowledge graph,1,1,158,1.0,1.0,1.0,1,271,1.0,1.0,1.0,1,539,1.0,1.0,1.0 +codedb_trigram_tokens,q098,Search memory nodes,1,1,204,1.0,1.0,1.0,1,305,1.0,1.0,1.0,1,507,1.0,1.0,1.0 +codedb_trigram_tokens,q099,Show unstaged git diff,1,1,187,1.0,1.0,1.0,1,297,1.0,1.0,1.0,1,508,1.0,1.0,1.0 +codedb_trigram_tokens,q100,Stage files with git add,113,38,9484,0.0,0.0,0.0,23,9231,0.0,0.0,0.0,12,9385,0.0,0.0,0.0 +codedb_trigram_tokens,q101,Create a Shopify product,2,1,240,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,684,1.0,1.0,1.0 +codedb_trigram_tokens,q102,Cancel a Stripe subscription,1,1,200,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,546,1.0,1.0,1.0 +codedb_trigram_tokens,q103,Create a payment intent,1,1,234,1.0,1.0,1.0,1,288,1.0,1.0,1.0,1,694,1.0,1.0,1.0 +codedb_trigram_tokens,q104,Acknowledge a PagerDuty incident,1,1,223,1.0,1.0,1.0,1,363,1.0,1.0,1.0,1,626,1.0,1.0,1.0 +codedb_trigram_tokens,q105,Create a Datadog monitor,1,1,259,1.0,1.0,1.0,1,459,1.0,1.0,1.0,1,709,1.0,1.0,1.0 +codedb_trigram_tokens,q106,Resolve a Sentry issue,1,1,245,1.0,1.0,1.0,1,348,1.0,1.0,1.0,1,724,1.0,1.0,1.0 +codedb_trigram_tokens,q107,Scale a Kubernetes deployment,1,1,174,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,622,1.0,1.0,1.0 +codedb_trigram_tokens,q108,Apply a Kubernetes manifest,1,1,246,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,632,1.0,1.0,1.0 +codedb_trigram_tokens,q109,Stop a docker container,1,1,219,1.0,1.0,1.0,1,348,1.0,1.0,1.0,1,608,1.0,1.0,1.0 +codedb_trigram_tokens,q110,Build a docker image,1,1,238,1.0,1.0,1.0,1,330,1.0,1.0,1.0,1,563,1.0,1.0,1.0 +codedb_trigram_tokens,q111,Click an element in Playwright,1,1,230,1.0,1.0,1.0,1,374,1.0,1.0,1.0,1,734,1.0,1.0,1.0 +codedb_trigram_tokens,q112,Type text into a browser input,1,1,252,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,682,1.0,1.0,1.0 +codedb_trigram_tokens,q113,List GitLab projects,1,1,170,1.0,1.0,1.0,1,267,1.0,1.0,1.0,1,524,1.0,1.0,1.0 +codedb_trigram_tokens,q114,Create a Bitbucket pull request,9,3,943,0.0,0.0,1.0,2,1016,0.0,1.0,1.0,1,994,1.0,1.0,1.0 +codedb_trigram_tokens,q115,Create a Gmail draft,2,1,288,1.0,1.0,1.0,1,398,1.0,1.0,1.0,1,673,1.0,1.0,1.0 +codedb_trigram_tokens,q116,Share a Google Drive file,4,2,510,0.0,1.0,1.0,1,417,1.0,1.0,1.0,1,778,1.0,1.0,1.0 +codedb_trigram_tokens,q117,List Google Calendar events,2,1,269,1.0,1.0,1.0,1,376,1.0,1.0,1.0,1,770,1.0,1.0,1.0 +codedb_trigram_tokens,q118,Delete a calendar event,1,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,748,1.0,1.0,1.0 +codedb_trigram_tokens,q119,List Vercel projects,1,1,170,1.0,1.0,1.0,1,262,1.0,1.0,1.0,1,529,1.0,1.0,1.0 +codedb_trigram_tokens,q120,Select rows from Supabase,1,1,237,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,766,1.0,1.0,1.0 +codedb_trigram_tokens,q121,Post a Figma comment,1,1,255,1.0,1.0,1.0,1,406,1.0,1.0,1.0,1,873,1.0,1.0,1.0 +codedb_trigram_tokens,q122,List Okta users,5,2,417,0.0,1.0,1.0,1,317,1.0,1.0,1.0,1,617,1.0,1.0,1.0 +codedb_trigram_tokens,q123,Range query Prometheus,1,1,221,1.0,1.0,1.0,1,275,1.0,1.0,1.0,1,681,1.0,1.0,1.0 +codedb_trigram_tokens,q124,Create a Grafana annotation,1,1,218,1.0,1.0,1.0,1,272,1.0,1.0,1.0,1,678,1.0,1.0,1.0 +codedb_trigram_tokens,q125,Apply terraform,1,1,146,1.0,1.0,1.0,1,238,1.0,1.0,1.0,1,595,1.0,1.0,1.0 +codedb_trigram_tokens,q126,List Asana tasks,1,1,192,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,616,1.0,1.0,1.0 +codedb_trigram_tokens,q127,Comment on a Trello card,2,1,260,1.0,1.0,1.0,1,438,1.0,1.0,1.0,1,934,1.0,1.0,1.0 +codedb_trigram_tokens,q128,Update a ClickUp task,1,1,257,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,660,1.0,1.0,1.0 +codedb_trigram_tokens,q129,Search Confluence,4,2,423,0.0,1.0,1.0,1,366,1.0,1.0,1.0,1,772,1.0,1.0,1.0 +codedb_trigram_tokens,q130,Create a HubSpot deal,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,873,1.0,1.0,1.0 +codedb_trigram_tokens,q131,List Zendesk tickets,1,1,189,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,617,1.0,1.0,1.0 +codedb_trigram_tokens,q132,List Discord channels,1,1,205,1.0,1.0,1.0,1,339,1.0,1.0,1.0,1,547,1.0,1.0,1.0 +codedb_trigram_tokens,q133,Set a Redis key,2,1,185,1.0,1.0,1.0,1,288,1.0,1.0,1.0,1,553,1.0,1.0,1.0 +codedb_trigram_tokens,q134,Insert MongoDB documents,1,1,244,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,607,1.0,1.0,1.0 +codedb_trigram_tokens,q135,Run a MySQL query,1,1,173,1.0,1.0,1.0,1,281,1.0,1.0,1.0,1,568,1.0,1.0,1.0 +codedb_trigram_tokens,q136,Index a document in Elasticsearch,1,1,242,1.0,1.0,1.0,1,318,1.0,1.0,1.0,1,661,1.0,1.0,1.0 +codedb_trigram_tokens,q137,Get an S3 object,2,1,249,1.0,1.0,1.0,1,301,1.0,1.0,1.0,1,728,1.0,1.0,1.0 +codedb_trigram_tokens,q138,List Cloudflare zones,1,1,230,1.0,1.0,1.0,1,363,1.0,1.0,1.0,1,613,1.0,1.0,1.0 +codedb_trigram_tokens,q139,Create a Stripe invoice,1,1,182,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,547,1.0,1.0,1.0 +codedb_trigram_tokens,q140,List GitHub Actions workflows,1,1,268,1.0,1.0,1.0,1,408,1.0,1.0,1.0,1,765,1.0,1.0,1.0 +codedb_trigram_tokens,q141,Get GitHub Actions job logs,1,1,296,1.0,1.0,1.0,1,396,1.0,1.0,1.0,1,723,1.0,1.0,1.0 +codedb_trigram_tokens,q142,Create a GitHub repository,6,2,421,0.0,1.0,1.0,2,712,0.0,1.0,1.0,1,691,1.0,1.0,1.0 +codedb_trigram_tokens,q143,Checkout a git branch,2,1,189,1.0,1.0,1.0,1,373,1.0,1.0,1.0,1,739,1.0,1.0,1.0 +codedb_trigram_tokens,q144,Show git log,5,2,364,0.0,1.0,1.0,1,294,1.0,1.0,1.0,1,694,1.0,1.0,1.0 +codedb_trigram_tokens,q145,List postgres schemas,1,1,169,1.0,1.0,1.0,1,277,1.0,1.0,1.0,1,527,1.0,1.0,1.0 +codedb_trigram_tokens,q146,Open nodes in the knowledge graph,2,1,205,1.0,1.0,1.0,1,308,1.0,1.0,1.0,1,570,1.0,1.0,1.0 +codedb_trigram_tokens,q147,Get file metadata from the filesystem,8,3,723,0.0,0.0,1.0,2,752,0.0,1.0,1.0,1,730,1.0,1.0,1.0 +codedb_trigram_tokens,q148,Write a new local file,87,29,7985,0.0,0.0,0.0,18,7922,0.0,0.0,0.0,9,7733,0.0,0.0,0.0 +codedb_trigram_tokens,q149,Search GitHub issues,8,3,679,0.0,0.0,1.0,2,744,0.0,1.0,1.0,1,724,1.0,1.0,1.0 +codedb_trigram_tokens,q150,Get the authenticated GitHub user,8,3,614,0.0,0.0,1.0,2,644,0.0,1.0,1.0,1,622,1.0,1.0,1.0 +codedb_word,q001,Create a GitHub issue,1,1,403,1.0,1.0,1.0,1,639,1.0,1.0,1.0,1,1048,1.0,1.0,1.0 +codedb_word,q002,Create a Jira ticket,1,1,265,1.0,1.0,1.0,1,385,1.0,1.0,1.0,1,742,1.0,1.0,1.0 +codedb_word,q003,Send a Slack message,2,1,249,1.0,1.0,1.0,1,416,1.0,1.0,1.0,1,813,1.0,1.0,1.0 +codedb_word,q004,List open pull requests,2,1,237,1.0,1.0,1.0,1,530,1.0,1.0,1.0,1,1113,1.0,1.0,1.0 +codedb_word,q005,Open a pull request on GitHub,1,1,417,1.0,1.0,1.0,1,713,1.0,1.0,1.0,1,1200,1.0,1.0,1.0 +codedb_word,q006,Run a SQL query against postgres,1,1,165,1.0,1.0,1.0,1,281,1.0,1.0,1.0,1,546,1.0,1.0,1.0 +codedb_word,q007,Read a local file from disk,2,1,223,1.0,1.0,1.0,1,444,1.0,1.0,1.0,1,848,1.0,1.0,1.0 +codedb_word,q008,What files are in this folder,4,2,427,0.0,1.0,1.0,1,350,1.0,1.0,1.0,1,664,1.0,1.0,1.0 +codedb_word,q009,Remember this fact in the knowledge graph,89,30,8108,0.0,0.0,0.0,18,7832,0.0,0.0,0.0,9,7625,0.0,0.0,0.0 +codedb_word,q010,Search the web with Brave,1,1,256,1.0,1.0,1.0,1,365,1.0,1.0,1.0,1,671,1.0,1.0,1.0 +codedb_word,q011,Fetch the contents of a URL,7,3,664,0.0,0.0,1.0,2,693,0.0,1.0,1.0,1,671,1.0,1.0,1.0 +codedb_word,q012,Show git status,1,1,263,1.0,1.0,1.0,1,482,1.0,1.0,1.0,1,838,1.0,1.0,1.0 +codedb_word,q013,Commit staged changes,1,1,192,1.0,1.0,1.0,1,305,1.0,1.0,1.0,1,663,1.0,1.0,1.0 +codedb_word,q014,Create a Linear issue,3,1,322,1.0,1.0,1.0,1,463,1.0,1.0,1.0,1,871,1.0,1.0,1.0 +codedb_word,q015,Make a task in my project tracker Linear,146,49,12388,0.0,0.0,0.0,30,12128,0.0,0.0,0.0,15,11768,0.0,0.0,0.0 +codedb_word,q016,Post a comment on a GitHub issue,9,3,1027,0.0,0.0,1.0,2,1107,0.0,1.0,1.0,1,1083,1.0,1.0,1.0 +codedb_word,q017,Merge the pull request,1,1,337,1.0,1.0,1.0,1,531,1.0,1.0,1.0,1,936,1.0,1.0,1.0 +codedb_word,q018,Trigger a GitHub Actions workflow,1,1,398,1.0,1.0,1.0,1,578,1.0,1.0,1.0,1,1011,1.0,1.0,1.0 +codedb_word,q019,Get Dependabot alerts,1,1,269,1.0,1.0,1.0,1,421,1.0,1.0,1.0,1,827,1.0,1.0,1.0 +codedb_word,q020,List Sentry issues,1,1,238,1.0,1.0,1.0,1,428,1.0,1.0,1.0,1,813,1.0,1.0,1.0 +codedb_word,q021,Create a Stripe customer,6,2,407,0.0,1.0,1.0,2,678,0.0,1.0,1.0,1,658,1.0,1.0,1.0 +codedb_word,q022,Refund a Stripe payment,1,1,234,1.0,1.0,1.0,1,379,1.0,1.0,1.0,1,660,1.0,1.0,1.0 +codedb_word,q023,Send an email with Gmail,1,1,237,1.0,1.0,1.0,1,392,1.0,1.0,1.0,1,775,1.0,1.0,1.0 +codedb_word,q024,Search my Gmail inbox,9,3,642,0.0,0.0,1.0,2,676,0.0,1.0,1.0,1,656,1.0,1.0,1.0 +codedb_word,q025,Upload a file to Google Drive,257,86,20636,0.0,0.0,0.0,52,20030,0.0,0.0,0.0,26,19458,0.0,0.0,0.0 +codedb_word,q026,Create a calendar event,3,1,300,1.0,1.0,1.0,1,481,1.0,1.0,1.0,1,820,1.0,1.0,1.0 +codedb_word,q027,List Kubernetes pods,1,1,262,1.0,1.0,1.0,1,416,1.0,1.0,1.0,1,719,1.0,1.0,1.0 +codedb_word,q028,Get logs from a k8s pod,1,1,283,1.0,1.0,1.0,1,459,1.0,1.0,1.0,1,913,1.0,1.0,1.0 +codedb_word,q029,Run a docker container,1,1,219,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,641,1.0,1.0,1.0 +codedb_word,q030,Show docker container logs,1,1,235,1.0,1.0,1.0,1,379,1.0,1.0,1.0,1,763,1.0,1.0,1.0 +codedb_word,q031,Navigate the browser to a URL,1,1,186,1.0,1.0,1.0,1,300,1.0,1.0,1.0,1,561,1.0,1.0,1.0 +codedb_word,q032,Take a screenshot of the page,1,1,227,1.0,1.0,1.0,1,442,1.0,1.0,1.0,1,730,1.0,1.0,1.0 +codedb_word,q033,Create a GitLab merge request,1,1,291,1.0,1.0,1.0,1,441,1.0,1.0,1.0,1,833,1.0,1.0,1.0 +codedb_word,q034,Search Notion pages,1,1,242,1.0,1.0,1.0,1,393,1.0,1.0,1.0,1,734,1.0,1.0,1.0 +codedb_word,q035,Create a Notion page,3,1,232,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,682,1.0,1.0,1.0 +codedb_word,q036,Query a Notion database,2,1,279,1.0,1.0,1.0,1,450,1.0,1.0,1.0,1,821,1.0,1.0,1.0 +codedb_word,q037,Create a Zendesk ticket,3,1,257,1.0,1.0,1.0,1,429,1.0,1.0,1.0,1,733,1.0,1.0,1.0 +codedb_word,q038,Create a HubSpot contact,2,1,221,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,873,1.0,1.0,1.0 +codedb_word,q039,Create a Salesforce lead,1,1,259,1.0,1.0,1.0,1,423,1.0,1.0,1.0,1,696,1.0,1.0,1.0 +codedb_word,q040,Run a SOQL query,1,1,173,1.0,1.0,1.0,1,331,1.0,1.0,1.0,1,746,1.0,1.0,1.0 +codedb_word,q041,List Shopify orders,1,1,230,1.0,1.0,1.0,1,376,1.0,1.0,1.0,1,645,1.0,1.0,1.0 +codedb_word,q042,Create a PagerDuty incident,4,2,486,0.0,1.0,1.0,1,387,1.0,1.0,1.0,1,715,1.0,1.0,1.0 +codedb_word,q043,Who is on call in PagerDuty,6,2,442,0.0,1.0,1.0,2,833,0.0,1.0,1.0,1,810,1.0,1.0,1.0 +codedb_word,q044,Query Datadog metrics,1,1,287,1.0,1.0,1.0,1,449,1.0,1.0,1.0,1,758,1.0,1.0,1.0 +codedb_word,q045,Search Datadog logs,1,1,283,1.0,1.0,1.0,1,492,1.0,1.0,1.0,1,828,1.0,1.0,1.0 +codedb_word,q046,List S3 buckets,1,1,153,1.0,1.0,1.0,1,311,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +codedb_word,q047,Upload an object to S3,1,1,250,1.0,1.0,1.0,1,423,1.0,1.0,1.0,1,740,1.0,1.0,1.0 +codedb_word,q048,Purge Cloudflare cache,1,1,231,1.0,1.0,1.0,1,340,1.0,1.0,1.0,1,746,1.0,1.0,1.0 +codedb_word,q049,Create a Cloudflare DNS record,1,1,217,1.0,1.0,1.0,1,363,1.0,1.0,1.0,1,714,1.0,1.0,1.0 +codedb_word,q050,List Vercel deployments,1,1,177,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,648,1.0,1.0,1.0 +codedb_word,q051,Insert a row into Supabase,1,1,240,1.0,1.0,1.0,1,332,1.0,1.0,1.0,1,768,1.0,1.0,1.0 +codedb_word,q052,Export images from Figma,1,1,256,1.0,1.0,1.0,1,431,1.0,1.0,1.0,1,789,1.0,1.0,1.0 +codedb_word,q053,Send an SMS with Twilio,2,1,232,1.0,1.0,1.0,1,379,1.0,1.0,1.0,1,704,1.0,1.0,1.0 +codedb_word,q054,Create an Okta user,1,1,183,1.0,1.0,1.0,1,366,1.0,1.0,1.0,1,625,1.0,1.0,1.0 +codedb_word,q055,Instant query Prometheus,1,1,221,1.0,1.0,1.0,1,313,1.0,1.0,1.0,1,749,1.0,1.0,1.0 +codedb_word,q056,Search Grafana dashboards,1,1,217,1.0,1.0,1.0,1,309,1.0,1.0,1.0,1,745,1.0,1.0,1.0 +codedb_word,q057,Trigger a CircleCI pipeline,1,1,200,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,747,1.0,1.0,1.0 +codedb_word,q058,Run terraform plan,1,1,166,1.0,1.0,1.0,1,281,1.0,1.0,1.0,1,687,1.0,1.0,1.0 +codedb_word,q059,Create an Asana task,3,1,267,1.0,1.0,1.0,1,428,1.0,1.0,1.0,1,824,1.0,1.0,1.0 +codedb_word,q060,Create a Trello card,2,1,240,1.0,1.0,1.0,1,370,1.0,1.0,1.0,1,776,1.0,1.0,1.0 +codedb_word,q061,Create a ClickUp task,2,1,257,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +codedb_word,q062,Create a Confluence page,2,1,260,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,710,1.0,1.0,1.0 +codedb_word,q063,Send a Discord message,2,1,265,1.0,1.0,1.0,1,363,1.0,1.0,1.0,1,763,1.0,1.0,1.0 +codedb_word,q064,Search Elasticsearch,2,1,239,1.0,1.0,1.0,1,331,1.0,1.0,1.0,1,767,1.0,1.0,1.0 +codedb_word,q065,Get a Redis key,1,1,185,1.0,1.0,1.0,1,310,1.0,1.0,1.0,1,639,1.0,1.0,1.0 +codedb_word,q066,Find MongoDB documents,2,1,258,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,738,1.0,1.0,1.0 +codedb_word,q067,List SQLite tables,1,1,156,1.0,1.0,1.0,1,252,1.0,1.0,1.0,1,542,1.0,1.0,1.0 +codedb_word,q068,Describe a postgres table,1,1,170,1.0,1.0,1.0,1,286,1.0,1.0,1.0,1,597,1.0,1.0,1.0 +codedb_word,q069,github_create_issue,1,1,392,1.0,1.0,1.0,1,596,1.0,1.0,1.0,1,1087,1.0,1.0,1.0 +codedb_word,q070,createIssue,6,2,616,0.0,1.0,1.0,2,976,0.0,1.0,1.0,1,958,1.0,1.0,1.0 +codedb_word,q071,list-directory,2,1,234,1.0,1.0,1.0,1,344,1.0,1.0,1.0,1,653,1.0,1.0,1.0 +codedb_word,q072,sequential thinking next thought,1,1,224,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,797,1.0,1.0,1.0 +codedb_word,q073,What time is it in Tokyo,1,1,262,1.0,1.0,1.0,1,369,1.0,1.0,1.0,1,685,1.0,1.0,1.0 +codedb_word,q074,Convert time between timezones,1,1,244,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,704,1.0,1.0,1.0 +codedb_word,q075,Star a GitHub repo,1,1,345,1.0,1.0,1.0,1,525,1.0,1.0,1.0,1,963,1.0,1.0,1.0 +codedb_word,q076,Fork a repository,1,1,287,1.0,1.0,1.0,1,426,1.0,1.0,1.0,1,845,1.0,1.0,1.0 +codedb_word,q077,Get the latest GitHub release,1,1,210,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +codedb_word,q078,Create a gist,1,1,229,1.0,1.0,1.0,1,321,1.0,1.0,1.0,1,757,1.0,1.0,1.0 +codedb_word,q079,Search GitHub code,1,1,300,1.0,1.0,1.0,1,536,1.0,1.0,1.0,1,995,1.0,1.0,1.0 +codedb_word,q080,List GitHub notifications,1,1,237,1.0,1.0,1.0,1,473,1.0,1.0,1.0,1,913,1.0,1.0,1.0 +codedb_word,q081,Assign Copilot to an issue,1,1,301,1.0,1.0,1.0,1,448,1.0,1.0,1.0,1,887,1.0,1.0,1.0 +codedb_word,q082,Get secret scanning alerts,1,1,295,1.0,1.0,1.0,1,470,1.0,1.0,1.0,1,827,1.0,1.0,1.0 +codedb_word,q083,Create or update a file on GitHub,9,3,798,0.0,0.0,1.0,2,844,0.0,1.0,1.0,1,820,1.0,1.0,1.0 +codedb_word,q084,List collaborators on a repo,1,1,342,1.0,1.0,1.0,1,550,1.0,1.0,1.0,1,1085,1.0,1.0,1.0 +codedb_word,q085,Reply in a Slack thread,2,1,247,1.0,1.0,1.0,1,375,1.0,1.0,1.0,1,769,1.0,1.0,1.0 +codedb_word,q086,Add an emoji reaction on Slack,1,1,293,1.0,1.0,1.0,1,399,1.0,1.0,1.0,1,777,1.0,1.0,1.0 +codedb_word,q087,Search Slack messages,1,1,251,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,704,1.0,1.0,1.0 +codedb_word,q088,Create a Slack channel,5,2,507,0.0,1.0,1.0,1,414,1.0,1.0,1.0,1,751,1.0,1.0,1.0 +codedb_word,q089,Transition a Jira issue,1,1,231,1.0,1.0,1.0,1,414,1.0,1.0,1.0,1,823,1.0,1.0,1.0 +codedb_word,q090,Search Jira with JQL,1,1,218,1.0,1.0,1.0,1,324,1.0,1.0,1.0,1,636,1.0,1.0,1.0 +codedb_word,q091,Add a worklog to Jira,1,1,249,1.0,1.0,1.0,1,411,1.0,1.0,1.0,1,777,1.0,1.0,1.0 +codedb_word,q092,List Linear teams,1,1,208,1.0,1.0,1.0,1,302,1.0,1.0,1.0,1,773,1.0,1.0,1.0 +codedb_word,q093,Comment on a Linear issue,3,1,256,1.0,1.0,1.0,1,378,1.0,1.0,1.0,1,778,1.0,1.0,1.0 +codedb_word,q094,Move a local file,1,1,196,1.0,1.0,1.0,1,358,1.0,1.0,1.0,1,754,1.0,1.0,1.0 +codedb_word,q095,Recursively search files by pattern,1,1,221,1.0,1.0,1.0,1,351,1.0,1.0,1.0,1,693,1.0,1.0,1.0 +codedb_word,q096,Edit a file in place with old and new string,1,1,323,1.0,1.0,1.0,1,565,1.0,1.0,1.0,1,950,1.0,1.0,1.0 +codedb_word,q097,Read the whole knowledge graph,8,3,511,0.0,0.0,1.0,2,563,0.0,1.0,1.0,1,542,1.0,1.0,1.0 +codedb_word,q098,Search memory nodes,1,1,178,1.0,1.0,1.0,1,305,1.0,1.0,1.0,1,546,1.0,1.0,1.0 +codedb_word,q099,Show unstaged git diff,1,1,190,1.0,1.0,1.0,1,447,1.0,1.0,1.0,1,796,1.0,1.0,1.0 +codedb_word,q100,Stage files with git add,115,39,9731,0.0,0.0,0.0,23,9285,0.0,0.0,0.0,12,9371,0.0,0.0,0.0 +codedb_word,q101,Create a Shopify product,2,1,240,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +codedb_word,q102,Cancel a Stripe subscription,1,1,193,1.0,1.0,1.0,1,321,1.0,1.0,1.0,1,660,1.0,1.0,1.0 +codedb_word,q103,Create a payment intent,2,1,234,1.0,1.0,1.0,1,326,1.0,1.0,1.0,1,762,1.0,1.0,1.0 +codedb_word,q104,Acknowledge a PagerDuty incident,1,1,219,1.0,1.0,1.0,1,363,1.0,1.0,1.0,1,717,1.0,1.0,1.0 +codedb_word,q105,Create a Datadog monitor,1,1,297,1.0,1.0,1.0,1,459,1.0,1.0,1.0,1,816,1.0,1.0,1.0 +codedb_word,q106,Resolve a Sentry issue,1,1,251,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,797,1.0,1.0,1.0 +codedb_word,q107,Scale a Kubernetes deployment,1,1,266,1.0,1.0,1.0,1,420,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +codedb_word,q108,Apply a Kubernetes manifest,1,1,246,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +codedb_word,q109,Stop a docker container,1,1,219,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,641,1.0,1.0,1.0 +codedb_word,q110,Build a docker image,1,1,238,1.0,1.0,1.0,1,342,1.0,1.0,1.0,1,601,1.0,1.0,1.0 +codedb_word,q111,Click an element in Playwright,2,1,213,1.0,1.0,1.0,1,380,1.0,1.0,1.0,1,760,1.0,1.0,1.0 +codedb_word,q112,Type text into a browser input,1,1,296,1.0,1.0,1.0,1,429,1.0,1.0,1.0,1,862,1.0,1.0,1.0 +codedb_word,q113,List GitLab projects,1,1,254,1.0,1.0,1.0,1,398,1.0,1.0,1.0,1,797,1.0,1.0,1.0 +codedb_word,q114,Create a Bitbucket pull request,1,1,365,1.0,1.0,1.0,1,669,1.0,1.0,1.0,1,1202,1.0,1.0,1.0 +codedb_word,q115,Create a Gmail draft,1,1,344,1.0,1.0,1.0,1,494,1.0,1.0,1.0,1,744,1.0,1.0,1.0 +codedb_word,q116,Share a Google Drive file,1,1,270,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,791,1.0,1.0,1.0 +codedb_word,q117,List Google Calendar events,1,1,269,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,770,1.0,1.0,1.0 +codedb_word,q118,Delete a calendar event,1,1,300,1.0,1.0,1.0,1,481,1.0,1.0,1.0,1,820,1.0,1.0,1.0 +codedb_word,q119,List Vercel projects,1,1,170,1.0,1.0,1.0,1,262,1.0,1.0,1.0,1,567,1.0,1.0,1.0 +codedb_word,q120,Select rows from Supabase,1,1,269,1.0,1.0,1.0,1,425,1.0,1.0,1.0,1,830,1.0,1.0,1.0 +codedb_word,q121,Post a Figma comment,1,1,247,1.0,1.0,1.0,1,391,1.0,1.0,1.0,1,871,1.0,1.0,1.0 +codedb_word,q122,List Okta users,1,1,188,1.0,1.0,1.0,1,317,1.0,1.0,1.0,1,625,1.0,1.0,1.0 +codedb_word,q123,Range query Prometheus,1,1,221,1.0,1.0,1.0,1,313,1.0,1.0,1.0,1,749,1.0,1.0,1.0 +codedb_word,q124,Create a Grafana annotation,1,1,218,1.0,1.0,1.0,1,310,1.0,1.0,1.0,1,746,1.0,1.0,1.0 +codedb_word,q125,Apply terraform,1,1,194,1.0,1.0,1.0,1,292,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +codedb_word,q126,List Asana tasks,1,1,192,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,707,1.0,1.0,1.0 +codedb_word,q127,Comment on a Trello card,1,1,242,1.0,1.0,1.0,1,444,1.0,1.0,1.0,1,835,1.0,1.0,1.0 +codedb_word,q128,Update a ClickUp task,1,1,257,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +codedb_word,q129,Search Confluence,4,2,423,0.0,1.0,1.0,1,366,1.0,1.0,1.0,1,772,1.0,1.0,1.0 +codedb_word,q130,Create a HubSpot deal,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,873,1.0,1.0,1.0 +codedb_word,q131,List Zendesk tickets,2,1,189,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,724,1.0,1.0,1.0 +codedb_word,q132,List Discord channels,1,1,223,1.0,1.0,1.0,1,339,1.0,1.0,1.0,1,638,1.0,1.0,1.0 +codedb_word,q133,Set a Redis key,2,1,185,1.0,1.0,1.0,1,310,1.0,1.0,1.0,1,639,1.0,1.0,1.0 +codedb_word,q134,Insert MongoDB documents,1,1,244,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +codedb_word,q135,Run a MySQL query,1,1,158,1.0,1.0,1.0,1,271,1.0,1.0,1.0,1,523,1.0,1.0,1.0 +codedb_word,q136,Index a document in Elasticsearch,1,1,254,1.0,1.0,1.0,1,388,1.0,1.0,1.0,1,740,1.0,1.0,1.0 +codedb_word,q137,Get an S3 object,1,1,249,1.0,1.0,1.0,1,432,1.0,1.0,1.0,1,712,1.0,1.0,1.0 +codedb_word,q138,List Cloudflare zones,1,1,230,1.0,1.0,1.0,1,339,1.0,1.0,1.0,1,745,1.0,1.0,1.0 +codedb_word,q139,Create a Stripe invoice,1,1,191,1.0,1.0,1.0,1,339,1.0,1.0,1.0,1,660,1.0,1.0,1.0 +codedb_word,q140,List GitHub Actions workflows,4,2,672,0.0,1.0,1.0,1,558,1.0,1.0,1.0,1,1032,1.0,1.0,1.0 +codedb_word,q141,Get GitHub Actions job logs,2,1,349,1.0,1.0,1.0,1,559,1.0,1.0,1.0,1,1033,1.0,1.0,1.0 +codedb_word,q142,Create a GitHub repository,24,8,2112,0.0,0.0,0.0,5,2116,0.0,0.0,0.0,3,2504,0.0,0.0,1.0 +codedb_word,q143,Checkout a git branch,1,1,242,1.0,1.0,1.0,1,504,1.0,1.0,1.0,1,1049,1.0,1.0,1.0 +codedb_word,q144,Show git log,1,1,338,1.0,1.0,1.0,1,492,1.0,1.0,1.0,1,900,1.0,1.0,1.0 +codedb_word,q145,List postgres schemas,1,1,169,1.0,1.0,1.0,1,277,1.0,1.0,1.0,1,634,1.0,1.0,1.0 +codedb_word,q146,Open nodes in the knowledge graph,2,1,180,1.0,1.0,1.0,1,297,1.0,1.0,1.0,1,523,1.0,1.0,1.0 +codedb_word,q147,Get file metadata from the filesystem,5,2,525,0.0,1.0,1.0,1,412,1.0,1.0,1.0,1,810,1.0,1.0,1.0 +codedb_word,q148,Write a new local file,2,1,206,1.0,1.0,1.0,1,384,1.0,1.0,1.0,1,847,1.0,1.0,1.0 +codedb_word,q149,Search GitHub issues,2,1,322,1.0,1.0,1.0,1,558,1.0,1.0,1.0,1,998,1.0,1.0,1.0 +codedb_word,q150,Get the authenticated GitHub user,2,1,179,1.0,1.0,1.0,1,302,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +decision_heuristic,q001,Create a GitHub issue,1,1,388,1.0,1.0,1.0,1,598,1.0,1.0,1.0,1,955,1.0,1.0,1.0 +decision_heuristic,q002,Create a Jira ticket,1,1,312,1.0,1.0,1.0,1,412,1.0,1.0,1.0,1,763,1.0,1.0,1.0 +decision_heuristic,q003,Send a Slack message,1,1,232,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,707,1.0,1.0,1.0 +decision_heuristic,q004,List open pull requests,1,1,234,1.0,1.0,1.0,1,429,1.0,1.0,1.0,1,1035,1.0,1.0,1.0 +decision_heuristic,q005,Open a pull request on GitHub,1,1,368,1.0,1.0,1.0,1,653,1.0,1.0,1.0,1,1052,1.0,1.0,1.0 +decision_heuristic,q006,Run a SQL query against postgres,1,1,167,1.0,1.0,1.0,1,280,1.0,1.0,1.0,1,590,1.0,1.0,1.0 +decision_heuristic,q007,Read a local file from disk,1,1,223,1.0,1.0,1.0,1,385,1.0,1.0,1.0,1,782,1.0,1.0,1.0 +decision_heuristic,q008,What files are in this folder,6,2,473,0.0,1.0,1.0,2,730,0.0,1.0,1.0,1,708,1.0,1.0,1.0 +decision_heuristic,q009,Remember this fact in the knowledge graph,2,1,154,1.0,1.0,1.0,1,261,1.0,1.0,1.0,1,524,1.0,1.0,1.0 +decision_heuristic,q010,Search the web with Brave,1,1,245,1.0,1.0,1.0,1,412,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +decision_heuristic,q011,Fetch the contents of a URL,1,1,258,1.0,1.0,1.0,1,372,1.0,1.0,1.0,1,707,1.0,1.0,1.0 +decision_heuristic,q012,Show git status,1,1,196,1.0,1.0,1.0,1,304,1.0,1.0,1.0,1,600,1.0,1.0,1.0 +decision_heuristic,q013,Commit staged changes,1,1,218,1.0,1.0,1.0,1,328,1.0,1.0,1.0,1,663,1.0,1.0,1.0 +decision_heuristic,q014,Create a Linear issue,1,1,340,1.0,1.0,1.0,1,463,1.0,1.0,1.0,1,765,1.0,1.0,1.0 +decision_heuristic,q015,Make a task in my project tracker Linear,2,1,352,1.0,1.0,1.0,1,432,1.0,1.0,1.0,1,769,1.0,1.0,1.0 +decision_heuristic,q016,Post a comment on a GitHub issue,2,1,333,1.0,1.0,1.0,1,601,1.0,1.0,1.0,1,938,1.0,1.0,1.0 +decision_heuristic,q017,Merge the pull request,1,1,283,1.0,1.0,1.0,1,523,1.0,1.0,1.0,1,1121,1.0,1.0,1.0 +decision_heuristic,q018,Trigger a GitHub Actions workflow,1,1,294,1.0,1.0,1.0,1,450,1.0,1.0,1.0,1,737,1.0,1.0,1.0 +decision_heuristic,q019,Get Dependabot alerts,1,1,287,1.0,1.0,1.0,1,445,1.0,1.0,1.0,1,798,1.0,1.0,1.0 +decision_heuristic,q020,List Sentry issues,1,1,238,1.0,1.0,1.0,1,396,1.0,1.0,1.0,1,796,1.0,1.0,1.0 +decision_heuristic,q021,Create a Stripe customer,2,1,186,1.0,1.0,1.0,1,300,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +decision_heuristic,q022,Refund a Stripe payment,1,1,234,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,662,1.0,1.0,1.0 +decision_heuristic,q023,Send an email with Gmail,1,1,224,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,703,1.0,1.0,1.0 +decision_heuristic,q024,Search my Gmail inbox,1,1,205,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,714,1.0,1.0,1.0 +decision_heuristic,q025,Upload a file to Google Drive,4,2,496,0.0,1.0,1.0,1,418,1.0,1.0,1.0,1,791,1.0,1.0,1.0 +decision_heuristic,q026,Create a calendar event,2,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,895,1.0,1.0,1.0 +decision_heuristic,q027,List Kubernetes pods,1,1,184,1.0,1.0,1.0,1,314,1.0,1.0,1.0,1,718,1.0,1.0,1.0 +decision_heuristic,q028,Get logs from a k8s pod,1,1,262,1.0,1.0,1.0,1,455,1.0,1.0,1.0,1,834,1.0,1.0,1.0 +decision_heuristic,q029,Run a docker container,1,1,219,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,644,1.0,1.0,1.0 +decision_heuristic,q030,Show docker container logs,1,1,188,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,686,1.0,1.0,1.0 +decision_heuristic,q031,Navigate the browser to a URL,1,1,180,1.0,1.0,1.0,1,265,1.0,1.0,1.0,1,536,1.0,1.0,1.0 +decision_heuristic,q032,Take a screenshot of the page,1,1,226,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,711,1.0,1.0,1.0 +decision_heuristic,q033,Create a GitLab merge request,3,1,297,1.0,1.0,1.0,1,475,1.0,1.0,1.0,1,792,1.0,1.0,1.0 +decision_heuristic,q034,Search Notion pages,1,1,195,1.0,1.0,1.0,1,366,1.0,1.0,1.0,1,686,1.0,1.0,1.0 +decision_heuristic,q035,Create a Notion page,3,1,230,1.0,1.0,1.0,1,371,1.0,1.0,1.0,1,687,1.0,1.0,1.0 +decision_heuristic,q036,Query a Notion database,1,1,279,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,687,1.0,1.0,1.0 +decision_heuristic,q037,Create a Zendesk ticket,1,1,257,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,856,1.0,1.0,1.0 +decision_heuristic,q038,Create a HubSpot contact,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,904,1.0,1.0,1.0 +decision_heuristic,q039,Create a Salesforce lead,2,1,252,1.0,1.0,1.0,1,402,1.0,1.0,1.0,1,787,1.0,1.0,1.0 +decision_heuristic,q040,Run a SOQL query,1,1,186,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,724,1.0,1.0,1.0 +decision_heuristic,q041,List Shopify orders,1,1,208,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,682,1.0,1.0,1.0 +decision_heuristic,q042,Create a PagerDuty incident,1,1,259,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,838,1.0,1.0,1.0 +decision_heuristic,q043,Who is on call in PagerDuty,1,1,227,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,721,1.0,1.0,1.0 +decision_heuristic,q044,Query Datadog metrics,1,1,300,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,728,1.0,1.0,1.0 +decision_heuristic,q045,Search Datadog logs,1,1,242,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,877,1.0,1.0,1.0 +decision_heuristic,q046,List S3 buckets,1,1,186,1.0,1.0,1.0,1,318,1.0,1.0,1.0,1,667,1.0,1.0,1.0 +decision_heuristic,q047,Upload an object to S3,1,1,250,1.0,1.0,1.0,1,437,1.0,1.0,1.0,1,950,1.0,1.0,1.0 +decision_heuristic,q048,Purge Cloudflare cache,1,1,231,1.0,1.0,1.0,1,340,1.0,1.0,1.0,1,746,1.0,1.0,1.0 +decision_heuristic,q049,Create a Cloudflare DNS record,1,1,217,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,738,1.0,1.0,1.0 +decision_heuristic,q050,List Vercel deployments,1,1,177,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,579,1.0,1.0,1.0 +decision_heuristic,q051,Insert a row into Supabase,1,1,238,1.0,1.0,1.0,1,355,1.0,1.0,1.0,1,665,1.0,1.0,1.0 +decision_heuristic,q052,Export images from Figma,1,1,239,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,658,1.0,1.0,1.0 +decision_heuristic,q053,Send an SMS with Twilio,1,1,249,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,740,1.0,1.0,1.0 +decision_heuristic,q054,Create an Okta user,3,1,183,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,656,1.0,1.0,1.0 +decision_heuristic,q055,Instant query Prometheus,1,1,221,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,680,1.0,1.0,1.0 +decision_heuristic,q056,Search Grafana dashboards,1,1,217,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,690,1.0,1.0,1.0 +decision_heuristic,q057,Trigger a CircleCI pipeline,1,1,200,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,755,1.0,1.0,1.0 +decision_heuristic,q058,Run terraform plan,1,1,172,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,717,1.0,1.0,1.0 +decision_heuristic,q059,Create an Asana task,2,1,267,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,825,1.0,1.0,1.0 +decision_heuristic,q060,Create a Trello card,1,1,240,1.0,1.0,1.0,1,394,1.0,1.0,1.0,1,884,1.0,1.0,1.0 +decision_heuristic,q061,Create a ClickUp task,2,1,257,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,751,1.0,1.0,1.0 +decision_heuristic,q062,Create a Confluence page,1,1,260,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,703,1.0,1.0,1.0 +decision_heuristic,q063,Send a Discord message,1,1,224,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,682,1.0,1.0,1.0 +decision_heuristic,q064,Search Elasticsearch,1,1,239,1.0,1.0,1.0,1,404,1.0,1.0,1.0,1,699,1.0,1.0,1.0 +decision_heuristic,q065,Get a Redis key,1,1,182,1.0,1.0,1.0,1,280,1.0,1.0,1.0,1,589,1.0,1.0,1.0 +decision_heuristic,q066,Find MongoDB documents,1,1,258,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,738,1.0,1.0,1.0 +decision_heuristic,q067,List SQLite tables,1,1,151,1.0,1.0,1.0,1,240,1.0,1.0,1.0,1,514,1.0,1.0,1.0 +decision_heuristic,q068,Describe a postgres table,1,1,186,1.0,1.0,1.0,1,272,1.0,1.0,1.0,1,597,1.0,1.0,1.0 +decision_heuristic,q069,github_create_issue,1,1,386,1.0,1.0,1.0,1,596,1.0,1.0,1.0,1,1019,1.0,1.0,1.0 +decision_heuristic,q070,createIssue,1,1,340,1.0,1.0,1.0,1,546,1.0,1.0,1.0,1,961,1.0,1.0,1.0 +decision_heuristic,q071,list-directory,2,1,182,1.0,1.0,1.0,1,296,1.0,1.0,1.0,1,607,1.0,1.0,1.0 +decision_heuristic,q072,sequential thinking next thought,1,1,224,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,797,1.0,1.0,1.0 +decision_heuristic,q073,What time is it in Tokyo,1,1,217,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,783,1.0,1.0,1.0 +decision_heuristic,q074,Convert time between timezones,1,1,204,1.0,1.0,1.0,1,335,1.0,1.0,1.0,1,704,1.0,1.0,1.0 +decision_heuristic,q075,Star a GitHub repo,1,1,195,1.0,1.0,1.0,1,309,1.0,1.0,1.0,1,604,1.0,1.0,1.0 +decision_heuristic,q076,Fork a repository,1,1,235,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,792,1.0,1.0,1.0 +decision_heuristic,q077,Get the latest GitHub release,1,1,249,1.0,1.0,1.0,1,366,1.0,1.0,1.0,1,647,1.0,1.0,1.0 +decision_heuristic,q078,Create a gist,1,1,229,1.0,1.0,1.0,1,429,1.0,1.0,1.0,1,912,1.0,1.0,1.0 +decision_heuristic,q079,Search GitHub code,1,1,283,1.0,1.0,1.0,1,406,1.0,1.0,1.0,1,697,1.0,1.0,1.0 +decision_heuristic,q080,List GitHub notifications,1,1,169,1.0,1.0,1.0,1,292,1.0,1.0,1.0,1,585,1.0,1.0,1.0 +decision_heuristic,q081,Assign Copilot to an issue,1,1,267,1.0,1.0,1.0,1,434,1.0,1.0,1.0,1,928,1.0,1.0,1.0 +decision_heuristic,q082,Get secret scanning alerts,1,1,295,1.0,1.0,1.0,1,470,1.0,1.0,1.0,1,830,1.0,1.0,1.0 +decision_heuristic,q083,Create or update a file on GitHub,1,1,347,1.0,1.0,1.0,1,519,1.0,1.0,1.0,1,869,1.0,1.0,1.0 +decision_heuristic,q084,List collaborators on a repo,1,1,278,1.0,1.0,1.0,1,433,1.0,1.0,1.0,1,816,1.0,1.0,1.0 +decision_heuristic,q085,Reply in a Slack thread,1,1,254,1.0,1.0,1.0,1,358,1.0,1.0,1.0,1,717,1.0,1.0,1.0 +decision_heuristic,q086,Add an emoji reaction on Slack,1,1,216,1.0,1.0,1.0,1,357,1.0,1.0,1.0,1,709,1.0,1.0,1.0 +decision_heuristic,q087,Search Slack messages,1,1,213,1.0,1.0,1.0,1,337,1.0,1.0,1.0,1,689,1.0,1.0,1.0 +decision_heuristic,q088,Create a Slack channel,3,1,196,1.0,1.0,1.0,1,342,1.0,1.0,1.0,1,719,1.0,1.0,1.0 +decision_heuristic,q089,Transition a Jira issue,1,1,208,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,763,1.0,1.0,1.0 +decision_heuristic,q090,Search Jira with JQL,1,1,194,1.0,1.0,1.0,1,370,1.0,1.0,1.0,1,764,1.0,1.0,1.0 +decision_heuristic,q091,Add a worklog to Jira,1,1,249,1.0,1.0,1.0,1,350,1.0,1.0,1.0,1,765,1.0,1.0,1.0 +decision_heuristic,q092,List Linear teams,1,1,184,1.0,1.0,1.0,1,380,1.0,1.0,1.0,1,764,1.0,1.0,1.0 +decision_heuristic,q093,Comment on a Linear issue,1,1,334,1.0,1.0,1.0,1,464,1.0,1.0,1.0,1,760,1.0,1.0,1.0 +decision_heuristic,q094,Move a local file,1,1,214,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,699,1.0,1.0,1.0 +decision_heuristic,q095,Recursively search files by pattern,1,1,280,1.0,1.0,1.0,1,433,1.0,1.0,1.0,1,705,1.0,1.0,1.0 +decision_heuristic,q096,Edit a file in place with old and new string,1,1,227,1.0,1.0,1.0,1,394,1.0,1.0,1.0,1,682,1.0,1.0,1.0 +decision_heuristic,q097,Read the whole knowledge graph,1,1,152,1.0,1.0,1.0,1,259,1.0,1.0,1.0,1,529,1.0,1.0,1.0 +decision_heuristic,q098,Search memory nodes,1,1,202,1.0,1.0,1.0,1,301,1.0,1.0,1.0,1,546,1.0,1.0,1.0 +decision_heuristic,q099,Show unstaged git diff,1,1,190,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,602,1.0,1.0,1.0 +decision_heuristic,q100,Stage files with git add,3,1,213,1.0,1.0,1.0,1,336,1.0,1.0,1.0,1,677,1.0,1.0,1.0 +decision_heuristic,q101,Create a Shopify product,1,1,240,1.0,1.0,1.0,1,380,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +decision_heuristic,q102,Cancel a Stripe subscription,1,1,188,1.0,1.0,1.0,1,301,1.0,1.0,1.0,1,632,1.0,1.0,1.0 +decision_heuristic,q103,Create a payment intent,1,1,234,1.0,1.0,1.0,1,417,1.0,1.0,1.0,1,897,1.0,1.0,1.0 +decision_heuristic,q104,Acknowledge a PagerDuty incident,1,1,261,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,717,1.0,1.0,1.0 +decision_heuristic,q105,Create a Datadog monitor,2,1,259,1.0,1.0,1.0,1,459,1.0,1.0,1.0,1,927,1.0,1.0,1.0 +decision_heuristic,q106,Resolve a Sentry issue,2,1,251,1.0,1.0,1.0,1,397,1.0,1.0,1.0,1,817,1.0,1.0,1.0 +decision_heuristic,q107,Scale a Kubernetes deployment,1,1,263,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +decision_heuristic,q108,Apply a Kubernetes manifest,1,1,253,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +decision_heuristic,q109,Stop a docker container,1,1,188,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,641,1.0,1.0,1.0 +decision_heuristic,q110,Build a docker image,1,1,224,1.0,1.0,1.0,1,317,1.0,1.0,1.0,1,601,1.0,1.0,1.0 +decision_heuristic,q111,Click an element in Playwright,1,1,216,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,566,1.0,1.0,1.0 +decision_heuristic,q112,Type text into a browser input,1,1,217,1.0,1.0,1.0,1,302,1.0,1.0,1.0,1,554,1.0,1.0,1.0 +decision_heuristic,q113,List GitLab projects,1,1,179,1.0,1.0,1.0,1,304,1.0,1.0,1.0,1,599,1.0,1.0,1.0 +decision_heuristic,q114,Create a Bitbucket pull request,1,1,313,1.0,1.0,1.0,1,471,1.0,1.0,1.0,1,1091,1.0,1.0,1.0 +decision_heuristic,q115,Create a Gmail draft,1,1,205,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,835,1.0,1.0,1.0 +decision_heuristic,q116,Share a Google Drive file,1,1,254,1.0,1.0,1.0,1,417,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +decision_heuristic,q117,List Google Calendar events,1,1,269,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,784,1.0,1.0,1.0 +decision_heuristic,q118,Delete a calendar event,1,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,751,1.0,1.0,1.0 +decision_heuristic,q119,List Vercel projects,1,1,177,1.0,1.0,1.0,1,301,1.0,1.0,1.0,1,550,1.0,1.0,1.0 +decision_heuristic,q120,Select rows from Supabase,1,1,237,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,709,1.0,1.0,1.0 +decision_heuristic,q121,Post a Figma comment,1,1,204,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +decision_heuristic,q122,List Okta users,1,1,177,1.0,1.0,1.0,1,306,1.0,1.0,1.0,1,606,1.0,1.0,1.0 +decision_heuristic,q123,Range query Prometheus,1,1,221,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,680,1.0,1.0,1.0 +decision_heuristic,q124,Create a Grafana annotation,1,1,218,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,881,1.0,1.0,1.0 +decision_heuristic,q125,Apply terraform,1,1,171,1.0,1.0,1.0,1,292,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +decision_heuristic,q126,List Asana tasks,1,1,244,1.0,1.0,1.0,1,404,1.0,1.0,1.0,1,682,1.0,1.0,1.0 +decision_heuristic,q127,Comment on a Trello card,1,1,260,1.0,1.0,1.0,1,422,1.0,1.0,1.0,1,757,1.0,1.0,1.0 +decision_heuristic,q128,Update a ClickUp task,1,1,257,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,761,1.0,1.0,1.0 +decision_heuristic,q129,Search Confluence,1,1,221,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,731,1.0,1.0,1.0 +decision_heuristic,q130,Create a HubSpot deal,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,904,1.0,1.0,1.0 +decision_heuristic,q131,List Zendesk tickets,1,1,189,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,645,1.0,1.0,1.0 +decision_heuristic,q132,List Discord channels,1,1,166,1.0,1.0,1.0,1,321,1.0,1.0,1.0,1,625,1.0,1.0,1.0 +decision_heuristic,q133,Set a Redis key,1,1,182,1.0,1.0,1.0,1,280,1.0,1.0,1.0,1,600,1.0,1.0,1.0 +decision_heuristic,q134,Insert MongoDB documents,1,1,256,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +decision_heuristic,q135,Run a MySQL query,3,1,182,1.0,1.0,1.0,1,268,1.0,1.0,1.0,1,543,1.0,1.0,1.0 +decision_heuristic,q136,Index a document in Elasticsearch,1,1,242,1.0,1.0,1.0,1,396,1.0,1.0,1.0,1,753,1.0,1.0,1.0 +decision_heuristic,q137,Get an S3 object,1,1,249,1.0,1.0,1.0,1,355,1.0,1.0,1.0,1,706,1.0,1.0,1.0 +decision_heuristic,q138,List Cloudflare zones,1,1,182,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,629,1.0,1.0,1.0 +decision_heuristic,q139,Create a Stripe invoice,1,1,204,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,632,1.0,1.0,1.0 +decision_heuristic,q140,List GitHub Actions workflows,1,1,293,1.0,1.0,1.0,1,433,1.0,1.0,1.0,1,736,1.0,1.0,1.0 +decision_heuristic,q141,Get GitHub Actions job logs,1,1,269,1.0,1.0,1.0,1,450,1.0,1.0,1.0,1,747,1.0,1.0,1.0 +decision_heuristic,q142,Create a GitHub repository,1,1,250,1.0,1.0,1.0,1,388,1.0,1.0,1.0,1,750,1.0,1.0,1.0 +decision_heuristic,q143,Checkout a git branch,1,1,212,1.0,1.0,1.0,1,373,1.0,1.0,1.0,1,666,1.0,1.0,1.0 +decision_heuristic,q144,Show git log,1,1,206,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,600,1.0,1.0,1.0 +decision_heuristic,q145,List postgres schemas,1,1,158,1.0,1.0,1.0,1,277,1.0,1.0,1.0,1,555,1.0,1.0,1.0 +decision_heuristic,q146,Open nodes in the knowledge graph,1,1,163,1.0,1.0,1.0,1,260,1.0,1.0,1.0,1,549,1.0,1.0,1.0 +decision_heuristic,q147,Get file metadata from the filesystem,4,2,462,0.0,1.0,1.0,1,348,1.0,1.0,1.0,1,688,1.0,1.0,1.0 +decision_heuristic,q148,Write a new local file,2,1,217,1.0,1.0,1.0,1,366,1.0,1.0,1.0,1,714,1.0,1.0,1.0 +decision_heuristic,q149,Search GitHub issues,1,1,270,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,670,1.0,1.0,1.0 +decision_heuristic,q150,Get the authenticated GitHub user,11,4,771,0.0,0.0,0.0,3,1050,0.0,0.0,1.0,2,1402,0.0,1.0,1.0 +embed_bge_small,q001,Create a GitHub issue,1,1,375,1.0,1.0,1.0,1,560,1.0,1.0,1.0,1,1034,1.0,1.0,1.0 +embed_bge_small,q002,Create a Jira ticket,1,1,345,1.0,1.0,1.0,1,493,1.0,1.0,1.0,1,902,1.0,1.0,1.0 +embed_bge_small,q003,Send a Slack message,1,1,269,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,795,1.0,1.0,1.0 +embed_bge_small,q004,List open pull requests,1,1,237,1.0,1.0,1.0,1,477,1.0,1.0,1.0,1,828,1.0,1.0,1.0 +embed_bge_small,q005,Open a pull request on GitHub,1,1,314,1.0,1.0,1.0,1,547,1.0,1.0,1.0,1,1094,1.0,1.0,1.0 +embed_bge_small,q006,Run a SQL query against postgres,1,1,180,1.0,1.0,1.0,1,295,1.0,1.0,1.0,1,596,1.0,1.0,1.0 +embed_bge_small,q007,Read a local file from disk,1,1,223,1.0,1.0,1.0,1,385,1.0,1.0,1.0,1,677,1.0,1.0,1.0 +embed_bge_small,q008,What files are in this folder,1,1,190,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,656,1.0,1.0,1.0 +embed_bge_small,q009,Remember this fact in the knowledge graph,3,1,164,1.0,1.0,1.0,1,266,1.0,1.0,1.0,1,585,1.0,1.0,1.0 +embed_bge_small,q010,Search the web with Brave,1,1,245,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,642,1.0,1.0,1.0 +embed_bge_small,q011,Fetch the contents of a URL,1,1,205,1.0,1.0,1.0,1,369,1.0,1.0,1.0,1,643,1.0,1.0,1.0 +embed_bge_small,q012,Show git status,1,1,194,1.0,1.0,1.0,1,312,1.0,1.0,1.0,1,696,1.0,1.0,1.0 +embed_bge_small,q013,Commit staged changes,1,1,182,1.0,1.0,1.0,1,314,1.0,1.0,1.0,1,640,1.0,1.0,1.0 +embed_bge_small,q014,Create a Linear issue,2,1,275,1.0,1.0,1.0,1,404,1.0,1.0,1.0,1,875,1.0,1.0,1.0 +embed_bge_small,q015,Make a task in my project tracker Linear,3,1,284,1.0,1.0,1.0,1,483,1.0,1.0,1.0,1,870,1.0,1.0,1.0 +embed_bge_small,q016,Post a comment on a GitHub issue,1,1,388,1.0,1.0,1.0,1,617,1.0,1.0,1.0,1,1118,1.0,1.0,1.0 +embed_bge_small,q017,Merge the pull request,1,1,283,1.0,1.0,1.0,1,528,1.0,1.0,1.0,1,1003,1.0,1.0,1.0 +embed_bge_small,q018,Trigger a GitHub Actions workflow,1,1,294,1.0,1.0,1.0,1,457,1.0,1.0,1.0,1,780,1.0,1.0,1.0 +embed_bge_small,q019,Get Dependabot alerts,1,1,286,1.0,1.0,1.0,1,481,1.0,1.0,1.0,1,819,1.0,1.0,1.0 +embed_bge_small,q020,List Sentry issues,1,1,238,1.0,1.0,1.0,1,396,1.0,1.0,1.0,1,766,1.0,1.0,1.0 +embed_bge_small,q021,Create a Stripe customer,2,1,223,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +embed_bge_small,q022,Refund a Stripe payment,1,1,197,1.0,1.0,1.0,1,295,1.0,1.0,1.0,1,647,1.0,1.0,1.0 +embed_bge_small,q023,Send an email with Gmail,1,1,224,1.0,1.0,1.0,1,399,1.0,1.0,1.0,1,743,1.0,1.0,1.0 +embed_bge_small,q024,Search my Gmail inbox,1,1,178,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,678,1.0,1.0,1.0 +embed_bge_small,q025,Upload a file to Google Drive,2,1,307,1.0,1.0,1.0,1,418,1.0,1.0,1.0,1,806,1.0,1.0,1.0 +embed_bge_small,q026,Create a calendar event,1,1,317,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,858,1.0,1.0,1.0 +embed_bge_small,q027,List Kubernetes pods,1,1,184,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,661,1.0,1.0,1.0 +embed_bge_small,q028,Get logs from a k8s pod,1,1,252,1.0,1.0,1.0,1,371,1.0,1.0,1.0,1,738,1.0,1.0,1.0 +embed_bge_small,q029,Run a docker container,1,1,223,1.0,1.0,1.0,1,331,1.0,1.0,1.0,1,594,1.0,1.0,1.0 +embed_bge_small,q030,Show docker container logs,1,1,213,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,678,1.0,1.0,1.0 +embed_bge_small,q031,Navigate the browser to a URL,1,1,180,1.0,1.0,1.0,1,294,1.0,1.0,1.0,1,613,1.0,1.0,1.0 +embed_bge_small,q032,Take a screenshot of the page,1,1,212,1.0,1.0,1.0,1,346,1.0,1.0,1.0,1,633,1.0,1.0,1.0 +embed_bge_small,q033,Create a GitLab merge request,1,1,297,1.0,1.0,1.0,1,475,1.0,1.0,1.0,1,943,1.0,1.0,1.0 +embed_bge_small,q034,Search Notion pages,1,1,229,1.0,1.0,1.0,1,383,1.0,1.0,1.0,1,672,1.0,1.0,1.0 +embed_bge_small,q035,Create a Notion page,1,1,247,1.0,1.0,1.0,1,385,1.0,1.0,1.0,1,781,1.0,1.0,1.0 +embed_bge_small,q036,Query a Notion database,1,1,261,1.0,1.0,1.0,1,370,1.0,1.0,1.0,1,675,1.0,1.0,1.0 +embed_bge_small,q037,Create a Zendesk ticket,2,1,221,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,855,1.0,1.0,1.0 +embed_bge_small,q038,Create a HubSpot contact,1,1,221,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,786,1.0,1.0,1.0 +embed_bge_small,q039,Create a Salesforce lead,1,1,252,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,915,1.0,1.0,1.0 +embed_bge_small,q040,Run a SOQL query,1,1,187,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,632,1.0,1.0,1.0 +embed_bge_small,q041,List Shopify orders,1,1,210,1.0,1.0,1.0,1,333,1.0,1.0,1.0,1,667,1.0,1.0,1.0 +embed_bge_small,q042,Create a PagerDuty incident,2,1,259,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,823,1.0,1.0,1.0 +embed_bge_small,q043,Who is on call in PagerDuty,1,1,253,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,719,1.0,1.0,1.0 +embed_bge_small,q044,Query Datadog metrics,1,1,284,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,756,1.0,1.0,1.0 +embed_bge_small,q045,Search Datadog logs,1,1,298,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,773,1.0,1.0,1.0 +embed_bge_small,q046,List S3 buckets,1,1,204,1.0,1.0,1.0,1,327,1.0,1.0,1.0,1,624,1.0,1.0,1.0 +embed_bge_small,q047,Upload an object to S3,1,1,252,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,734,1.0,1.0,1.0 +embed_bge_small,q048,Purge Cloudflare cache,1,1,183,1.0,1.0,1.0,1,296,1.0,1.0,1.0,1,618,1.0,1.0,1.0 +embed_bge_small,q049,Create a Cloudflare DNS record,1,1,250,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,808,1.0,1.0,1.0 +embed_bge_small,q050,List Vercel deployments,1,1,177,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,601,1.0,1.0,1.0 +embed_bge_small,q051,Insert a row into Supabase,1,1,238,1.0,1.0,1.0,1,355,1.0,1.0,1.0,1,691,1.0,1.0,1.0 +embed_bge_small,q052,Export images from Figma,1,1,239,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,670,1.0,1.0,1.0 +embed_bge_small,q053,Send an SMS with Twilio,1,1,249,1.0,1.0,1.0,1,428,1.0,1.0,1.0,1,820,1.0,1.0,1.0 +embed_bge_small,q054,Create an Okta user,1,1,178,1.0,1.0,1.0,1,337,1.0,1.0,1.0,1,806,1.0,1.0,1.0 +embed_bge_small,q055,Instant query Prometheus,1,1,221,1.0,1.0,1.0,1,379,1.0,1.0,1.0,1,687,1.0,1.0,1.0 +embed_bge_small,q056,Search Grafana dashboards,1,1,185,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,663,1.0,1.0,1.0 +embed_bge_small,q057,Trigger a CircleCI pipeline,1,1,200,1.0,1.0,1.0,1,371,1.0,1.0,1.0,1,773,1.0,1.0,1.0 +embed_bge_small,q058,Run terraform plan,1,1,172,1.0,1.0,1.0,1,332,1.0,1.0,1.0,1,710,1.0,1.0,1.0 +embed_bge_small,q059,Create an Asana task,1,1,264,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,856,1.0,1.0,1.0 +embed_bge_small,q060,Create a Trello card,1,1,240,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,752,1.0,1.0,1.0 +embed_bge_small,q061,Create a ClickUp task,1,1,219,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,782,1.0,1.0,1.0 +embed_bge_small,q062,Create a Confluence page,1,1,260,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,866,1.0,1.0,1.0 +embed_bge_small,q063,Send a Discord message,1,1,245,1.0,1.0,1.0,1,384,1.0,1.0,1.0,1,742,1.0,1.0,1.0 +embed_bge_small,q064,Search Elasticsearch,1,1,195,1.0,1.0,1.0,1,300,1.0,1.0,1.0,1,635,1.0,1.0,1.0 +embed_bge_small,q065,Get a Redis key,1,1,182,1.0,1.0,1.0,1,280,1.0,1.0,1.0,1,550,1.0,1.0,1.0 +embed_bge_small,q066,Find MongoDB documents,1,1,219,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,729,1.0,1.0,1.0 +embed_bge_small,q067,List SQLite tables,1,1,142,1.0,1.0,1.0,1,240,1.0,1.0,1.0,1,526,1.0,1.0,1.0 +embed_bge_small,q068,Describe a postgres table,1,1,188,1.0,1.0,1.0,1,274,1.0,1.0,1.0,1,593,1.0,1.0,1.0 +embed_bge_small,q069,github_create_issue,2,1,367,1.0,1.0,1.0,1,536,1.0,1.0,1.0,1,1012,1.0,1.0,1.0 +embed_bge_small,q070,createIssue,39,13,3554,0.0,0.0,0.0,8,3600,0.0,0.0,0.0,4,3528,0.0,0.0,0.0 +embed_bge_small,q071,list-directory,1,1,182,1.0,1.0,1.0,1,271,1.0,1.0,1.0,1,489,1.0,1.0,1.0 +embed_bge_small,q072,sequential thinking next thought,1,1,239,1.0,1.0,1.0,1,344,1.0,1.0,1.0,1,650,1.0,1.0,1.0 +embed_bge_small,q073,What time is it in Tokyo,1,1,215,1.0,1.0,1.0,1,378,1.0,1.0,1.0,1,785,1.0,1.0,1.0 +embed_bge_small,q074,Convert time between timezones,1,1,234,1.0,1.0,1.0,1,402,1.0,1.0,1.0,1,699,1.0,1.0,1.0 +embed_bge_small,q075,Star a GitHub repo,1,1,214,1.0,1.0,1.0,1,437,1.0,1.0,1.0,1,950,1.0,1.0,1.0 +embed_bge_small,q076,Fork a repository,1,1,277,1.0,1.0,1.0,1,468,1.0,1.0,1.0,1,950,1.0,1.0,1.0 +embed_bge_small,q077,Get the latest GitHub release,1,1,239,1.0,1.0,1.0,1,366,1.0,1.0,1.0,1,673,1.0,1.0,1.0 +embed_bge_small,q078,Create a gist,1,1,229,1.0,1.0,1.0,1,383,1.0,1.0,1.0,1,887,1.0,1.0,1.0 +embed_bge_small,q079,Search GitHub code,1,1,190,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,722,1.0,1.0,1.0 +embed_bge_small,q080,List GitHub notifications,1,1,176,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,713,1.0,1.0,1.0 +embed_bge_small,q081,Assign Copilot to an issue,1,1,301,1.0,1.0,1.0,1,434,1.0,1.0,1.0,1,826,1.0,1.0,1.0 +embed_bge_small,q082,Get secret scanning alerts,1,1,294,1.0,1.0,1.0,1,483,1.0,1.0,1.0,1,781,1.0,1.0,1.0 +embed_bge_small,q083,Create or update a file on GitHub,1,1,327,1.0,1.0,1.0,1,476,1.0,1.0,1.0,1,878,1.0,1.0,1.0 +embed_bge_small,q084,List collaborators on a repo,1,1,223,1.0,1.0,1.0,1,365,1.0,1.0,1.0,1,696,1.0,1.0,1.0 +embed_bge_small,q085,Reply in a Slack thread,1,1,277,1.0,1.0,1.0,1,402,1.0,1.0,1.0,1,777,1.0,1.0,1.0 +embed_bge_small,q086,Add an emoji reaction on Slack,1,1,289,1.0,1.0,1.0,1,440,1.0,1.0,1.0,1,795,1.0,1.0,1.0 +embed_bge_small,q087,Search Slack messages,1,1,251,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,720,1.0,1.0,1.0 +embed_bge_small,q088,Create a Slack channel,1,1,196,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,781,1.0,1.0,1.0 +embed_bge_small,q089,Transition a Jira issue,1,1,223,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,745,1.0,1.0,1.0 +embed_bge_small,q090,Search Jira with JQL,1,1,239,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,618,1.0,1.0,1.0 +embed_bge_small,q091,Add a worklog to Jira,1,1,261,1.0,1.0,1.0,1,429,1.0,1.0,1.0,1,909,1.0,1.0,1.0 +embed_bge_small,q092,List Linear teams,1,1,184,1.0,1.0,1.0,1,398,1.0,1.0,1.0,1,718,1.0,1.0,1.0 +embed_bge_small,q093,Comment on a Linear issue,1,1,218,1.0,1.0,1.0,1,324,1.0,1.0,1.0,1,771,1.0,1.0,1.0 +embed_bge_small,q094,Move a local file,1,1,225,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +embed_bge_small,q095,Recursively search files by pattern,1,1,202,1.0,1.0,1.0,1,304,1.0,1.0,1.0,1,609,1.0,1.0,1.0 +embed_bge_small,q096,Edit a file in place with old and new string,1,1,249,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,785,1.0,1.0,1.0 +embed_bge_small,q097,Read the whole knowledge graph,1,1,162,1.0,1.0,1.0,1,270,1.0,1.0,1.0,1,582,1.0,1.0,1.0 +embed_bge_small,q098,Search memory nodes,1,1,194,1.0,1.0,1.0,1,285,1.0,1.0,1.0,1,632,1.0,1.0,1.0 +embed_bge_small,q099,Show unstaged git diff,1,1,190,1.0,1.0,1.0,1,299,1.0,1.0,1.0,1,620,1.0,1.0,1.0 +embed_bge_small,q100,Stage files with git add,1,1,201,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,723,1.0,1.0,1.0 +embed_bge_small,q101,Create a Shopify product,1,1,235,1.0,1.0,1.0,1,365,1.0,1.0,1.0,1,670,1.0,1.0,1.0 +embed_bge_small,q102,Cancel a Stripe subscription,1,1,204,1.0,1.0,1.0,1,301,1.0,1.0,1.0,1,629,1.0,1.0,1.0 +embed_bge_small,q103,Create a payment intent,1,1,234,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,740,1.0,1.0,1.0 +embed_bge_small,q104,Acknowledge a PagerDuty incident,1,1,223,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +embed_bge_small,q105,Create a Datadog monitor,1,1,259,1.0,1.0,1.0,1,459,1.0,1.0,1.0,1,799,1.0,1.0,1.0 +embed_bge_small,q106,Resolve a Sentry issue,3,1,274,1.0,1.0,1.0,1,397,1.0,1.0,1.0,1,800,1.0,1.0,1.0 +embed_bge_small,q107,Scale a Kubernetes deployment,1,1,241,1.0,1.0,1.0,1,391,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +embed_bge_small,q108,Apply a Kubernetes manifest,1,1,195,1.0,1.0,1.0,1,330,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +embed_bge_small,q109,Stop a docker container,1,1,219,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,636,1.0,1.0,1.0 +embed_bge_small,q110,Build a docker image,1,1,211,1.0,1.0,1.0,1,319,1.0,1.0,1.0,1,757,1.0,1.0,1.0 +embed_bge_small,q111,Click an element in Playwright,1,1,213,1.0,1.0,1.0,1,318,1.0,1.0,1.0,1,594,1.0,1.0,1.0 +embed_bge_small,q112,Type text into a browser input,1,1,231,1.0,1.0,1.0,1,331,1.0,1.0,1.0,1,624,1.0,1.0,1.0 +embed_bge_small,q113,List GitLab projects,1,1,216,1.0,1.0,1.0,1,372,1.0,1.0,1.0,1,735,1.0,1.0,1.0 +embed_bge_small,q114,Create a Bitbucket pull request,1,1,372,1.0,1.0,1.0,1,606,1.0,1.0,1.0,1,1091,1.0,1.0,1.0 +embed_bge_small,q115,Create a Gmail draft,1,1,249,1.0,1.0,1.0,1,375,1.0,1.0,1.0,1,739,1.0,1.0,1.0 +embed_bge_small,q116,Share a Google Drive file,1,1,271,1.0,1.0,1.0,1,417,1.0,1.0,1.0,1,717,1.0,1.0,1.0 +embed_bge_small,q117,List Google Calendar events,2,1,267,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,724,1.0,1.0,1.0 +embed_bge_small,q118,Delete a calendar event,1,1,205,1.0,1.0,1.0,1,318,1.0,1.0,1.0,1,651,1.0,1.0,1.0 +embed_bge_small,q119,List Vercel projects,1,1,175,1.0,1.0,1.0,1,260,1.0,1.0,1.0,1,524,1.0,1.0,1.0 +embed_bge_small,q120,Select rows from Supabase,1,1,237,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,623,1.0,1.0,1.0 +embed_bge_small,q121,Post a Figma comment,1,1,204,1.0,1.0,1.0,1,350,1.0,1.0,1.0,1,699,1.0,1.0,1.0 +embed_bge_small,q122,List Okta users,1,1,177,1.0,1.0,1.0,1,271,1.0,1.0,1.0,1,613,1.0,1.0,1.0 +embed_bge_small,q123,Range query Prometheus,1,1,221,1.0,1.0,1.0,1,385,1.0,1.0,1.0,1,736,1.0,1.0,1.0 +embed_bge_small,q124,Create a Grafana annotation,1,1,218,1.0,1.0,1.0,1,429,1.0,1.0,1.0,1,911,1.0,1.0,1.0 +embed_bge_small,q125,Apply terraform,1,1,171,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,653,1.0,1.0,1.0 +embed_bge_small,q126,List Asana tasks,1,1,244,1.0,1.0,1.0,1,404,1.0,1.0,1.0,1,705,1.0,1.0,1.0 +embed_bge_small,q127,Comment on a Trello card,1,1,260,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +embed_bge_small,q128,Update a ClickUp task,1,1,219,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,731,1.0,1.0,1.0 +embed_bge_small,q129,Search Confluence,1,1,221,1.0,1.0,1.0,1,371,1.0,1.0,1.0,1,652,1.0,1.0,1.0 +embed_bge_small,q130,Create a HubSpot deal,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,715,1.0,1.0,1.0 +embed_bge_small,q131,List Zendesk tickets,1,1,189,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,674,1.0,1.0,1.0 +embed_bge_small,q132,List Discord channels,1,1,184,1.0,1.0,1.0,1,300,1.0,1.0,1.0,1,642,1.0,1.0,1.0 +embed_bge_small,q133,Set a Redis key,1,1,182,1.0,1.0,1.0,1,280,1.0,1.0,1.0,1,641,1.0,1.0,1.0 +embed_bge_small,q134,Insert MongoDB documents,1,1,249,1.0,1.0,1.0,1,374,1.0,1.0,1.0,1,759,1.0,1.0,1.0 +embed_bge_small,q135,Run a MySQL query,1,1,179,1.0,1.0,1.0,1,294,1.0,1.0,1.0,1,595,1.0,1.0,1.0 +embed_bge_small,q136,Index a document in Elasticsearch,1,1,242,1.0,1.0,1.0,1,372,1.0,1.0,1.0,1,681,1.0,1.0,1.0 +embed_bge_small,q137,Get an S3 object,1,1,249,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,641,1.0,1.0,1.0 +embed_bge_small,q138,List Cloudflare zones,1,1,215,1.0,1.0,1.0,1,326,1.0,1.0,1.0,1,570,1.0,1.0,1.0 +embed_bge_small,q139,Create a Stripe invoice,1,1,204,1.0,1.0,1.0,1,341,1.0,1.0,1.0,1,662,1.0,1.0,1.0 +embed_bge_small,q140,List GitHub Actions workflows,1,1,293,1.0,1.0,1.0,1,452,1.0,1.0,1.0,1,859,1.0,1.0,1.0 +embed_bge_small,q141,Get GitHub Actions job logs,1,1,269,1.0,1.0,1.0,1,442,1.0,1.0,1.0,1,805,1.0,1.0,1.0 +embed_bge_small,q142,Create a GitHub repository,1,1,274,1.0,1.0,1.0,1,519,1.0,1.0,1.0,1,1091,1.0,1.0,1.0 +embed_bge_small,q143,Checkout a git branch,1,1,182,1.0,1.0,1.0,1,295,1.0,1.0,1.0,1,693,1.0,1.0,1.0 +embed_bge_small,q144,Show git log,1,1,194,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,668,1.0,1.0,1.0 +embed_bge_small,q145,List postgres schemas,1,1,152,1.0,1.0,1.0,1,254,1.0,1.0,1.0,1,493,1.0,1.0,1.0 +embed_bge_small,q146,Open nodes in the knowledge graph,1,1,163,1.0,1.0,1.0,1,262,1.0,1.0,1.0,1,599,1.0,1.0,1.0 +embed_bge_small,q147,Get file metadata from the filesystem,1,1,190,1.0,1.0,1.0,1,302,1.0,1.0,1.0,1,685,1.0,1.0,1.0 +embed_bge_small,q148,Write a new local file,1,1,280,1.0,1.0,1.0,1,421,1.0,1.0,1.0,1,850,1.0,1.0,1.0 +embed_bge_small,q149,Search GitHub issues,1,1,208,1.0,1.0,1.0,1,314,1.0,1.0,1.0,1,724,1.0,1.0,1.0 +embed_bge_small,q150,Get the authenticated GitHub user,1,1,179,1.0,1.0,1.0,1,343,1.0,1.0,1.0,1,670,1.0,1.0,1.0 +hashed_ngram_embed,q001,Create a GitHub issue,1,1,334,1.0,1.0,1.0,1,509,1.0,1.0,1.0,1,1047,1.0,1.0,1.0 +hashed_ngram_embed,q002,Create a Jira ticket,1,1,345,1.0,1.0,1.0,1,535,1.0,1.0,1.0,1,940,1.0,1.0,1.0 +hashed_ngram_embed,q003,Send a Slack message,1,1,249,1.0,1.0,1.0,1,394,1.0,1.0,1.0,1,709,1.0,1.0,1.0 +hashed_ngram_embed,q004,List open pull requests,2,1,363,1.0,1.0,1.0,1,517,1.0,1.0,1.0,1,1044,1.0,1.0,1.0 +hashed_ngram_embed,q005,Open a pull request on GitHub,1,1,327,1.0,1.0,1.0,1,520,1.0,1.0,1.0,1,1144,1.0,1.0,1.0 +hashed_ngram_embed,q006,Run a SQL query against postgres,1,1,183,1.0,1.0,1.0,1,295,1.0,1.0,1.0,1,575,1.0,1.0,1.0 +hashed_ngram_embed,q007,Read a local file from disk,2,1,223,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,766,1.0,1.0,1.0 +hashed_ngram_embed,q008,What files are in this folder,5,2,464,0.0,1.0,1.0,1,388,1.0,1.0,1.0,1,715,1.0,1.0,1.0 +hashed_ngram_embed,q009,Remember this fact in the knowledge graph,4,2,343,0.0,1.0,1.0,1,272,1.0,1.0,1.0,1,554,1.0,1.0,1.0 +hashed_ngram_embed,q010,Search the web with Brave,1,1,236,1.0,1.0,1.0,1,359,1.0,1.0,1.0,1,684,1.0,1.0,1.0 +hashed_ngram_embed,q011,Fetch the contents of a URL,1,1,221,1.0,1.0,1.0,1,336,1.0,1.0,1.0,1,705,1.0,1.0,1.0 +hashed_ngram_embed,q012,Show git status,2,1,196,1.0,1.0,1.0,1,324,1.0,1.0,1.0,1,637,1.0,1.0,1.0 +hashed_ngram_embed,q013,Commit staged changes,2,1,182,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,754,1.0,1.0,1.0 +hashed_ngram_embed,q014,Create a Linear issue,2,1,333,1.0,1.0,1.0,1,471,1.0,1.0,1.0,1,895,1.0,1.0,1.0 +hashed_ngram_embed,q015,Make a task in my project tracker Linear,6,2,580,0.0,1.0,1.0,2,849,0.0,1.0,1.0,1,825,1.0,1.0,1.0 +hashed_ngram_embed,q016,Post a comment on a GitHub issue,2,1,337,1.0,1.0,1.0,1,509,1.0,1.0,1.0,1,1018,1.0,1.0,1.0 +hashed_ngram_embed,q017,Merge the pull request,1,1,283,1.0,1.0,1.0,1,474,1.0,1.0,1.0,1,933,1.0,1.0,1.0 +hashed_ngram_embed,q018,Trigger a GitHub Actions workflow,1,1,294,1.0,1.0,1.0,1,491,1.0,1.0,1.0,1,862,1.0,1.0,1.0 +hashed_ngram_embed,q019,Get Dependabot alerts,1,1,286,1.0,1.0,1.0,1,455,1.0,1.0,1.0,1,773,1.0,1.0,1.0 +hashed_ngram_embed,q020,List Sentry issues,1,1,257,1.0,1.0,1.0,1,396,1.0,1.0,1.0,1,845,1.0,1.0,1.0 +hashed_ngram_embed,q021,Create a Stripe customer,1,1,186,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +hashed_ngram_embed,q022,Refund a Stripe payment,1,1,234,1.0,1.0,1.0,1,350,1.0,1.0,1.0,1,662,1.0,1.0,1.0 +hashed_ngram_embed,q023,Send an email with Gmail,1,1,218,1.0,1.0,1.0,1,328,1.0,1.0,1.0,1,733,1.0,1.0,1.0 +hashed_ngram_embed,q024,Search my Gmail inbox,1,1,250,1.0,1.0,1.0,1,373,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +hashed_ngram_embed,q025,Upload a file to Google Drive,1,1,272,1.0,1.0,1.0,1,418,1.0,1.0,1.0,1,826,1.0,1.0,1.0 +hashed_ngram_embed,q026,Create a calendar event,1,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,837,1.0,1.0,1.0 +hashed_ngram_embed,q027,List Kubernetes pods,1,1,206,1.0,1.0,1.0,1,314,1.0,1.0,1.0,1,749,1.0,1.0,1.0 +hashed_ngram_embed,q028,Get logs from a k8s pod,1,1,244,1.0,1.0,1.0,1,397,1.0,1.0,1.0,1,783,1.0,1.0,1.0 +hashed_ngram_embed,q029,Run a docker container,4,2,411,0.0,1.0,1.0,1,345,1.0,1.0,1.0,1,600,1.0,1.0,1.0 +hashed_ngram_embed,q030,Show docker container logs,1,1,188,1.0,1.0,1.0,1,298,1.0,1.0,1.0,1,609,1.0,1.0,1.0 +hashed_ngram_embed,q031,Navigate the browser to a URL,1,1,180,1.0,1.0,1.0,1,265,1.0,1.0,1.0,1,658,1.0,1.0,1.0 +hashed_ngram_embed,q032,Take a screenshot of the page,1,1,242,1.0,1.0,1.0,1,408,1.0,1.0,1.0,1,707,1.0,1.0,1.0 +hashed_ngram_embed,q033,Create a GitLab merge request,1,1,297,1.0,1.0,1.0,1,475,1.0,1.0,1.0,1,883,1.0,1.0,1.0 +hashed_ngram_embed,q034,Search Notion pages,1,1,229,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,708,1.0,1.0,1.0 +hashed_ngram_embed,q035,Create a Notion page,1,1,230,1.0,1.0,1.0,1,419,1.0,1.0,1.0,1,815,1.0,1.0,1.0 +hashed_ngram_embed,q036,Query a Notion database,1,1,261,1.0,1.0,1.0,1,373,1.0,1.0,1.0,1,691,1.0,1.0,1.0 +hashed_ngram_embed,q037,Create a Zendesk ticket,2,1,257,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,881,1.0,1.0,1.0 +hashed_ngram_embed,q038,Create a HubSpot contact,1,1,221,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,805,1.0,1.0,1.0 +hashed_ngram_embed,q039,Create a Salesforce lead,1,1,252,1.0,1.0,1.0,1,416,1.0,1.0,1.0,1,766,1.0,1.0,1.0 +hashed_ngram_embed,q040,Run a SOQL query,1,1,187,1.0,1.0,1.0,1,296,1.0,1.0,1.0,1,600,1.0,1.0,1.0 +hashed_ngram_embed,q041,List Shopify orders,1,1,212,1.0,1.0,1.0,1,333,1.0,1.0,1.0,1,691,1.0,1.0,1.0 +hashed_ngram_embed,q042,Create a PagerDuty incident,1,1,255,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,818,1.0,1.0,1.0 +hashed_ngram_embed,q043,Who is on call in PagerDuty,3,1,219,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,749,1.0,1.0,1.0 +hashed_ngram_embed,q044,Query Datadog metrics,1,1,300,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,753,1.0,1.0,1.0 +hashed_ngram_embed,q045,Search Datadog logs,1,1,255,1.0,1.0,1.0,1,422,1.0,1.0,1.0,1,736,1.0,1.0,1.0 +hashed_ngram_embed,q046,List S3 buckets,1,1,186,1.0,1.0,1.0,1,316,1.0,1.0,1.0,1,613,1.0,1.0,1.0 +hashed_ngram_embed,q047,Upload an object to S3,1,1,250,1.0,1.0,1.0,1,380,1.0,1.0,1.0,1,733,1.0,1.0,1.0 +hashed_ngram_embed,q048,Purge Cloudflare cache,1,1,231,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,732,1.0,1.0,1.0 +hashed_ngram_embed,q049,Create a Cloudflare DNS record,1,1,217,1.0,1.0,1.0,1,378,1.0,1.0,1.0,1,782,1.0,1.0,1.0 +hashed_ngram_embed,q050,List Vercel deployments,1,1,195,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,637,1.0,1.0,1.0 +hashed_ngram_embed,q051,Insert a row into Supabase,1,1,238,1.0,1.0,1.0,1,355,1.0,1.0,1.0,1,741,1.0,1.0,1.0 +hashed_ngram_embed,q052,Export images from Figma,1,1,218,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,667,1.0,1.0,1.0 +hashed_ngram_embed,q053,Send an SMS with Twilio,1,1,249,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,848,1.0,1.0,1.0 +hashed_ngram_embed,q054,Create an Okta user,1,1,183,1.0,1.0,1.0,1,344,1.0,1.0,1.0,1,744,1.0,1.0,1.0 +hashed_ngram_embed,q055,Instant query Prometheus,1,1,221,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,708,1.0,1.0,1.0 +hashed_ngram_embed,q056,Search Grafana dashboards,1,1,198,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,677,1.0,1.0,1.0 +hashed_ngram_embed,q057,Trigger a CircleCI pipeline,1,1,200,1.0,1.0,1.0,1,325,1.0,1.0,1.0,1,676,1.0,1.0,1.0 +hashed_ngram_embed,q058,Run terraform plan,1,1,172,1.0,1.0,1.0,1,293,1.0,1.0,1.0,1,582,1.0,1.0,1.0 +hashed_ngram_embed,q059,Create an Asana task,1,1,267,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,872,1.0,1.0,1.0 +hashed_ngram_embed,q060,Create a Trello card,1,1,240,1.0,1.0,1.0,1,411,1.0,1.0,1.0,1,895,1.0,1.0,1.0 +hashed_ngram_embed,q061,Create a ClickUp task,1,1,257,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,858,1.0,1.0,1.0 +hashed_ngram_embed,q062,Create a Confluence page,1,1,260,1.0,1.0,1.0,1,409,1.0,1.0,1.0,1,943,1.0,1.0,1.0 +hashed_ngram_embed,q063,Send a Discord message,1,1,224,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,692,1.0,1.0,1.0 +hashed_ngram_embed,q064,Search Elasticsearch,1,1,239,1.0,1.0,1.0,1,404,1.0,1.0,1.0,1,696,1.0,1.0,1.0 +hashed_ngram_embed,q065,Get a Redis key,1,1,182,1.0,1.0,1.0,1,312,1.0,1.0,1.0,1,622,1.0,1.0,1.0 +hashed_ngram_embed,q066,Find MongoDB documents,2,1,244,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,788,1.0,1.0,1.0 +hashed_ngram_embed,q067,List SQLite tables,1,1,156,1.0,1.0,1.0,1,243,1.0,1.0,1.0,1,591,1.0,1.0,1.0 +hashed_ngram_embed,q068,Describe a postgres table,1,1,186,1.0,1.0,1.0,1,290,1.0,1.0,1.0,1,705,1.0,1.0,1.0 +hashed_ngram_embed,q069,github_create_issue,2,1,347,1.0,1.0,1.0,1,567,1.0,1.0,1.0,1,1062,1.0,1.0,1.0 +hashed_ngram_embed,q070,createIssue,2,1,319,1.0,1.0,1.0,1,501,1.0,1.0,1.0,1,917,1.0,1.0,1.0 +hashed_ngram_embed,q071,list-directory,2,1,186,1.0,1.0,1.0,1,281,1.0,1.0,1.0,1,621,1.0,1.0,1.0 +hashed_ngram_embed,q072,sequential thinking next thought,1,1,302,1.0,1.0,1.0,1,430,1.0,1.0,1.0,1,746,1.0,1.0,1.0 +hashed_ngram_embed,q073,What time is it in Tokyo,1,1,217,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,725,1.0,1.0,1.0 +hashed_ngram_embed,q074,Convert time between timezones,1,1,213,1.0,1.0,1.0,1,378,1.0,1.0,1.0,1,779,1.0,1.0,1.0 +hashed_ngram_embed,q075,Star a GitHub repo,1,1,244,1.0,1.0,1.0,1,414,1.0,1.0,1.0,1,861,1.0,1.0,1.0 +hashed_ngram_embed,q076,Fork a repository,1,1,235,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,824,1.0,1.0,1.0 +hashed_ngram_embed,q077,Get the latest GitHub release,1,1,257,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,707,1.0,1.0,1.0 +hashed_ngram_embed,q078,Create a gist,1,1,236,1.0,1.0,1.0,1,387,1.0,1.0,1.0,1,824,1.0,1.0,1.0 +hashed_ngram_embed,q079,Search GitHub code,3,1,190,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,757,1.0,1.0,1.0 +hashed_ngram_embed,q080,List GitHub notifications,1,1,175,1.0,1.0,1.0,1,280,1.0,1.0,1.0,1,601,1.0,1.0,1.0 +hashed_ngram_embed,q081,Assign Copilot to an issue,1,1,259,1.0,1.0,1.0,1,468,1.0,1.0,1.0,1,852,1.0,1.0,1.0 +hashed_ngram_embed,q082,Get secret scanning alerts,1,1,294,1.0,1.0,1.0,1,440,1.0,1.0,1.0,1,756,1.0,1.0,1.0 +hashed_ngram_embed,q083,Create or update a file on GitHub,1,1,314,1.0,1.0,1.0,1,491,1.0,1.0,1.0,1,909,1.0,1.0,1.0 +hashed_ngram_embed,q084,List collaborators on a repo,1,1,272,1.0,1.0,1.0,1,413,1.0,1.0,1.0,1,789,1.0,1.0,1.0 +hashed_ngram_embed,q085,Reply in a Slack thread,1,1,242,1.0,1.0,1.0,1,385,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +hashed_ngram_embed,q086,Add an emoji reaction on Slack,1,1,289,1.0,1.0,1.0,1,500,1.0,1.0,1.0,1,945,1.0,1.0,1.0 +hashed_ngram_embed,q087,Search Slack messages,1,1,248,1.0,1.0,1.0,1,379,1.0,1.0,1.0,1,757,1.0,1.0,1.0 +hashed_ngram_embed,q088,Create a Slack channel,1,1,196,1.0,1.0,1.0,1,339,1.0,1.0,1.0,1,791,1.0,1.0,1.0 +hashed_ngram_embed,q089,Transition a Jira issue,1,1,271,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,772,1.0,1.0,1.0 +hashed_ngram_embed,q090,Search Jira with JQL,1,1,193,1.0,1.0,1.0,1,305,1.0,1.0,1.0,1,749,1.0,1.0,1.0 +hashed_ngram_embed,q091,Add a worklog to Jira,1,1,233,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,862,1.0,1.0,1.0 +hashed_ngram_embed,q092,List Linear teams,1,1,184,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,713,1.0,1.0,1.0 +hashed_ngram_embed,q093,Comment on a Linear issue,1,1,195,1.0,1.0,1.0,1,355,1.0,1.0,1.0,1,824,1.0,1.0,1.0 +hashed_ngram_embed,q094,Move a local file,2,1,209,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,778,1.0,1.0,1.0 +hashed_ngram_embed,q095,Recursively search files by pattern,1,1,225,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,649,1.0,1.0,1.0 +hashed_ngram_embed,q096,Edit a file in place with old and new string,1,1,282,1.0,1.0,1.0,1,524,1.0,1.0,1.0,1,864,1.0,1.0,1.0 +hashed_ngram_embed,q097,Read the whole knowledge graph,1,1,162,1.0,1.0,1.0,1,262,1.0,1.0,1.0,1,631,1.0,1.0,1.0 +hashed_ngram_embed,q098,Search memory nodes,1,1,192,1.0,1.0,1.0,1,350,1.0,1.0,1.0,1,662,1.0,1.0,1.0 +hashed_ngram_embed,q099,Show unstaged git diff,1,1,174,1.0,1.0,1.0,1,299,1.0,1.0,1.0,1,629,1.0,1.0,1.0 +hashed_ngram_embed,q100,Stage files with git add,2,1,260,1.0,1.0,1.0,1,400,1.0,1.0,1.0,1,774,1.0,1.0,1.0 +hashed_ngram_embed,q101,Create a Shopify product,1,1,240,1.0,1.0,1.0,1,380,1.0,1.0,1.0,1,719,1.0,1.0,1.0 +hashed_ngram_embed,q102,Cancel a Stripe subscription,1,1,203,1.0,1.0,1.0,1,300,1.0,1.0,1.0,1,640,1.0,1.0,1.0 +hashed_ngram_embed,q103,Create a payment intent,1,1,234,1.0,1.0,1.0,1,466,1.0,1.0,1.0,1,917,1.0,1.0,1.0 +hashed_ngram_embed,q104,Acknowledge a PagerDuty incident,1,1,223,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,765,1.0,1.0,1.0 +hashed_ngram_embed,q105,Create a Datadog monitor,1,1,259,1.0,1.0,1.0,1,461,1.0,1.0,1.0,1,905,1.0,1.0,1.0 +hashed_ngram_embed,q106,Resolve a Sentry issue,2,1,251,1.0,1.0,1.0,1,397,1.0,1.0,1.0,1,892,1.0,1.0,1.0 +hashed_ngram_embed,q107,Scale a Kubernetes deployment,1,1,262,1.0,1.0,1.0,1,438,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +hashed_ngram_embed,q108,Apply a Kubernetes manifest,1,1,263,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,755,1.0,1.0,1.0 +hashed_ngram_embed,q109,Stop a docker container,1,1,188,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,604,1.0,1.0,1.0 +hashed_ngram_embed,q110,Build a docker image,1,1,211,1.0,1.0,1.0,1,317,1.0,1.0,1.0,1,590,1.0,1.0,1.0 +hashed_ngram_embed,q111,Click an element in Playwright,1,1,216,1.0,1.0,1.0,1,343,1.0,1.0,1.0,1,646,1.0,1.0,1.0 +hashed_ngram_embed,q112,Type text into a browser input,1,1,217,1.0,1.0,1.0,1,316,1.0,1.0,1.0,1,693,1.0,1.0,1.0 +hashed_ngram_embed,q113,List GitLab projects,1,1,209,1.0,1.0,1.0,1,373,1.0,1.0,1.0,1,709,1.0,1.0,1.0 +hashed_ngram_embed,q114,Create a Bitbucket pull request,1,1,313,1.0,1.0,1.0,1,564,1.0,1.0,1.0,1,1080,1.0,1.0,1.0 +hashed_ngram_embed,q115,Create a Gmail draft,1,1,235,1.0,1.0,1.0,1,374,1.0,1.0,1.0,1,814,1.0,1.0,1.0 +hashed_ngram_embed,q116,Share a Google Drive file,1,1,254,1.0,1.0,1.0,1,418,1.0,1.0,1.0,1,833,1.0,1.0,1.0 +hashed_ngram_embed,q117,List Google Calendar events,1,1,219,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,825,1.0,1.0,1.0 +hashed_ngram_embed,q118,Delete a calendar event,1,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,687,1.0,1.0,1.0 +hashed_ngram_embed,q119,List Vercel projects,1,1,175,1.0,1.0,1.0,1,269,1.0,1.0,1.0,1,593,1.0,1.0,1.0 +hashed_ngram_embed,q120,Select rows from Supabase,1,1,237,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,659,1.0,1.0,1.0 +hashed_ngram_embed,q121,Post a Figma comment,1,1,204,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,715,1.0,1.0,1.0 +hashed_ngram_embed,q122,List Okta users,1,1,177,1.0,1.0,1.0,1,284,1.0,1.0,1.0,1,636,1.0,1.0,1.0 +hashed_ngram_embed,q123,Range query Prometheus,1,1,221,1.0,1.0,1.0,1,347,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +hashed_ngram_embed,q124,Create a Grafana annotation,1,1,240,1.0,1.0,1.0,1,434,1.0,1.0,1.0,1,813,1.0,1.0,1.0 +hashed_ngram_embed,q125,Apply terraform,1,1,171,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,673,1.0,1.0,1.0 +hashed_ngram_embed,q126,List Asana tasks,1,1,228,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,712,1.0,1.0,1.0 +hashed_ngram_embed,q127,Comment on a Trello card,1,1,260,1.0,1.0,1.0,1,412,1.0,1.0,1.0,1,787,1.0,1.0,1.0 +hashed_ngram_embed,q128,Update a ClickUp task,1,1,257,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,787,1.0,1.0,1.0 +hashed_ngram_embed,q129,Search Confluence,1,1,221,1.0,1.0,1.0,1,369,1.0,1.0,1.0,1,703,1.0,1.0,1.0 +hashed_ngram_embed,q130,Create a HubSpot deal,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,819,1.0,1.0,1.0 +hashed_ngram_embed,q131,List Zendesk tickets,1,1,222,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,739,1.0,1.0,1.0 +hashed_ngram_embed,q132,List Discord channels,1,1,184,1.0,1.0,1.0,1,321,1.0,1.0,1.0,1,635,1.0,1.0,1.0 +hashed_ngram_embed,q133,Set a Redis key,2,1,182,1.0,1.0,1.0,1,321,1.0,1.0,1.0,1,663,1.0,1.0,1.0 +hashed_ngram_embed,q134,Insert MongoDB documents,1,1,244,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,779,1.0,1.0,1.0 +hashed_ngram_embed,q135,Run a MySQL query,1,1,182,1.0,1.0,1.0,1,281,1.0,1.0,1.0,1,625,1.0,1.0,1.0 +hashed_ngram_embed,q136,Index a document in Elasticsearch,1,1,242,1.0,1.0,1.0,1,392,1.0,1.0,1.0,1,869,1.0,1.0,1.0 +hashed_ngram_embed,q137,Get an S3 object,2,1,234,1.0,1.0,1.0,1,392,1.0,1.0,1.0,1,714,1.0,1.0,1.0 +hashed_ngram_embed,q138,List Cloudflare zones,1,1,215,1.0,1.0,1.0,1,341,1.0,1.0,1.0,1,722,1.0,1.0,1.0 +hashed_ngram_embed,q139,Create a Stripe invoice,1,1,185,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,676,1.0,1.0,1.0 +hashed_ngram_embed,q140,List GitHub Actions workflows,1,1,293,1.0,1.0,1.0,1,435,1.0,1.0,1.0,1,824,1.0,1.0,1.0 +hashed_ngram_embed,q141,Get GitHub Actions job logs,1,1,269,1.0,1.0,1.0,1,460,1.0,1.0,1.0,1,815,1.0,1.0,1.0 +hashed_ngram_embed,q142,Create a GitHub repository,1,1,235,1.0,1.0,1.0,1,388,1.0,1.0,1.0,1,928,1.0,1.0,1.0 +hashed_ngram_embed,q143,Checkout a git branch,1,1,212,1.0,1.0,1.0,1,384,1.0,1.0,1.0,1,737,1.0,1.0,1.0 +hashed_ngram_embed,q144,Show git log,1,1,206,1.0,1.0,1.0,1,306,1.0,1.0,1.0,1,606,1.0,1.0,1.0 +hashed_ngram_embed,q145,List postgres schemas,1,1,158,1.0,1.0,1.0,1,277,1.0,1.0,1.0,1,606,1.0,1.0,1.0 +hashed_ngram_embed,q146,Open nodes in the knowledge graph,1,1,163,1.0,1.0,1.0,1,267,1.0,1.0,1.0,1,575,1.0,1.0,1.0 +hashed_ngram_embed,q147,Get file metadata from the filesystem,4,2,410,0.0,1.0,1.0,1,334,1.0,1.0,1.0,1,683,1.0,1.0,1.0 +hashed_ngram_embed,q148,Write a new local file,1,1,199,1.0,1.0,1.0,1,392,1.0,1.0,1.0,1,875,1.0,1.0,1.0 +hashed_ngram_embed,q149,Search GitHub issues,1,1,208,1.0,1.0,1.0,1,360,1.0,1.0,1.0,1,830,1.0,1.0,1.0 +hashed_ngram_embed,q150,Get the authenticated GitHub user,2,1,179,1.0,1.0,1.0,1,300,1.0,1.0,1.0,1,603,1.0,1.0,1.0 +hybrid_codedb_embed,q001,Create a GitHub issue,1,1,380,1.0,1.0,1.0,1,584,1.0,1.0,1.0,1,1088,1.0,1.0,1.0 +hybrid_codedb_embed,q002,Create a Jira ticket,1,1,345,1.0,1.0,1.0,1,540,1.0,1.0,1.0,1,924,1.0,1.0,1.0 +hybrid_codedb_embed,q003,Send a Slack message,1,1,252,1.0,1.0,1.0,1,408,1.0,1.0,1.0,1,778,1.0,1.0,1.0 +hybrid_codedb_embed,q004,List open pull requests,1,1,237,1.0,1.0,1.0,1,477,1.0,1.0,1.0,1,769,1.0,1.0,1.0 +hybrid_codedb_embed,q005,Open a pull request on GitHub,1,1,368,1.0,1.0,1.0,1,564,1.0,1.0,1.0,1,1144,1.0,1.0,1.0 +hybrid_codedb_embed,q006,Run a SQL query against postgres,1,1,180,1.0,1.0,1.0,1,295,1.0,1.0,1.0,1,551,1.0,1.0,1.0 +hybrid_codedb_embed,q007,Read a local file from disk,2,1,223,1.0,1.0,1.0,1,406,1.0,1.0,1.0,1,774,1.0,1.0,1.0 +hybrid_codedb_embed,q008,What files are in this folder,2,1,191,1.0,1.0,1.0,1,314,1.0,1.0,1.0,1,686,1.0,1.0,1.0 +hybrid_codedb_embed,q009,Remember this fact in the knowledge graph,3,1,164,1.0,1.0,1.0,1,262,1.0,1.0,1.0,1,568,1.0,1.0,1.0 +hybrid_codedb_embed,q010,Search the web with Brave,1,1,245,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,691,1.0,1.0,1.0 +hybrid_codedb_embed,q011,Fetch the contents of a URL,1,1,258,1.0,1.0,1.0,1,369,1.0,1.0,1.0,1,659,1.0,1.0,1.0 +hybrid_codedb_embed,q012,Show git status,1,1,186,1.0,1.0,1.0,1,312,1.0,1.0,1.0,1,675,1.0,1.0,1.0 +hybrid_codedb_embed,q013,Commit staged changes,1,1,182,1.0,1.0,1.0,1,327,1.0,1.0,1.0,1,706,1.0,1.0,1.0 +hybrid_codedb_embed,q014,Create a Linear issue,1,1,275,1.0,1.0,1.0,1,463,1.0,1.0,1.0,1,867,1.0,1.0,1.0 +hybrid_codedb_embed,q015,Make a task in my project tracker Linear,5,2,502,0.0,1.0,1.0,1,426,1.0,1.0,1.0,1,857,1.0,1.0,1.0 +hybrid_codedb_embed,q016,Post a comment on a GitHub issue,2,1,388,1.0,1.0,1.0,1,628,1.0,1.0,1.0,1,1039,1.0,1.0,1.0 +hybrid_codedb_embed,q017,Merge the pull request,1,1,299,1.0,1.0,1.0,1,495,1.0,1.0,1.0,1,1093,1.0,1.0,1.0 +hybrid_codedb_embed,q018,Trigger a GitHub Actions workflow,1,1,294,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,735,1.0,1.0,1.0 +hybrid_codedb_embed,q019,Get Dependabot alerts,1,1,269,1.0,1.0,1.0,1,464,1.0,1.0,1.0,1,748,1.0,1.0,1.0 +hybrid_codedb_embed,q020,List Sentry issues,1,1,238,1.0,1.0,1.0,1,393,1.0,1.0,1.0,1,809,1.0,1.0,1.0 +hybrid_codedb_embed,q021,Create a Stripe customer,1,1,223,1.0,1.0,1.0,1,341,1.0,1.0,1.0,1,662,1.0,1.0,1.0 +hybrid_codedb_embed,q022,Refund a Stripe payment,1,1,197,1.0,1.0,1.0,1,295,1.0,1.0,1.0,1,644,1.0,1.0,1.0 +hybrid_codedb_embed,q023,Send an email with Gmail,1,1,224,1.0,1.0,1.0,1,399,1.0,1.0,1.0,1,743,1.0,1.0,1.0 +hybrid_codedb_embed,q024,Search my Gmail inbox,1,1,178,1.0,1.0,1.0,1,315,1.0,1.0,1.0,1,668,1.0,1.0,1.0 +hybrid_codedb_embed,q025,Upload a file to Google Drive,3,1,277,1.0,1.0,1.0,1,446,1.0,1.0,1.0,1,806,1.0,1.0,1.0 +hybrid_codedb_embed,q026,Create a calendar event,1,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,801,1.0,1.0,1.0 +hybrid_codedb_embed,q027,List Kubernetes pods,1,1,184,1.0,1.0,1.0,1,314,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +hybrid_codedb_embed,q028,Get logs from a k8s pod,1,1,262,1.0,1.0,1.0,1,408,1.0,1.0,1.0,1,757,1.0,1.0,1.0 +hybrid_codedb_embed,q029,Run a docker container,1,1,223,1.0,1.0,1.0,1,331,1.0,1.0,1.0,1,587,1.0,1.0,1.0 +hybrid_codedb_embed,q030,Show docker container logs,1,1,226,1.0,1.0,1.0,1,383,1.0,1.0,1.0,1,683,1.0,1.0,1.0 +hybrid_codedb_embed,q031,Navigate the browser to a URL,1,1,180,1.0,1.0,1.0,1,294,1.0,1.0,1.0,1,603,1.0,1.0,1.0 +hybrid_codedb_embed,q032,Take a screenshot of the page,1,1,212,1.0,1.0,1.0,1,316,1.0,1.0,1.0,1,667,1.0,1.0,1.0 +hybrid_codedb_embed,q033,Create a GitLab merge request,1,1,297,1.0,1.0,1.0,1,475,1.0,1.0,1.0,1,983,1.0,1.0,1.0 +hybrid_codedb_embed,q034,Search Notion pages,1,1,180,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,640,1.0,1.0,1.0 +hybrid_codedb_embed,q035,Create a Notion page,1,1,230,1.0,1.0,1.0,1,419,1.0,1.0,1.0,1,834,1.0,1.0,1.0 +hybrid_codedb_embed,q036,Query a Notion database,1,1,261,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,656,1.0,1.0,1.0 +hybrid_codedb_embed,q037,Create a Zendesk ticket,1,1,257,1.0,1.0,1.0,1,426,1.0,1.0,1.0,1,770,1.0,1.0,1.0 +hybrid_codedb_embed,q038,Create a HubSpot contact,1,1,221,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,781,1.0,1.0,1.0 +hybrid_codedb_embed,q039,Create a Salesforce lead,1,1,252,1.0,1.0,1.0,1,380,1.0,1.0,1.0,1,796,1.0,1.0,1.0 +hybrid_codedb_embed,q040,Run a SOQL query,1,1,187,1.0,1.0,1.0,1,293,1.0,1.0,1.0,1,622,1.0,1.0,1.0 +hybrid_codedb_embed,q041,List Shopify orders,1,1,193,1.0,1.0,1.0,1,333,1.0,1.0,1.0,1,618,1.0,1.0,1.0 +hybrid_codedb_embed,q042,Create a PagerDuty incident,1,1,259,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,645,1.0,1.0,1.0 +hybrid_codedb_embed,q043,Who is on call in PagerDuty,1,1,219,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,718,1.0,1.0,1.0 +hybrid_codedb_embed,q044,Query Datadog metrics,1,1,284,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,739,1.0,1.0,1.0 +hybrid_codedb_embed,q045,Search Datadog logs,1,1,242,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,778,1.0,1.0,1.0 +hybrid_codedb_embed,q046,List S3 buckets,1,1,203,1.0,1.0,1.0,1,361,1.0,1.0,1.0,1,577,1.0,1.0,1.0 +hybrid_codedb_embed,q047,Upload an object to S3,1,1,250,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,795,1.0,1.0,1.0 +hybrid_codedb_embed,q048,Purge Cloudflare cache,1,1,183,1.0,1.0,1.0,1,341,1.0,1.0,1.0,1,567,1.0,1.0,1.0 +hybrid_codedb_embed,q049,Create a Cloudflare DNS record,1,1,249,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,808,1.0,1.0,1.0 +hybrid_codedb_embed,q050,List Vercel deployments,1,1,195,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,531,1.0,1.0,1.0 +hybrid_codedb_embed,q051,Insert a row into Supabase,1,1,245,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,636,1.0,1.0,1.0 +hybrid_codedb_embed,q052,Export images from Figma,1,1,239,1.0,1.0,1.0,1,328,1.0,1.0,1.0,1,641,1.0,1.0,1.0 +hybrid_codedb_embed,q053,Send an SMS with Twilio,1,1,249,1.0,1.0,1.0,1,409,1.0,1.0,1.0,1,771,1.0,1.0,1.0 +hybrid_codedb_embed,q054,Create an Okta user,1,1,183,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,795,1.0,1.0,1.0 +hybrid_codedb_embed,q055,Instant query Prometheus,1,1,221,1.0,1.0,1.0,1,375,1.0,1.0,1.0,1,650,1.0,1.0,1.0 +hybrid_codedb_embed,q056,Search Grafana dashboards,1,1,217,1.0,1.0,1.0,1,319,1.0,1.0,1.0,1,627,1.0,1.0,1.0 +hybrid_codedb_embed,q057,Trigger a CircleCI pipeline,1,1,200,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,737,1.0,1.0,1.0 +hybrid_codedb_embed,q058,Run terraform plan,1,1,172,1.0,1.0,1.0,1,295,1.0,1.0,1.0,1,699,1.0,1.0,1.0 +hybrid_codedb_embed,q059,Create an Asana task,1,1,249,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,795,1.0,1.0,1.0 +hybrid_codedb_embed,q060,Create a Trello card,1,1,240,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,627,1.0,1.0,1.0 +hybrid_codedb_embed,q061,Create a ClickUp task,1,1,219,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,742,1.0,1.0,1.0 +hybrid_codedb_embed,q062,Create a Confluence page,1,1,260,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +hybrid_codedb_embed,q063,Send a Discord message,1,1,224,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,758,1.0,1.0,1.0 +hybrid_codedb_embed,q064,Search Elasticsearch,1,1,211,1.0,1.0,1.0,1,324,1.0,1.0,1.0,1,617,1.0,1.0,1.0 +hybrid_codedb_embed,q065,Get a Redis key,1,1,182,1.0,1.0,1.0,1,283,1.0,1.0,1.0,1,549,1.0,1.0,1.0 +hybrid_codedb_embed,q066,Find MongoDB documents,1,1,244,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,672,1.0,1.0,1.0 +hybrid_codedb_embed,q067,List SQLite tables,1,1,142,1.0,1.0,1.0,1,240,1.0,1.0,1.0,1,433,1.0,1.0,1.0 +hybrid_codedb_embed,q068,Describe a postgres table,1,1,188,1.0,1.0,1.0,1,274,1.0,1.0,1.0,1,590,1.0,1.0,1.0 +hybrid_codedb_embed,q069,github_create_issue,2,1,367,1.0,1.0,1.0,1,536,1.0,1.0,1.0,1,1012,1.0,1.0,1.0 +hybrid_codedb_embed,q070,createIssue,39,13,3554,0.0,0.0,0.0,8,3600,0.0,0.0,0.0,4,3528,0.0,0.0,0.0 +hybrid_codedb_embed,q071,list-directory,1,1,182,1.0,1.0,1.0,1,271,1.0,1.0,1.0,1,489,1.0,1.0,1.0 +hybrid_codedb_embed,q072,sequential thinking next thought,1,1,183,1.0,1.0,1.0,1,296,1.0,1.0,1.0,1,579,1.0,1.0,1.0 +hybrid_codedb_embed,q073,What time is it in Tokyo,1,1,239,1.0,1.0,1.0,1,391,1.0,1.0,1.0,1,757,1.0,1.0,1.0 +hybrid_codedb_embed,q074,Convert time between timezones,1,1,234,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +hybrid_codedb_embed,q075,Star a GitHub repo,1,1,226,1.0,1.0,1.0,1,348,1.0,1.0,1.0,1,838,1.0,1.0,1.0 +hybrid_codedb_embed,q076,Fork a repository,1,1,239,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,883,1.0,1.0,1.0 +hybrid_codedb_embed,q077,Get the latest GitHub release,1,1,239,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +hybrid_codedb_embed,q078,Create a gist,1,1,229,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,703,1.0,1.0,1.0 +hybrid_codedb_embed,q079,Search GitHub code,1,1,190,1.0,1.0,1.0,1,340,1.0,1.0,1.0,1,722,1.0,1.0,1.0 +hybrid_codedb_embed,q080,List GitHub notifications,1,1,174,1.0,1.0,1.0,1,307,1.0,1.0,1.0,1,671,1.0,1.0,1.0 +hybrid_codedb_embed,q081,Assign Copilot to an issue,1,1,267,1.0,1.0,1.0,1,500,1.0,1.0,1.0,1,873,1.0,1.0,1.0 +hybrid_codedb_embed,q082,Get secret scanning alerts,1,1,295,1.0,1.0,1.0,1,470,1.0,1.0,1.0,1,723,1.0,1.0,1.0 +hybrid_codedb_embed,q083,Create or update a file on GitHub,1,1,320,1.0,1.0,1.0,1,547,1.0,1.0,1.0,1,975,1.0,1.0,1.0 +hybrid_codedb_embed,q084,List collaborators on a repo,1,1,240,1.0,1.0,1.0,1,379,1.0,1.0,1.0,1,745,1.0,1.0,1.0 +hybrid_codedb_embed,q085,Reply in a Slack thread,1,1,242,1.0,1.0,1.0,1,399,1.0,1.0,1.0,1,708,1.0,1.0,1.0 +hybrid_codedb_embed,q086,Add an emoji reaction on Slack,1,1,271,1.0,1.0,1.0,1,454,1.0,1.0,1.0,1,833,1.0,1.0,1.0 +hybrid_codedb_embed,q087,Search Slack messages,1,1,232,1.0,1.0,1.0,1,374,1.0,1.0,1.0,1,668,1.0,1.0,1.0 +hybrid_codedb_embed,q088,Create a Slack channel,1,1,196,1.0,1.0,1.0,1,339,1.0,1.0,1.0,1,751,1.0,1.0,1.0 +hybrid_codedb_embed,q089,Transition a Jira issue,1,1,222,1.0,1.0,1.0,1,342,1.0,1.0,1.0,1,760,1.0,1.0,1.0 +hybrid_codedb_embed,q090,Search Jira with JQL,1,1,203,1.0,1.0,1.0,1,357,1.0,1.0,1.0,1,633,1.0,1.0,1.0 +hybrid_codedb_embed,q091,Add a worklog to Jira,1,1,214,1.0,1.0,1.0,1,483,1.0,1.0,1.0,1,892,1.0,1.0,1.0 +hybrid_codedb_embed,q092,List Linear teams,1,1,184,1.0,1.0,1.0,1,330,1.0,1.0,1.0,1,732,1.0,1.0,1.0 +hybrid_codedb_embed,q093,Comment on a Linear issue,1,1,218,1.0,1.0,1.0,1,464,1.0,1.0,1.0,1,845,1.0,1.0,1.0 +hybrid_codedb_embed,q094,Move a local file,1,1,225,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,694,1.0,1.0,1.0 +hybrid_codedb_embed,q095,Recursively search files by pattern,1,1,192,1.0,1.0,1.0,1,302,1.0,1.0,1.0,1,695,1.0,1.0,1.0 +hybrid_codedb_embed,q096,Edit a file in place with old and new string,1,1,249,1.0,1.0,1.0,1,413,1.0,1.0,1.0,1,693,1.0,1.0,1.0 +hybrid_codedb_embed,q097,Read the whole knowledge graph,1,1,154,1.0,1.0,1.0,1,262,1.0,1.0,1.0,1,533,1.0,1.0,1.0 +hybrid_codedb_embed,q098,Search memory nodes,1,1,160,1.0,1.0,1.0,1,288,1.0,1.0,1.0,1,568,1.0,1.0,1.0 +hybrid_codedb_embed,q099,Show unstaged git diff,1,1,190,1.0,1.0,1.0,1,313,1.0,1.0,1.0,1,523,1.0,1.0,1.0 +hybrid_codedb_embed,q100,Stage files with git add,1,1,244,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,840,1.0,1.0,1.0 +hybrid_codedb_embed,q101,Create a Shopify product,1,1,240,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +hybrid_codedb_embed,q102,Cancel a Stripe subscription,1,1,176,1.0,1.0,1.0,1,290,1.0,1.0,1.0,1,610,1.0,1.0,1.0 +hybrid_codedb_embed,q103,Create a payment intent,1,1,234,1.0,1.0,1.0,1,282,1.0,1.0,1.0,1,636,1.0,1.0,1.0 +hybrid_codedb_embed,q104,Acknowledge a PagerDuty incident,1,1,223,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,649,1.0,1.0,1.0 +hybrid_codedb_embed,q105,Create a Datadog monitor,1,1,259,1.0,1.0,1.0,1,459,1.0,1.0,1.0,1,700,1.0,1.0,1.0 +hybrid_codedb_embed,q106,Resolve a Sentry issue,2,1,274,1.0,1.0,1.0,1,412,1.0,1.0,1.0,1,759,1.0,1.0,1.0 +hybrid_codedb_embed,q107,Scale a Kubernetes deployment,1,1,226,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +hybrid_codedb_embed,q108,Apply a Kubernetes manifest,1,1,195,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +hybrid_codedb_embed,q109,Stop a docker container,1,1,188,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,664,1.0,1.0,1.0 +hybrid_codedb_embed,q110,Build a docker image,1,1,211,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,642,1.0,1.0,1.0 +hybrid_codedb_embed,q111,Click an element in Playwright,1,1,216,1.0,1.0,1.0,1,326,1.0,1.0,1.0,1,566,1.0,1.0,1.0 +hybrid_codedb_embed,q112,Type text into a browser input,1,1,215,1.0,1.0,1.0,1,316,1.0,1.0,1.0,1,636,1.0,1.0,1.0 +hybrid_codedb_embed,q113,List GitLab projects,1,1,216,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,641,1.0,1.0,1.0 +hybrid_codedb_embed,q114,Create a Bitbucket pull request,1,1,439,1.0,1.0,1.0,1,606,1.0,1.0,1.0,1,1091,1.0,1.0,1.0 +hybrid_codedb_embed,q115,Create a Gmail draft,1,1,188,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,805,1.0,1.0,1.0 +hybrid_codedb_embed,q116,Share a Google Drive file,1,1,271,1.0,1.0,1.0,1,417,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +hybrid_codedb_embed,q117,List Google Calendar events,1,1,267,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +hybrid_codedb_embed,q118,Delete a calendar event,1,1,263,1.0,1.0,1.0,1,406,1.0,1.0,1.0,1,704,1.0,1.0,1.0 +hybrid_codedb_embed,q119,List Vercel projects,1,1,151,1.0,1.0,1.0,1,253,1.0,1.0,1.0,1,550,1.0,1.0,1.0 +hybrid_codedb_embed,q120,Select rows from Supabase,1,1,237,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,689,1.0,1.0,1.0 +hybrid_codedb_embed,q121,Post a Figma comment,1,1,204,1.0,1.0,1.0,1,308,1.0,1.0,1.0,1,688,1.0,1.0,1.0 +hybrid_codedb_embed,q122,List Okta users,1,1,161,1.0,1.0,1.0,1,295,1.0,1.0,1.0,1,634,1.0,1.0,1.0 +hybrid_codedb_embed,q123,Range query Prometheus,1,1,221,1.0,1.0,1.0,1,375,1.0,1.0,1.0,1,674,1.0,1.0,1.0 +hybrid_codedb_embed,q124,Create a Grafana annotation,1,1,218,1.0,1.0,1.0,1,326,1.0,1.0,1.0,1,692,1.0,1.0,1.0 +hybrid_codedb_embed,q125,Apply terraform,1,1,190,1.0,1.0,1.0,1,238,1.0,1.0,1.0,1,562,1.0,1.0,1.0 +hybrid_codedb_embed,q126,List Asana tasks,1,1,192,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,624,1.0,1.0,1.0 +hybrid_codedb_embed,q127,Comment on a Trello card,1,1,260,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,715,1.0,1.0,1.0 +hybrid_codedb_embed,q128,Update a ClickUp task,1,1,219,1.0,1.0,1.0,1,399,1.0,1.0,1.0,1,709,1.0,1.0,1.0 +hybrid_codedb_embed,q129,Search Confluence,1,1,221,1.0,1.0,1.0,1,371,1.0,1.0,1.0,1,636,1.0,1.0,1.0 +hybrid_codedb_embed,q130,Create a HubSpot deal,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,781,1.0,1.0,1.0 +hybrid_codedb_embed,q131,List Zendesk tickets,1,1,189,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,573,1.0,1.0,1.0 +hybrid_codedb_embed,q132,List Discord channels,1,1,184,1.0,1.0,1.0,1,300,1.0,1.0,1.0,1,594,1.0,1.0,1.0 +hybrid_codedb_embed,q133,Set a Redis key,1,1,182,1.0,1.0,1.0,1,280,1.0,1.0,1.0,1,563,1.0,1.0,1.0 +hybrid_codedb_embed,q134,Insert MongoDB documents,1,1,256,1.0,1.0,1.0,1,399,1.0,1.0,1.0,1,689,1.0,1.0,1.0 +hybrid_codedb_embed,q135,Run a MySQL query,1,1,179,1.0,1.0,1.0,1,286,1.0,1.0,1.0,1,594,1.0,1.0,1.0 +hybrid_codedb_embed,q136,Index a document in Elasticsearch,1,1,242,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,567,1.0,1.0,1.0 +hybrid_codedb_embed,q137,Get an S3 object,1,1,249,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,624,1.0,1.0,1.0 +hybrid_codedb_embed,q138,List Cloudflare zones,1,1,182,1.0,1.0,1.0,1,285,1.0,1.0,1.0,1,525,1.0,1.0,1.0 +hybrid_codedb_embed,q139,Create a Stripe invoice,1,1,205,1.0,1.0,1.0,1,360,1.0,1.0,1.0,1,647,1.0,1.0,1.0 +hybrid_codedb_embed,q140,List GitHub Actions workflows,1,1,268,1.0,1.0,1.0,1,453,1.0,1.0,1.0,1,778,1.0,1.0,1.0 +hybrid_codedb_embed,q141,Get GitHub Actions job logs,1,1,269,1.0,1.0,1.0,1,442,1.0,1.0,1.0,1,730,1.0,1.0,1.0 +hybrid_codedb_embed,q142,Create a GitHub repository,1,1,346,1.0,1.0,1.0,1,556,1.0,1.0,1.0,1,968,1.0,1.0,1.0 +hybrid_codedb_embed,q143,Checkout a git branch,1,1,212,1.0,1.0,1.0,1,366,1.0,1.0,1.0,1,771,1.0,1.0,1.0 +hybrid_codedb_embed,q144,Show git log,1,1,194,1.0,1.0,1.0,1,312,1.0,1.0,1.0,1,629,1.0,1.0,1.0 +hybrid_codedb_embed,q145,List postgres schemas,1,1,169,1.0,1.0,1.0,1,262,1.0,1.0,1.0,1,455,1.0,1.0,1.0 +hybrid_codedb_embed,q146,Open nodes in the knowledge graph,1,1,163,1.0,1.0,1.0,1,262,1.0,1.0,1.0,1,564,1.0,1.0,1.0 +hybrid_codedb_embed,q147,Get file metadata from the filesystem,1,1,190,1.0,1.0,1.0,1,324,1.0,1.0,1.0,1,717,1.0,1.0,1.0 +hybrid_codedb_embed,q148,Write a new local file,1,1,269,1.0,1.0,1.0,1,384,1.0,1.0,1.0,1,856,1.0,1.0,1.0 +hybrid_codedb_embed,q149,Search GitHub issues,1,1,208,1.0,1.0,1.0,1,363,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +hybrid_codedb_embed,q150,Get the authenticated GitHub user,2,1,179,1.0,1.0,1.0,1,285,1.0,1.0,1.0,1,625,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q001,Create a GitHub issue,1,1,264,1.0,1.0,1.0,1,495,1.0,1.0,1.0,1,884,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q002,Create a Jira ticket,4,2,471,0.0,1.0,1.0,1,383,1.0,1.0,1.0,1,774,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q003,Send a Slack message,1,1,249,1.0,1.0,1.0,1,366,1.0,1.0,1.0,1,734,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q004,List open pull requests,1,1,234,1.0,1.0,1.0,1,519,1.0,1.0,1.0,1,1035,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q005,Open a pull request on GitHub,1,1,458,1.0,1.0,1.0,1,643,1.0,1.0,1.0,1,1013,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q006,Run a SQL query against postgres,1,1,167,1.0,1.0,1.0,1,272,1.0,1.0,1.0,1,544,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q007,Read a local file from disk,3,1,213,1.0,1.0,1.0,1,418,1.0,1.0,1.0,1,813,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q008,What files are in this folder,9,3,690,0.0,0.0,1.0,2,753,0.0,1.0,1.0,1,731,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q009,Remember this fact in the knowledge graph,4,2,383,0.0,1.0,1.0,1,311,1.0,1.0,1.0,1,612,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q010,Search the web with Brave,1,1,245,1.0,1.0,1.0,1,317,1.0,1.0,1.0,1,653,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q011,Fetch the contents of a URL,1,1,258,1.0,1.0,1.0,1,388,1.0,1.0,1.0,1,653,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q012,Show git status,1,1,178,1.0,1.0,1.0,1,283,1.0,1.0,1.0,1,639,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q013,Commit staged changes,1,1,218,1.0,1.0,1.0,1,328,1.0,1.0,1.0,1,611,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q014,Create a Linear issue,4,2,543,0.0,1.0,1.0,1,471,1.0,1.0,1.0,1,840,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q015,Make a task in my project tracker Linear,3,1,290,1.0,1.0,1.0,1,451,1.0,1.0,1.0,1,733,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q016,Post a comment on a GitHub issue,5,2,537,0.0,1.0,1.0,1,414,1.0,1.0,1.0,1,971,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q017,Merge the pull request,2,1,283,1.0,1.0,1.0,1,516,1.0,1.0,1.0,1,1093,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q018,Trigger a GitHub Actions workflow,1,1,296,1.0,1.0,1.0,1,449,1.0,1.0,1.0,1,733,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q019,Get Dependabot alerts,1,1,287,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q020,List Sentry issues,1,1,268,1.0,1.0,1.0,1,426,1.0,1.0,1.0,1,813,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q021,Create a Stripe customer,2,1,186,1.0,1.0,1.0,1,331,1.0,1.0,1.0,1,658,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q022,Refund a Stripe payment,1,1,234,1.0,1.0,1.0,1,347,1.0,1.0,1.0,1,613,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q023,Send an email with Gmail,1,1,262,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,668,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q024,Search my Gmail inbox,7,3,642,0.0,0.0,1.0,2,711,0.0,1.0,1.0,1,691,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q025,Upload a file to Google Drive,5,2,526,0.0,1.0,1.0,1,415,1.0,1.0,1.0,1,907,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q026,Create a calendar event,2,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,805,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q027,List Kubernetes pods,1,1,230,1.0,1.0,1.0,1,370,1.0,1.0,1.0,1,665,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q028,Get logs from a k8s pod,1,1,294,1.0,1.0,1.0,1,455,1.0,1.0,1.0,1,759,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q029,Run a docker container,2,1,219,1.0,1.0,1.0,1,348,1.0,1.0,1.0,1,688,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q030,Show docker container logs,2,1,188,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,747,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q031,Navigate the browser to a URL,1,1,180,1.0,1.0,1.0,1,265,1.0,1.0,1.0,1,742,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q032,Take a screenshot of the page,1,1,212,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,609,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q033,Create a GitLab merge request,3,1,297,1.0,1.0,1.0,1,475,1.0,1.0,1.0,1,841,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q034,Search Notion pages,1,1,260,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,686,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q035,Create a Notion page,8,3,695,0.0,0.0,1.0,2,727,0.0,1.0,1.0,1,706,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q036,Query a Notion database,1,1,261,1.0,1.0,1.0,1,387,1.0,1.0,1.0,1,730,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q037,Create a Zendesk ticket,2,1,257,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,761,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q038,Create a HubSpot contact,1,1,216,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,779,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q039,Create a Salesforce lead,1,1,273,1.0,1.0,1.0,1,423,1.0,1.0,1.0,1,682,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q040,Run a SOQL query,1,1,186,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,724,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q041,List Shopify orders,1,1,230,1.0,1.0,1.0,1,376,1.0,1.0,1.0,1,606,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q042,Create a PagerDuty incident,3,1,259,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,838,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q043,Who is on call in PagerDuty,1,1,213,1.0,1.0,1.0,1,346,1.0,1.0,1.0,1,749,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q044,Query Datadog metrics,1,1,300,1.0,1.0,1.0,1,449,1.0,1.0,1.0,1,668,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q045,Search Datadog logs,1,1,284,1.0,1.0,1.0,1,493,1.0,1.0,1.0,1,863,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q046,List S3 buckets,1,1,133,1.0,1.0,1.0,1,262,1.0,1.0,1.0,1,590,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q047,Upload an object to S3,1,1,262,1.0,1.0,1.0,1,487,1.0,1.0,1.0,1,857,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q048,Purge Cloudflare cache,1,1,231,1.0,1.0,1.0,1,286,1.0,1.0,1.0,1,643,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q049,Create a Cloudflare DNS record,1,1,217,1.0,1.0,1.0,1,363,1.0,1.0,1.0,1,660,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q050,List Vercel deployments,1,1,177,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,513,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q051,Insert a row into Supabase,1,1,238,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,620,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q052,Export images from Figma,1,1,196,1.0,1.0,1.0,1,325,1.0,1.0,1.0,1,730,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q053,Send an SMS with Twilio,1,1,249,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,665,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q054,Create an Okta user,3,1,183,1.0,1.0,1.0,1,285,1.0,1.0,1.0,1,657,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q055,Instant query Prometheus,1,1,221,1.0,1.0,1.0,1,274,1.0,1.0,1.0,1,595,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q056,Search Grafana dashboards,1,1,217,1.0,1.0,1.0,1,306,1.0,1.0,1.0,1,624,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q057,Trigger a CircleCI pipeline,1,1,200,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,664,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q058,Run terraform plan,1,1,172,1.0,1.0,1.0,1,227,1.0,1.0,1.0,1,598,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q059,Create an Asana task,3,1,267,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,783,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q060,Create a Trello card,1,1,259,1.0,1.0,1.0,1,316,1.0,1.0,1.0,1,784,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q061,Create a ClickUp task,3,1,257,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,751,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q062,Create a Confluence page,2,1,260,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,703,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q063,Send a Discord message,2,1,224,1.0,1.0,1.0,1,343,1.0,1.0,1.0,1,776,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q064,Search Elasticsearch,2,1,239,1.0,1.0,1.0,1,328,1.0,1.0,1.0,1,646,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q065,Get a Redis key,1,1,185,1.0,1.0,1.0,1,280,1.0,1.0,1.0,1,586,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q066,Find MongoDB documents,2,1,244,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,738,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q067,List SQLite tables,1,1,153,1.0,1.0,1.0,1,252,1.0,1.0,1.0,1,458,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q068,Describe a postgres table,1,1,170,1.0,1.0,1.0,1,272,1.0,1.0,1.0,1,543,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q069,github_create_issue,1,1,303,1.0,1.0,1.0,1,487,1.0,1.0,1.0,1,1053,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q070,createIssue,5,2,598,0.0,1.0,1.0,1,481,1.0,1.0,1.0,1,917,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q071,list-directory,2,1,182,1.0,1.0,1.0,1,296,1.0,1.0,1.0,1,644,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q072,sequential thinking next thought,1,1,186,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,713,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q073,What time is it in Tokyo,1,1,190,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,748,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q074,Convert time between timezones,1,1,204,1.0,1.0,1.0,1,335,1.0,1.0,1.0,1,650,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q075,Star a GitHub repo,1,1,195,1.0,1.0,1.0,1,301,1.0,1.0,1.0,1,755,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q076,Fork a repository,1,1,239,1.0,1.0,1.0,1,425,1.0,1.0,1.0,1,720,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q077,Get the latest GitHub release,1,1,206,1.0,1.0,1.0,1,344,1.0,1.0,1.0,1,640,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q078,Create a gist,1,1,229,1.0,1.0,1.0,1,385,1.0,1.0,1.0,1,770,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q079,Search GitHub code,3,1,246,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,673,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q080,List GitHub notifications,1,1,184,1.0,1.0,1.0,1,282,1.0,1.0,1.0,1,577,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q081,Assign Copilot to an issue,1,1,294,1.0,1.0,1.0,1,457,1.0,1.0,1.0,1,983,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q082,Get secret scanning alerts,1,1,295,1.0,1.0,1.0,1,470,1.0,1.0,1.0,1,747,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q083,Create or update a file on GitHub,1,1,276,1.0,1.0,1.0,1,446,1.0,1.0,1.0,1,864,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q084,List collaborators on a repo,1,1,254,1.0,1.0,1.0,1,412,1.0,1.0,1.0,1,837,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q085,Reply in a Slack thread,1,1,256,1.0,1.0,1.0,1,391,1.0,1.0,1.0,1,705,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q086,Add an emoji reaction on Slack,1,1,252,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,741,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q087,Search Slack messages,1,1,226,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,703,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q088,Create a Slack channel,10,4,837,0.0,0.0,0.0,2,705,0.0,1.0,1.0,1,685,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q089,Transition a Jira issue,1,1,213,1.0,1.0,1.0,1,337,1.0,1.0,1.0,1,789,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q090,Search Jira with JQL,1,1,179,1.0,1.0,1.0,1,337,1.0,1.0,1.0,1,734,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q091,Add a worklog to Jira,1,1,252,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,821,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q092,List Linear teams,1,1,249,1.0,1.0,1.0,1,437,1.0,1.0,1.0,1,728,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q093,Comment on a Linear issue,5,2,552,0.0,1.0,1.0,1,426,1.0,1.0,1.0,1,871,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q094,Move a local file,1,1,211,1.0,1.0,1.0,1,324,1.0,1.0,1.0,1,744,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q095,Recursively search files by pattern,1,1,280,1.0,1.0,1.0,1,419,1.0,1.0,1.0,1,721,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q096,Edit a file in place with old and new string,1,1,228,1.0,1.0,1.0,1,369,1.0,1.0,1.0,1,772,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q097,Read the whole knowledge graph,1,1,155,1.0,1.0,1.0,1,255,1.0,1.0,1.0,1,523,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q098,Search memory nodes,1,1,202,1.0,1.0,1.0,1,305,1.0,1.0,1.0,1,512,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q099,Show unstaged git diff,1,1,190,1.0,1.0,1.0,1,297,1.0,1.0,1.0,1,540,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q100,Stage files with git add,9,3,634,0.0,0.0,1.0,2,675,0.0,1.0,1.0,1,654,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q101,Create a Shopify product,1,1,240,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q102,Cancel a Stripe subscription,1,1,200,1.0,1.0,1.0,1,328,1.0,1.0,1.0,1,621,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q103,Create a payment intent,1,1,234,1.0,1.0,1.0,1,312,1.0,1.0,1.0,1,802,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q104,Acknowledge a PagerDuty incident,1,1,223,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,626,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q105,Create a Datadog monitor,1,1,259,1.0,1.0,1.0,1,459,1.0,1.0,1.0,1,832,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q106,Resolve a Sentry issue,1,1,230,1.0,1.0,1.0,1,391,1.0,1.0,1.0,1,823,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q107,Scale a Kubernetes deployment,1,1,262,1.0,1.0,1.0,1,374,1.0,1.0,1.0,1,675,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q108,Apply a Kubernetes manifest,1,1,273,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,670,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q109,Stop a docker container,1,1,219,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,587,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q110,Build a docker image,1,1,238,1.0,1.0,1.0,1,330,1.0,1.0,1.0,1,571,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q111,Click an element in Playwright,1,1,216,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,667,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q112,Type text into a browser input,1,1,217,1.0,1.0,1.0,1,316,1.0,1.0,1.0,1,614,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q113,List GitLab projects,1,1,178,1.0,1.0,1.0,1,296,1.0,1.0,1.0,1,565,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q114,Create a Bitbucket pull request,3,1,313,1.0,1.0,1.0,1,509,1.0,1.0,1.0,1,1020,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q115,Create a Gmail draft,1,1,227,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,730,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q116,Share a Google Drive file,2,1,218,1.0,1.0,1.0,1,417,1.0,1.0,1.0,1,778,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q117,List Google Calendar events,2,1,269,1.0,1.0,1.0,1,441,1.0,1.0,1.0,1,770,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q118,Delete a calendar event,1,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,754,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q119,List Vercel projects,1,1,177,1.0,1.0,1.0,1,286,1.0,1.0,1.0,1,550,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q120,Select rows from Supabase,1,1,237,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,737,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q121,Post a Figma comment,1,1,204,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,781,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q122,List Okta users,2,1,179,1.0,1.0,1.0,1,298,1.0,1.0,1.0,1,595,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q123,Range query Prometheus,1,1,221,1.0,1.0,1.0,1,274,1.0,1.0,1.0,1,595,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q124,Create a Grafana annotation,1,1,218,1.0,1.0,1.0,1,296,1.0,1.0,1.0,1,786,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q125,Apply terraform,1,1,190,1.0,1.0,1.0,1,238,1.0,1.0,1.0,1,595,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q126,List Asana tasks,1,1,244,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,641,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q127,Comment on a Trello card,1,1,260,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,892,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q128,Update a ClickUp task,1,1,257,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,761,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q129,Search Confluence,2,1,221,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,731,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q130,Create a HubSpot deal,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,872,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q131,List Zendesk tickets,1,1,189,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,604,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q132,List Discord channels,1,1,205,1.0,1.0,1.0,1,321,1.0,1.0,1.0,1,573,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q133,Set a Redis key,2,1,185,1.0,1.0,1.0,1,280,1.0,1.0,1.0,1,605,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q134,Insert MongoDB documents,1,1,256,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,607,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q135,Run a MySQL query,1,1,182,1.0,1.0,1.0,1,266,1.0,1.0,1.0,1,543,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q136,Index a document in Elasticsearch,1,1,242,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q137,Get an S3 object,2,1,249,1.0,1.0,1.0,1,373,1.0,1.0,1.0,1,687,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q138,List Cloudflare zones,1,1,230,1.0,1.0,1.0,1,363,1.0,1.0,1.0,1,600,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q139,Create a Stripe invoice,1,1,216,1.0,1.0,1.0,1,330,1.0,1.0,1.0,1,575,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q140,List GitHub Actions workflows,1,1,268,1.0,1.0,1.0,1,408,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q141,Get GitHub Actions job logs,1,1,296,1.0,1.0,1.0,1,449,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q142,Create a GitHub repository,2,1,239,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,707,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q143,Checkout a git branch,1,1,212,1.0,1.0,1.0,1,373,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q144,Show git log,2,1,188,1.0,1.0,1.0,1,294,1.0,1.0,1.0,1,634,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q145,List postgres schemas,1,1,169,1.0,1.0,1.0,1,277,1.0,1.0,1.0,1,514,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q146,Open nodes in the knowledge graph,1,1,180,1.0,1.0,1.0,1,308,1.0,1.0,1.0,1,507,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q147,Get file metadata from the filesystem,3,1,191,1.0,1.0,1.0,1,339,1.0,1.0,1.0,1,692,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q148,Write a new local file,7,3,667,0.0,0.0,1.0,2,737,0.0,1.0,1.0,1,716,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q149,Search GitHub issues,4,2,459,0.0,1.0,1.0,1,367,1.0,1.0,1.0,1,714,1.0,1.0,1.0 +hybrid_codedb_trigram_bm25,q150,Get the authenticated GitHub user,11,4,751,0.0,0.0,0.0,3,994,0.0,0.0,1.0,2,1377,0.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q001,Create a GitHub issue,1,1,388,1.0,1.0,1.0,1,598,1.0,1.0,1.0,1,1111,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q002,Create a Jira ticket,1,1,333,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,795,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q003,Send a Slack message,1,1,232,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,755,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q004,List open pull requests,1,1,234,1.0,1.0,1.0,1,489,1.0,1.0,1.0,1,836,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q005,Open a pull request on GitHub,1,1,368,1.0,1.0,1.0,1,653,1.0,1.0,1.0,1,1144,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q006,Run a SQL query against postgres,1,1,183,1.0,1.0,1.0,1,272,1.0,1.0,1.0,1,551,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q007,Read a local file from disk,1,1,223,1.0,1.0,1.0,1,428,1.0,1.0,1.0,1,797,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q008,What files are in this folder,6,2,473,0.0,1.0,1.0,2,714,0.0,1.0,1.0,1,692,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q009,Remember this fact in the knowledge graph,2,1,156,1.0,1.0,1.0,1,261,1.0,1.0,1.0,1,511,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q010,Search the web with Brave,1,1,258,1.0,1.0,1.0,1,374,1.0,1.0,1.0,1,712,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q011,Fetch the contents of a URL,1,1,264,1.0,1.0,1.0,1,380,1.0,1.0,1.0,1,707,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q012,Show git status,1,1,196,1.0,1.0,1.0,1,312,1.0,1.0,1.0,1,615,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q013,Commit staged changes,1,1,215,1.0,1.0,1.0,1,327,1.0,1.0,1.0,1,596,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q014,Create a Linear issue,1,1,293,1.0,1.0,1.0,1,463,1.0,1.0,1.0,1,876,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q015,Make a task in my project tracker Linear,3,1,326,1.0,1.0,1.0,1,494,1.0,1.0,1.0,1,806,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q016,Post a comment on a GitHub issue,2,1,333,1.0,1.0,1.0,1,569,1.0,1.0,1.0,1,1072,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q017,Merge the pull request,1,1,283,1.0,1.0,1.0,1,468,1.0,1.0,1.0,1,1121,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q018,Trigger a GitHub Actions workflow,1,1,294,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,729,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q019,Get Dependabot alerts,1,1,269,1.0,1.0,1.0,1,412,1.0,1.0,1.0,1,686,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q020,List Sentry issues,1,1,238,1.0,1.0,1.0,1,380,1.0,1.0,1.0,1,796,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q021,Create a Stripe customer,1,1,186,1.0,1.0,1.0,1,341,1.0,1.0,1.0,1,662,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q022,Refund a Stripe payment,1,1,234,1.0,1.0,1.0,1,332,1.0,1.0,1.0,1,632,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q023,Send an email with Gmail,1,1,224,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,716,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q024,Search my Gmail inbox,1,1,205,1.0,1.0,1.0,1,315,1.0,1.0,1.0,1,700,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q025,Upload a file to Google Drive,5,2,523,0.0,1.0,1.0,1,446,1.0,1.0,1.0,1,827,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q026,Create a calendar event,1,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,810,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q027,List Kubernetes pods,1,1,184,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,665,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q028,Get logs from a k8s pod,1,1,294,1.0,1.0,1.0,1,455,1.0,1.0,1.0,1,822,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q029,Run a docker container,1,1,223,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,597,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q030,Show docker container logs,1,1,188,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q031,Navigate the browser to a URL,1,1,180,1.0,1.0,1.0,1,294,1.0,1.0,1.0,1,536,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q032,Take a screenshot of the page,1,1,226,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,711,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q033,Create a GitLab merge request,1,1,297,1.0,1.0,1.0,1,475,1.0,1.0,1.0,1,844,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q034,Search Notion pages,1,1,195,1.0,1.0,1.0,1,318,1.0,1.0,1.0,1,639,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q035,Create a Notion page,2,1,230,1.0,1.0,1.0,1,371,1.0,1.0,1.0,1,784,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q036,Query a Notion database,1,1,279,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q037,Create a Zendesk ticket,1,1,257,1.0,1.0,1.0,1,429,1.0,1.0,1.0,1,723,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q038,Create a HubSpot contact,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,899,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q039,Create a Salesforce lead,1,1,252,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,682,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q040,Run a SOQL query,1,1,187,1.0,1.0,1.0,1,332,1.0,1.0,1.0,1,616,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q041,List Shopify orders,1,1,208,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,606,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q042,Create a PagerDuty incident,1,1,259,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,705,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q043,Who is on call in PagerDuty,1,1,219,1.0,1.0,1.0,1,363,1.0,1.0,1.0,1,738,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q044,Query Datadog metrics,1,1,284,1.0,1.0,1.0,1,449,1.0,1.0,1.0,1,767,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q045,Search Datadog logs,1,1,242,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,860,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q046,List S3 buckets,1,1,186,1.0,1.0,1.0,1,327,1.0,1.0,1.0,1,598,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q047,Upload an object to S3,1,1,250,1.0,1.0,1.0,1,437,1.0,1.0,1.0,1,868,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q048,Purge Cloudflare cache,1,1,183,1.0,1.0,1.0,1,286,1.0,1.0,1.0,1,643,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q049,Create a Cloudflare DNS record,1,1,249,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,660,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q050,List Vercel deployments,1,1,173,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,506,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q051,Insert a row into Supabase,1,1,230,1.0,1.0,1.0,1,347,1.0,1.0,1.0,1,598,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q052,Export images from Figma,1,1,239,1.0,1.0,1.0,1,328,1.0,1.0,1.0,1,695,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q053,Send an SMS with Twilio,1,1,249,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,668,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q054,Create an Okta user,1,1,183,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,676,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q055,Instant query Prometheus,1,1,221,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,680,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q056,Search Grafana dashboards,1,1,217,1.0,1.0,1.0,1,321,1.0,1.0,1.0,1,681,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q057,Trigger a CircleCI pipeline,1,1,200,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,664,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q058,Run terraform plan,1,1,172,1.0,1.0,1.0,1,319,1.0,1.0,1.0,1,717,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q059,Create an Asana task,1,1,267,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,825,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q060,Create a Trello card,1,1,240,1.0,1.0,1.0,1,316,1.0,1.0,1.0,1,700,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q061,Create a ClickUp task,1,1,257,1.0,1.0,1.0,1,415,1.0,1.0,1.0,1,673,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q062,Create a Confluence page,1,1,260,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,652,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q063,Send a Discord message,1,1,224,1.0,1.0,1.0,1,360,1.0,1.0,1.0,1,628,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q064,Search Elasticsearch,1,1,239,1.0,1.0,1.0,1,343,1.0,1.0,1.0,1,672,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q065,Get a Redis key,1,1,182,1.0,1.0,1.0,1,281,1.0,1.0,1.0,1,589,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q066,Find MongoDB documents,1,1,244,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,631,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q067,List SQLite tables,1,1,151,1.0,1.0,1.0,1,240,1.0,1.0,1.0,1,420,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q068,Describe a postgres table,1,1,188,1.0,1.0,1.0,1,278,1.0,1.0,1.0,1,543,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q069,github_create_issue,1,1,386,1.0,1.0,1.0,1,596,1.0,1.0,1.0,1,1019,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q070,createIssue,1,1,340,1.0,1.0,1.0,1,546,1.0,1.0,1.0,1,961,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q071,list-directory,2,1,182,1.0,1.0,1.0,1,296,1.0,1.0,1.0,1,607,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q072,sequential thinking next thought,1,1,186,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,713,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q073,What time is it in Tokyo,1,1,222,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,678,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q074,Convert time between timezones,1,1,204,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,650,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q075,Star a GitHub repo,1,1,205,1.0,1.0,1.0,1,333,1.0,1.0,1.0,1,684,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q076,Fork a repository,1,1,235,1.0,1.0,1.0,1,384,1.0,1.0,1.0,1,789,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q077,Get the latest GitHub release,1,1,239,1.0,1.0,1.0,1,341,1.0,1.0,1.0,1,680,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q078,Create a gist,1,1,229,1.0,1.0,1.0,1,391,1.0,1.0,1.0,1,739,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q079,Search GitHub code,1,1,283,1.0,1.0,1.0,1,395,1.0,1.0,1.0,1,660,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q080,List GitHub notifications,1,1,189,1.0,1.0,1.0,1,286,1.0,1.0,1.0,1,595,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q081,Assign Copilot to an issue,1,1,267,1.0,1.0,1.0,1,402,1.0,1.0,1.0,1,896,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q082,Get secret scanning alerts,1,1,295,1.0,1.0,1.0,1,470,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q083,Create or update a file on GitHub,1,1,367,1.0,1.0,1.0,1,548,1.0,1.0,1.0,1,966,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q084,List collaborators on a repo,1,1,264,1.0,1.0,1.0,1,416,1.0,1.0,1.0,1,824,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q085,Reply in a Slack thread,1,1,254,1.0,1.0,1.0,1,359,1.0,1.0,1.0,1,655,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q086,Add an emoji reaction on Slack,1,1,252,1.0,1.0,1.0,1,375,1.0,1.0,1.0,1,768,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q087,Search Slack messages,1,1,232,1.0,1.0,1.0,1,336,1.0,1.0,1.0,1,652,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q088,Create a Slack channel,1,1,196,1.0,1.0,1.0,1,342,1.0,1.0,1.0,1,751,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q089,Transition a Jira issue,1,1,208,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,724,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q090,Search Jira with JQL,1,1,219,1.0,1.0,1.0,1,312,1.0,1.0,1.0,1,647,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q091,Add a worklog to Jira,1,1,249,1.0,1.0,1.0,1,491,1.0,1.0,1.0,1,823,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q092,List Linear teams,1,1,184,1.0,1.0,1.0,1,277,1.0,1.0,1.0,1,755,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q093,Comment on a Linear issue,1,1,276,1.0,1.0,1.0,1,464,1.0,1.0,1.0,1,888,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q094,Move a local file,1,1,214,1.0,1.0,1.0,1,358,1.0,1.0,1.0,1,688,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q095,Recursively search files by pattern,1,1,280,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,735,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q096,Edit a file in place with old and new string,1,1,227,1.0,1.0,1.0,1,350,1.0,1.0,1.0,1,682,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q097,Read the whole knowledge graph,1,1,152,1.0,1.0,1.0,1,260,1.0,1.0,1.0,1,562,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q098,Search memory nodes,1,1,202,1.0,1.0,1.0,1,304,1.0,1.0,1.0,1,546,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q099,Show unstaged git diff,1,1,190,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,537,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q100,Stage files with git add,1,1,260,1.0,1.0,1.0,1,433,1.0,1.0,1.0,1,834,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q101,Create a Shopify product,1,1,240,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q102,Cancel a Stripe subscription,1,1,168,1.0,1.0,1.0,1,282,1.0,1.0,1.0,1,610,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q103,Create a payment intent,1,1,234,1.0,1.0,1.0,1,312,1.0,1.0,1.0,1,675,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q104,Acknowledge a PagerDuty incident,1,1,223,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,626,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q105,Create a Datadog monitor,1,1,259,1.0,1.0,1.0,1,459,1.0,1.0,1.0,1,747,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q106,Resolve a Sentry issue,2,1,251,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,754,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q107,Scale a Kubernetes deployment,1,1,252,1.0,1.0,1.0,1,371,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q108,Apply a Kubernetes manifest,1,1,221,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q109,Stop a docker container,1,1,188,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,587,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q110,Build a docker image,1,1,224,1.0,1.0,1.0,1,317,1.0,1.0,1.0,1,547,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q111,Click an element in Playwright,1,1,216,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,573,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q112,Type text into a browser input,1,1,217,1.0,1.0,1.0,1,302,1.0,1.0,1.0,1,577,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q113,List GitLab projects,1,1,209,1.0,1.0,1.0,1,328,1.0,1.0,1.0,1,587,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q114,Create a Bitbucket pull request,1,1,401,1.0,1.0,1.0,1,582,1.0,1.0,1.0,1,1091,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q115,Create a Gmail draft,1,1,205,1.0,1.0,1.0,1,448,1.0,1.0,1.0,1,730,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q116,Share a Google Drive file,1,1,254,1.0,1.0,1.0,1,417,1.0,1.0,1.0,1,728,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q117,List Google Calendar events,1,1,219,1.0,1.0,1.0,1,379,1.0,1.0,1.0,1,753,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q118,Delete a calendar event,1,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,731,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q119,List Vercel projects,1,1,151,1.0,1.0,1.0,1,262,1.0,1.0,1.0,1,550,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q120,Select rows from Supabase,1,1,237,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,697,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q121,Post a Figma comment,1,1,204,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,688,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q122,List Okta users,1,1,182,1.0,1.0,1.0,1,299,1.0,1.0,1.0,1,606,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q123,Range query Prometheus,1,1,221,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,680,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q124,Create a Grafana annotation,1,1,218,1.0,1.0,1.0,1,296,1.0,1.0,1.0,1,659,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q125,Apply terraform,1,1,190,1.0,1.0,1.0,1,238,1.0,1.0,1.0,1,595,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q126,List Asana tasks,1,1,192,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,627,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q127,Comment on a Trello card,1,1,260,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,745,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q128,Update a ClickUp task,1,1,257,1.0,1.0,1.0,1,399,1.0,1.0,1.0,1,673,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q129,Search Confluence,1,1,221,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,695,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q130,Create a HubSpot deal,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,899,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q131,List Zendesk tickets,1,1,189,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,590,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q132,List Discord channels,1,1,166,1.0,1.0,1.0,1,300,1.0,1.0,1.0,1,555,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q133,Set a Redis key,1,1,182,1.0,1.0,1.0,1,302,1.0,1.0,1.0,1,530,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q134,Insert MongoDB documents,1,1,256,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,607,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q135,Run a MySQL query,2,1,182,1.0,1.0,1.0,1,281,1.0,1.0,1.0,1,545,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q136,Index a document in Elasticsearch,1,1,242,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,623,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q137,Get an S3 object,1,1,249,1.0,1.0,1.0,1,375,1.0,1.0,1.0,1,635,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q138,List Cloudflare zones,1,1,182,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,542,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q139,Create a Stripe invoice,1,1,204,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,647,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q140,List GitHub Actions workflows,1,1,268,1.0,1.0,1.0,1,453,1.0,1.0,1.0,1,728,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q141,Get GitHub Actions job logs,1,1,269,1.0,1.0,1.0,1,442,1.0,1.0,1.0,1,746,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q142,Create a GitHub repository,1,1,235,1.0,1.0,1.0,1,388,1.0,1.0,1.0,1,927,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q143,Checkout a git branch,1,1,212,1.0,1.0,1.0,1,373,1.0,1.0,1.0,1,835,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q144,Show git log,1,1,206,1.0,1.0,1.0,1,312,1.0,1.0,1.0,1,600,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q145,List postgres schemas,1,1,158,1.0,1.0,1.0,1,277,1.0,1.0,1.0,1,500,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q146,Open nodes in the knowledge graph,1,1,163,1.0,1.0,1.0,1,262,1.0,1.0,1.0,1,549,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q147,Get file metadata from the filesystem,1,1,192,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,770,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q148,Write a new local file,1,1,198,1.0,1.0,1.0,1,366,1.0,1.0,1.0,1,784,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q149,Search GitHub issues,1,1,263,1.0,1.0,1.0,1,397,1.0,1.0,1.0,1,670,1.0,1.0,1.0 +hybrid_codedb_weighted_bm25,q150,Get the authenticated GitHub user,3,1,178,1.0,1.0,1.0,1,295,1.0,1.0,1.0,1,592,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q001,Create a GitHub issue,1,1,380,1.0,1.0,1.0,1,584,1.0,1.0,1.0,1,1042,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q002,Create a Jira ticket,1,1,330,1.0,1.0,1.0,1,535,1.0,1.0,1.0,1,929,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q003,Send a Slack message,1,1,252,1.0,1.0,1.0,1,374,1.0,1.0,1.0,1,719,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q004,List open pull requests,1,1,237,1.0,1.0,1.0,1,477,1.0,1.0,1.0,1,976,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q005,Open a pull request on GitHub,1,1,406,1.0,1.0,1.0,1,564,1.0,1.0,1.0,1,1144,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q006,Run a SQL query against postgres,1,1,180,1.0,1.0,1.0,1,295,1.0,1.0,1.0,1,596,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q007,Read a local file from disk,1,1,223,1.0,1.0,1.0,1,406,1.0,1.0,1.0,1,747,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q008,What files are in this folder,2,1,191,1.0,1.0,1.0,1,314,1.0,1.0,1.0,1,648,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q009,Remember this fact in the knowledge graph,3,1,164,1.0,1.0,1.0,1,261,1.0,1.0,1.0,1,568,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q010,Search the web with Brave,1,1,240,1.0,1.0,1.0,1,351,1.0,1.0,1.0,1,637,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q011,Fetch the contents of a URL,1,1,208,1.0,1.0,1.0,1,372,1.0,1.0,1.0,1,700,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q012,Show git status,1,1,186,1.0,1.0,1.0,1,312,1.0,1.0,1.0,1,616,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q013,Commit staged changes,1,1,182,1.0,1.0,1.0,1,327,1.0,1.0,1.0,1,706,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q014,Create a Linear issue,2,1,275,1.0,1.0,1.0,1,463,1.0,1.0,1.0,1,925,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q015,Make a task in my project tracker Linear,4,2,515,0.0,1.0,1.0,1,440,1.0,1.0,1.0,1,801,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q016,Post a comment on a GitHub issue,1,1,302,1.0,1.0,1.0,1,542,1.0,1.0,1.0,1,1028,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q017,Merge the pull request,1,1,283,1.0,1.0,1.0,1,495,1.0,1.0,1.0,1,1093,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q018,Trigger a GitHub Actions workflow,1,1,294,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,734,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q019,Get Dependabot alerts,1,1,269,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,706,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q020,List Sentry issues,1,1,238,1.0,1.0,1.0,1,393,1.0,1.0,1.0,1,772,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q021,Create a Stripe customer,1,1,223,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q022,Refund a Stripe payment,1,1,195,1.0,1.0,1.0,1,332,1.0,1.0,1.0,1,632,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q023,Send an email with Gmail,1,1,224,1.0,1.0,1.0,1,399,1.0,1.0,1.0,1,743,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q024,Search my Gmail inbox,1,1,178,1.0,1.0,1.0,1,315,1.0,1.0,1.0,1,678,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q025,Upload a file to Google Drive,3,1,307,1.0,1.0,1.0,1,418,1.0,1.0,1.0,1,806,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q026,Create a calendar event,1,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,824,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q027,List Kubernetes pods,1,1,184,1.0,1.0,1.0,1,314,1.0,1.0,1.0,1,706,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q028,Get logs from a k8s pod,1,1,262,1.0,1.0,1.0,1,455,1.0,1.0,1.0,1,740,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q029,Run a docker container,1,1,223,1.0,1.0,1.0,1,331,1.0,1.0,1.0,1,587,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q030,Show docker container logs,1,1,226,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,683,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q031,Navigate the browser to a URL,1,1,180,1.0,1.0,1.0,1,309,1.0,1.0,1.0,1,624,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q032,Take a screenshot of the page,1,1,212,1.0,1.0,1.0,1,314,1.0,1.0,1.0,1,629,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q033,Create a GitLab merge request,2,1,297,1.0,1.0,1.0,1,475,1.0,1.0,1.0,1,929,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q034,Search Notion pages,1,1,180,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,667,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q035,Create a Notion page,1,1,230,1.0,1.0,1.0,1,371,1.0,1.0,1.0,1,781,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q036,Query a Notion database,1,1,261,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,656,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q037,Create a Zendesk ticket,1,1,257,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,782,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q038,Create a HubSpot contact,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,866,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q039,Create a Salesforce lead,2,1,252,1.0,1.0,1.0,1,380,1.0,1.0,1.0,1,798,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q040,Run a SOQL query,1,1,186,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,694,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q041,List Shopify orders,1,1,193,1.0,1.0,1.0,1,333,1.0,1.0,1.0,1,647,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q042,Create a PagerDuty incident,1,1,259,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,821,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q043,Who is on call in PagerDuty,1,1,219,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,725,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q044,Query Datadog metrics,1,1,300,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,785,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q045,Search Datadog logs,1,1,242,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,785,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q046,List S3 buckets,1,1,128,1.0,1.0,1.0,1,233,1.0,1.0,1.0,1,482,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q047,Upload an object to S3,1,1,250,1.0,1.0,1.0,1,420,1.0,1.0,1.0,1,835,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q048,Purge Cloudflare cache,1,1,183,1.0,1.0,1.0,1,341,1.0,1.0,1.0,1,618,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q049,Create a Cloudflare DNS record,1,1,249,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,819,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q050,List Vercel deployments,1,1,177,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,542,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q051,Insert a row into Supabase,1,1,230,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,701,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q052,Export images from Figma,1,1,239,1.0,1.0,1.0,1,328,1.0,1.0,1.0,1,685,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q053,Send an SMS with Twilio,1,1,249,1.0,1.0,1.0,1,409,1.0,1.0,1.0,1,806,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q054,Create an Okta user,1,1,183,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,676,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q055,Instant query Prometheus,1,1,221,1.0,1.0,1.0,1,379,1.0,1.0,1.0,1,661,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q056,Search Grafana dashboards,1,1,217,1.0,1.0,1.0,1,319,1.0,1.0,1.0,1,677,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q057,Trigger a CircleCI pipeline,1,1,200,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,755,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q058,Run terraform plan,1,1,172,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,758,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q059,Create an Asana task,1,1,249,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,820,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q060,Create a Trello card,1,1,240,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,734,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q061,Create a ClickUp task,1,1,219,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,734,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q062,Create a Confluence page,1,1,260,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,751,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q063,Send a Discord message,1,1,224,1.0,1.0,1.0,1,343,1.0,1.0,1.0,1,718,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q064,Search Elasticsearch,1,1,208,1.0,1.0,1.0,1,310,1.0,1.0,1.0,1,653,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q065,Get a Redis key,1,1,182,1.0,1.0,1.0,1,280,1.0,1.0,1.0,1,566,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q066,Find MongoDB documents,1,1,244,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,729,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q067,List SQLite tables,1,1,151,1.0,1.0,1.0,1,240,1.0,1.0,1.0,1,517,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q068,Describe a postgres table,1,1,188,1.0,1.0,1.0,1,292,1.0,1.0,1.0,1,590,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q069,github_create_issue,1,1,378,1.0,1.0,1.0,1,564,1.0,1.0,1.0,1,1048,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q070,createIssue,39,13,3322,0.0,0.0,0.0,8,3368,0.0,0.0,0.0,4,3296,0.0,0.0,0.0 +hybrid_rrf_bm25_embed,q071,list-directory,1,1,184,1.0,1.0,1.0,1,269,1.0,1.0,1.0,1,508,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q072,sequential thinking next thought,1,1,239,1.0,1.0,1.0,1,344,1.0,1.0,1.0,1,650,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q073,What time is it in Tokyo,1,1,239,1.0,1.0,1.0,1,391,1.0,1.0,1.0,1,696,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q074,Convert time between timezones,1,1,216,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q075,Star a GitHub repo,1,1,245,1.0,1.0,1.0,1,400,1.0,1.0,1.0,1,801,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q076,Fork a repository,1,1,239,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,875,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q077,Get the latest GitHub release,1,1,239,1.0,1.0,1.0,1,344,1.0,1.0,1.0,1,714,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q078,Create a gist,1,1,229,1.0,1.0,1.0,1,437,1.0,1.0,1.0,1,877,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q079,Search GitHub code,1,1,190,1.0,1.0,1.0,1,385,1.0,1.0,1.0,1,719,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q080,List GitHub notifications,1,1,174,1.0,1.0,1.0,1,321,1.0,1.0,1.0,1,704,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q081,Assign Copilot to an issue,1,1,267,1.0,1.0,1.0,1,435,1.0,1.0,1.0,1,822,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q082,Get secret scanning alerts,1,1,295,1.0,1.0,1.0,1,470,1.0,1.0,1.0,1,756,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q083,Create or update a file on GitHub,1,1,389,1.0,1.0,1.0,1,536,1.0,1.0,1.0,1,1012,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q084,List collaborators on a repo,1,1,269,1.0,1.0,1.0,1,410,1.0,1.0,1.0,1,804,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q085,Reply in a Slack thread,1,1,242,1.0,1.0,1.0,1,355,1.0,1.0,1.0,1,665,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q086,Add an emoji reaction on Slack,1,1,270,1.0,1.0,1.0,1,414,1.0,1.0,1.0,1,764,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q087,Search Slack messages,1,1,213,1.0,1.0,1.0,1,316,1.0,1.0,1.0,1,636,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q088,Create a Slack channel,1,1,196,1.0,1.0,1.0,1,339,1.0,1.0,1.0,1,751,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q089,Transition a Jira issue,1,1,222,1.0,1.0,1.0,1,342,1.0,1.0,1.0,1,706,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q090,Search Jira with JQL,1,1,179,1.0,1.0,1.0,1,286,1.0,1.0,1.0,1,549,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q091,Add a worklog to Jira,1,1,214,1.0,1.0,1.0,1,332,1.0,1.0,1.0,1,689,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q092,List Linear teams,1,1,184,1.0,1.0,1.0,1,339,1.0,1.0,1.0,1,673,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q093,Comment on a Linear issue,1,1,195,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,794,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q094,Move a local file,1,1,224,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,673,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q095,Recursively search files by pattern,1,1,192,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,623,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q096,Edit a file in place with old and new string,1,1,249,1.0,1.0,1.0,1,413,1.0,1.0,1.0,1,841,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q097,Read the whole knowledge graph,1,1,154,1.0,1.0,1.0,1,258,1.0,1.0,1.0,1,546,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q098,Search memory nodes,1,1,178,1.0,1.0,1.0,1,311,1.0,1.0,1.0,1,570,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q099,Show unstaged git diff,1,1,190,1.0,1.0,1.0,1,313,1.0,1.0,1.0,1,588,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q100,Stage files with git add,1,1,244,1.0,1.0,1.0,1,411,1.0,1.0,1.0,1,789,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q101,Create a Shopify product,1,1,240,1.0,1.0,1.0,1,365,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q102,Cancel a Stripe subscription,1,1,167,1.0,1.0,1.0,1,301,1.0,1.0,1.0,1,610,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q103,Create a payment intent,1,1,234,1.0,1.0,1.0,1,350,1.0,1.0,1.0,1,739,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q104,Acknowledge a PagerDuty incident,1,1,223,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q105,Create a Datadog monitor,1,1,259,1.0,1.0,1.0,1,459,1.0,1.0,1.0,1,760,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q106,Resolve a Sentry issue,2,1,274,1.0,1.0,1.0,1,412,1.0,1.0,1.0,1,777,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q107,Scale a Kubernetes deployment,1,1,241,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q108,Apply a Kubernetes manifest,1,1,195,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q109,Stop a docker container,1,1,188,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,664,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q110,Build a docker image,1,1,211,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,642,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q111,Click an element in Playwright,1,1,216,1.0,1.0,1.0,1,326,1.0,1.0,1.0,1,569,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q112,Type text into a browser input,1,1,235,1.0,1.0,1.0,1,336,1.0,1.0,1.0,1,586,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q113,List GitLab projects,1,1,216,1.0,1.0,1.0,1,288,1.0,1.0,1.0,1,640,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q114,Create a Bitbucket pull request,1,1,313,1.0,1.0,1.0,1,606,1.0,1.0,1.0,1,1139,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q115,Create a Gmail draft,1,1,188,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,789,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q116,Share a Google Drive file,1,1,306,1.0,1.0,1.0,1,417,1.0,1.0,1.0,1,705,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q117,List Google Calendar events,1,1,219,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,673,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q118,Delete a calendar event,1,1,267,1.0,1.0,1.0,1,444,1.0,1.0,1.0,1,690,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q119,List Vercel projects,1,1,151,1.0,1.0,1.0,1,260,1.0,1.0,1.0,1,539,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q120,Select rows from Supabase,1,1,237,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,736,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q121,Post a Figma comment,1,1,204,1.0,1.0,1.0,1,308,1.0,1.0,1.0,1,683,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q122,List Okta users,1,1,177,1.0,1.0,1.0,1,271,1.0,1.0,1.0,1,606,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q123,Range query Prometheus,1,1,221,1.0,1.0,1.0,1,379,1.0,1.0,1.0,1,680,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q124,Create a Grafana annotation,1,1,218,1.0,1.0,1.0,1,375,1.0,1.0,1.0,1,788,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q125,Apply terraform,1,1,190,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,653,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q126,List Asana tasks,1,1,244,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,635,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q127,Comment on a Trello card,1,1,260,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,721,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q128,Update a ClickUp task,1,1,219,1.0,1.0,1.0,1,399,1.0,1.0,1.0,1,732,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q129,Search Confluence,1,1,221,1.0,1.0,1.0,1,371,1.0,1.0,1.0,1,636,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q130,Create a HubSpot deal,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,772,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q131,List Zendesk tickets,1,1,189,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,565,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q132,List Discord channels,1,1,166,1.0,1.0,1.0,1,300,1.0,1.0,1.0,1,562,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q133,Set a Redis key,1,1,182,1.0,1.0,1.0,1,280,1.0,1.0,1.0,1,642,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q134,Insert MongoDB documents,1,1,256,1.0,1.0,1.0,1,399,1.0,1.0,1.0,1,759,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q135,Run a MySQL query,1,1,179,1.0,1.0,1.0,1,294,1.0,1.0,1.0,1,662,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q136,Index a document in Elasticsearch,1,1,242,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,637,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q137,Get an S3 object,1,1,249,1.0,1.0,1.0,1,366,1.0,1.0,1.0,1,640,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q138,List Cloudflare zones,1,1,182,1.0,1.0,1.0,1,326,1.0,1.0,1.0,1,558,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q139,Create a Stripe invoice,1,1,185,1.0,1.0,1.0,1,340,1.0,1.0,1.0,1,662,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q140,List GitHub Actions workflows,1,1,293,1.0,1.0,1.0,1,438,1.0,1.0,1.0,1,845,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q141,Get GitHub Actions job logs,1,1,269,1.0,1.0,1.0,1,450,1.0,1.0,1.0,1,804,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q142,Create a GitHub repository,1,1,333,1.0,1.0,1.0,1,556,1.0,1.0,1.0,1,915,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q143,Checkout a git branch,1,1,212,1.0,1.0,1.0,1,373,1.0,1.0,1.0,1,759,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q144,Show git log,1,1,194,1.0,1.0,1.0,1,312,1.0,1.0,1.0,1,600,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q145,List postgres schemas,1,1,169,1.0,1.0,1.0,1,248,1.0,1.0,1.0,1,491,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q146,Open nodes in the knowledge graph,1,1,163,1.0,1.0,1.0,1,262,1.0,1.0,1.0,1,531,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q147,Get file metadata from the filesystem,1,1,190,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,713,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q148,Write a new local file,1,1,269,1.0,1.0,1.0,1,384,1.0,1.0,1.0,1,775,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q149,Search GitHub issues,1,1,208,1.0,1.0,1.0,1,363,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +hybrid_rrf_bm25_embed,q150,Get the authenticated GitHub user,2,1,179,1.0,1.0,1.0,1,301,1.0,1.0,1.0,1,656,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q001,Create a GitHub issue,1,1,380,1.0,1.0,1.0,1,557,1.0,1.0,1.0,1,1038,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q002,Create a Jira ticket,1,1,318,1.0,1.0,1.0,1,513,1.0,1.0,1.0,1,1003,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q003,Send a Slack message,1,1,247,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,673,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q004,List open pull requests,1,1,237,1.0,1.0,1.0,1,460,1.0,1.0,1.0,1,887,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q005,Open a pull request on GitHub,1,1,364,1.0,1.0,1.0,1,522,1.0,1.0,1.0,1,1144,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q006,Run a SQL query against postgres,1,1,180,1.0,1.0,1.0,1,295,1.0,1.0,1.0,1,590,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q007,Read a local file from disk,1,1,223,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,806,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q008,What files are in this folder,9,3,679,0.0,0.0,1.0,2,712,0.0,1.0,1.0,1,690,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q009,Remember this fact in the knowledge graph,2,1,154,1.0,1.0,1.0,1,255,1.0,1.0,1.0,1,507,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q010,Search the web with Brave,1,1,258,1.0,1.0,1.0,1,410,1.0,1.0,1.0,1,688,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q011,Fetch the contents of a URL,1,1,222,1.0,1.0,1.0,1,335,1.0,1.0,1.0,1,660,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q012,Show git status,1,1,188,1.0,1.0,1.0,1,315,1.0,1.0,1.0,1,667,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q013,Commit staged changes,1,1,182,1.0,1.0,1.0,1,305,1.0,1.0,1.0,1,725,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q014,Create a Linear issue,2,1,333,1.0,1.0,1.0,1,463,1.0,1.0,1.0,1,921,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q015,Make a task in my project tracker Linear,146,49,12289,0.0,0.0,0.0,30,12029,0.0,0.0,0.0,15,11669,0.0,0.0,0.0 +hybrid_rrf_bm25_trigram,q016,Post a comment on a GitHub issue,3,1,307,1.0,1.0,1.0,1,442,1.0,1.0,1.0,1,912,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q017,Merge the pull request,1,1,299,1.0,1.0,1.0,1,468,1.0,1.0,1.0,1,957,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q018,Trigger a GitHub Actions workflow,1,1,294,1.0,1.0,1.0,1,455,1.0,1.0,1.0,1,797,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q019,Get Dependabot alerts,1,1,269,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,691,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q020,List Sentry issues,1,1,238,1.0,1.0,1.0,1,380,1.0,1.0,1.0,1,796,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q021,Create a Stripe customer,1,1,186,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q022,Refund a Stripe payment,1,1,234,1.0,1.0,1.0,1,332,1.0,1.0,1.0,1,647,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q023,Send an email with Gmail,1,1,255,1.0,1.0,1.0,1,365,1.0,1.0,1.0,1,713,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q024,Search my Gmail inbox,1,1,174,1.0,1.0,1.0,1,294,1.0,1.0,1.0,1,615,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q025,Upload a file to Google Drive,5,2,523,0.0,1.0,1.0,1,418,1.0,1.0,1.0,1,736,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q026,Create a calendar event,1,1,300,1.0,1.0,1.0,1,446,1.0,1.0,1.0,1,840,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q027,List Kubernetes pods,1,1,184,1.0,1.0,1.0,1,314,1.0,1.0,1.0,1,699,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q028,Get logs from a k8s pod,1,1,262,1.0,1.0,1.0,1,441,1.0,1.0,1.0,1,830,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q029,Run a docker container,1,1,223,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,647,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q030,Show docker container logs,1,1,188,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,649,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q031,Navigate the browser to a URL,1,1,180,1.0,1.0,1.0,1,282,1.0,1.0,1.0,1,536,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q032,Take a screenshot of the page,1,1,212,1.0,1.0,1.0,1,314,1.0,1.0,1.0,1,612,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q033,Create a GitLab merge request,2,1,297,1.0,1.0,1.0,1,475,1.0,1.0,1.0,1,1054,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q034,Search Notion pages,1,1,211,1.0,1.0,1.0,1,319,1.0,1.0,1.0,1,614,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q035,Create a Notion page,1,1,230,1.0,1.0,1.0,1,371,1.0,1.0,1.0,1,832,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q036,Query a Notion database,1,1,261,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,688,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q037,Create a Zendesk ticket,1,1,257,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,780,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q038,Create a HubSpot contact,1,1,216,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,709,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q039,Create a Salesforce lead,2,1,252,1.0,1.0,1.0,1,402,1.0,1.0,1.0,1,710,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q040,Run a SOQL query,1,1,186,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,592,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q041,List Shopify orders,1,1,208,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,638,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q042,Create a PagerDuty incident,1,1,259,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,721,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q043,Who is on call in PagerDuty,1,1,219,1.0,1.0,1.0,1,363,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q044,Query Datadog metrics,1,1,300,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,799,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q045,Search Datadog logs,1,1,242,1.0,1.0,1.0,1,395,1.0,1.0,1.0,1,742,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q046,List S3 buckets,1,1,129,1.0,1.0,1.0,1,229,1.0,1.0,1.0,1,476,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q047,Upload an object to S3,1,1,250,1.0,1.0,1.0,1,426,1.0,1.0,1.0,1,850,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q048,Purge Cloudflare cache,1,1,183,1.0,1.0,1.0,1,335,1.0,1.0,1.0,1,548,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q049,Create a Cloudflare DNS record,1,1,249,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,703,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q050,List Vercel deployments,1,1,177,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,497,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q051,Insert a row into Supabase,1,1,203,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,672,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q052,Export images from Figma,1,1,218,1.0,1.0,1.0,1,328,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q053,Send an SMS with Twilio,1,1,254,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,740,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q054,Create an Okta user,1,1,183,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,753,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q055,Instant query Prometheus,1,1,221,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,666,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q056,Search Grafana dashboards,1,1,182,1.0,1.0,1.0,1,288,1.0,1.0,1.0,1,670,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q057,Trigger a CircleCI pipeline,1,1,216,1.0,1.0,1.0,1,325,1.0,1.0,1.0,1,755,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q058,Run terraform plan,1,1,172,1.0,1.0,1.0,1,287,1.0,1.0,1.0,1,593,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q059,Create an Asana task,1,1,267,1.0,1.0,1.0,1,429,1.0,1.0,1.0,1,841,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q060,Create a Trello card,1,1,240,1.0,1.0,1.0,1,374,1.0,1.0,1.0,1,699,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q061,Create a ClickUp task,1,1,257,1.0,1.0,1.0,1,415,1.0,1.0,1.0,1,724,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q062,Create a Confluence page,1,1,260,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,713,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q063,Send a Discord message,1,1,224,1.0,1.0,1.0,1,327,1.0,1.0,1.0,1,681,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q064,Search Elasticsearch,1,1,239,1.0,1.0,1.0,1,343,1.0,1.0,1.0,1,633,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q065,Get a Redis key,1,1,182,1.0,1.0,1.0,1,281,1.0,1.0,1.0,1,567,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q066,Find MongoDB documents,1,1,258,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,742,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q067,List SQLite tables,1,1,151,1.0,1.0,1.0,1,243,1.0,1.0,1.0,1,480,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q068,Describe a postgres table,1,1,188,1.0,1.0,1.0,1,274,1.0,1.0,1.0,1,640,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q069,github_create_issue,1,1,333,1.0,1.0,1.0,1,568,1.0,1.0,1.0,1,1120,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q070,createIssue,3,1,332,1.0,1.0,1.0,1,579,1.0,1.0,1.0,1,963,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q071,list-directory,1,1,186,1.0,1.0,1.0,1,348,1.0,1.0,1.0,1,580,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q072,sequential thinking next thought,1,1,278,1.0,1.0,1.0,1,434,1.0,1.0,1.0,1,862,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q073,What time is it in Tokyo,1,1,217,1.0,1.0,1.0,1,391,1.0,1.0,1.0,1,709,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q074,Convert time between timezones,1,1,204,1.0,1.0,1.0,1,319,1.0,1.0,1.0,1,687,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q075,Star a GitHub repo,1,1,226,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,759,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q076,Fork a repository,1,1,235,1.0,1.0,1.0,1,378,1.0,1.0,1.0,1,750,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q077,Get the latest GitHub release,1,1,239,1.0,1.0,1.0,1,383,1.0,1.0,1.0,1,724,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q078,Create a gist,1,1,234,1.0,1.0,1.0,1,340,1.0,1.0,1.0,1,737,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q079,Search GitHub code,1,1,191,1.0,1.0,1.0,1,341,1.0,1.0,1.0,1,725,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q080,List GitHub notifications,1,1,190,1.0,1.0,1.0,1,311,1.0,1.0,1.0,1,662,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q081,Assign Copilot to an issue,1,1,267,1.0,1.0,1.0,1,404,1.0,1.0,1.0,1,864,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q082,Get secret scanning alerts,1,1,295,1.0,1.0,1.0,1,470,1.0,1.0,1.0,1,733,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q083,Create or update a file on GitHub,1,1,369,1.0,1.0,1.0,1,585,1.0,1.0,1.0,1,1047,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q084,List collaborators on a repo,1,1,278,1.0,1.0,1.0,1,443,1.0,1.0,1.0,1,827,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q085,Reply in a Slack thread,1,1,242,1.0,1.0,1.0,1,355,1.0,1.0,1.0,1,627,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q086,Add an emoji reaction on Slack,1,1,294,1.0,1.0,1.0,1,412,1.0,1.0,1.0,1,686,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q087,Search Slack messages,1,1,232,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,654,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q088,Create a Slack channel,1,1,196,1.0,1.0,1.0,1,339,1.0,1.0,1.0,1,758,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q089,Transition a Jira issue,1,1,208,1.0,1.0,1.0,1,342,1.0,1.0,1.0,1,777,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q090,Search Jira with JQL,1,1,195,1.0,1.0,1.0,1,302,1.0,1.0,1.0,1,635,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q091,Add a worklog to Jira,1,1,247,1.0,1.0,1.0,1,404,1.0,1.0,1.0,1,756,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q092,List Linear teams,1,1,184,1.0,1.0,1.0,1,330,1.0,1.0,1.0,1,690,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q093,Comment on a Linear issue,1,1,190,1.0,1.0,1.0,1,360,1.0,1.0,1.0,1,767,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q094,Move a local file,3,1,224,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,675,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q095,Recursively search files by pattern,1,1,200,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,655,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q096,Edit a file in place with old and new string,1,1,230,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,729,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q097,Read the whole knowledge graph,1,1,152,1.0,1.0,1.0,1,253,1.0,1.0,1.0,1,561,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q098,Search memory nodes,1,1,176,1.0,1.0,1.0,1,299,1.0,1.0,1.0,1,618,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q099,Show unstaged git diff,1,1,190,1.0,1.0,1.0,1,313,1.0,1.0,1.0,1,613,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q100,Stage files with git add,1,1,220,1.0,1.0,1.0,1,387,1.0,1.0,1.0,1,762,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q101,Create a Shopify product,1,1,240,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,690,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q102,Cancel a Stripe subscription,1,1,183,1.0,1.0,1.0,1,301,1.0,1.0,1.0,1,663,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q103,Create a payment intent,1,1,234,1.0,1.0,1.0,1,339,1.0,1.0,1.0,1,737,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q104,Acknowledge a PagerDuty incident,1,1,223,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,665,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q105,Create a Datadog monitor,1,1,259,1.0,1.0,1.0,1,374,1.0,1.0,1.0,1,711,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q106,Resolve a Sentry issue,2,1,251,1.0,1.0,1.0,1,412,1.0,1.0,1.0,1,729,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q107,Scale a Kubernetes deployment,1,1,234,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q108,Apply a Kubernetes manifest,1,1,217,1.0,1.0,1.0,1,330,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q109,Stop a docker container,1,1,188,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,655,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q110,Build a docker image,1,1,211,1.0,1.0,1.0,1,317,1.0,1.0,1.0,1,598,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q111,Click an element in Playwright,1,1,216,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,564,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q112,Type text into a browser input,1,1,217,1.0,1.0,1.0,1,333,1.0,1.0,1.0,1,609,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q113,List GitLab projects,1,1,209,1.0,1.0,1.0,1,289,1.0,1.0,1.0,1,597,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q114,Create a Bitbucket pull request,1,1,313,1.0,1.0,1.0,1,546,1.0,1.0,1.0,1,1085,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q115,Create a Gmail draft,1,1,225,1.0,1.0,1.0,1,340,1.0,1.0,1.0,1,739,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q116,Share a Google Drive file,1,1,254,1.0,1.0,1.0,1,418,1.0,1.0,1.0,1,724,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q117,List Google Calendar events,1,1,219,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,697,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q118,Delete a calendar event,1,1,300,1.0,1.0,1.0,1,440,1.0,1.0,1.0,1,685,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q119,List Vercel projects,1,1,151,1.0,1.0,1.0,1,248,1.0,1.0,1.0,1,550,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q120,Select rows from Supabase,1,1,237,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,664,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q121,Post a Figma comment,1,1,201,1.0,1.0,1.0,1,308,1.0,1.0,1.0,1,649,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q122,List Okta users,1,1,177,1.0,1.0,1.0,1,288,1.0,1.0,1.0,1,618,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q123,Range query Prometheus,1,1,221,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,660,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q124,Create a Grafana annotation,1,1,252,1.0,1.0,1.0,1,391,1.0,1.0,1.0,1,782,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q125,Apply terraform,1,1,190,1.0,1.0,1.0,1,327,1.0,1.0,1.0,1,644,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q126,List Asana tasks,1,1,192,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,612,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q127,Comment on a Trello card,1,1,260,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,703,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q128,Update a ClickUp task,1,1,257,1.0,1.0,1.0,1,399,1.0,1.0,1.0,1,736,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q129,Search Confluence,1,1,221,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,682,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q130,Create a HubSpot deal,1,1,255,1.0,1.0,1.0,1,413,1.0,1.0,1.0,1,696,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q131,List Zendesk tickets,1,1,189,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,575,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q132,List Discord channels,1,1,166,1.0,1.0,1.0,1,300,1.0,1.0,1.0,1,577,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q133,Set a Redis key,1,1,182,1.0,1.0,1.0,1,281,1.0,1.0,1.0,1,600,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q134,Insert MongoDB documents,1,1,244,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,751,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q135,Run a MySQL query,1,1,179,1.0,1.0,1.0,1,294,1.0,1.0,1.0,1,585,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q136,Index a document in Elasticsearch,1,1,242,1.0,1.0,1.0,1,344,1.0,1.0,1.0,1,597,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q137,Get an S3 object,1,1,249,1.0,1.0,1.0,1,375,1.0,1.0,1.0,1,678,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q138,List Cloudflare zones,1,1,182,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,554,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q139,Create a Stripe invoice,1,1,204,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,680,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q140,List GitHub Actions workflows,1,1,293,1.0,1.0,1.0,1,438,1.0,1.0,1.0,1,783,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q141,Get GitHub Actions job logs,1,1,269,1.0,1.0,1.0,1,442,1.0,1.0,1.0,1,802,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q142,Create a GitHub repository,1,1,268,1.0,1.0,1.0,1,406,1.0,1.0,1.0,1,853,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q143,Checkout a git branch,1,1,212,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,926,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q144,Show git log,1,1,196,1.0,1.0,1.0,1,299,1.0,1.0,1.0,1,603,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q145,List postgres schemas,1,1,160,1.0,1.0,1.0,1,277,1.0,1.0,1.0,1,488,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q146,Open nodes in the knowledge graph,1,1,177,1.0,1.0,1.0,1,262,1.0,1.0,1.0,1,492,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q147,Get file metadata from the filesystem,1,1,220,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,709,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q148,Write a new local file,1,1,234,1.0,1.0,1.0,1,384,1.0,1.0,1.0,1,699,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q149,Search GitHub issues,1,1,228,1.0,1.0,1.0,1,335,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +hybrid_rrf_bm25_trigram,q150,Get the authenticated GitHub user,2,1,179,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,694,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q001,Create a GitHub issue,1,1,380,1.0,1.0,1.0,1,538,1.0,1.0,1.0,1,1014,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q002,Create a Jira ticket,1,1,330,1.0,1.0,1.0,1,535,1.0,1.0,1.0,1,981,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q003,Send a Slack message,1,1,253,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,673,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q004,List open pull requests,2,1,237,1.0,1.0,1.0,1,477,1.0,1.0,1.0,1,976,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q005,Open a pull request on GitHub,1,1,314,1.0,1.0,1.0,1,509,1.0,1.0,1.0,1,1144,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q006,Run a SQL query against postgres,1,1,180,1.0,1.0,1.0,1,295,1.0,1.0,1.0,1,560,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q007,Read a local file from disk,1,1,223,1.0,1.0,1.0,1,384,1.0,1.0,1.0,1,714,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q008,What files are in this folder,5,2,387,0.0,1.0,1.0,1,290,1.0,1.0,1.0,1,587,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q009,Remember this fact in the knowledge graph,5,2,334,0.0,1.0,1.0,1,263,1.0,1.0,1.0,1,601,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q010,Search the web with Brave,1,1,239,1.0,1.0,1.0,1,341,1.0,1.0,1.0,1,637,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q011,Fetch the contents of a URL,1,1,205,1.0,1.0,1.0,1,369,1.0,1.0,1.0,1,668,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q012,Show git status,1,1,206,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,675,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q013,Commit staged changes,1,1,182,1.0,1.0,1.0,1,295,1.0,1.0,1.0,1,695,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q014,Create a Linear issue,1,1,275,1.0,1.0,1.0,1,463,1.0,1.0,1.0,1,875,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q015,Make a task in my project tracker Linear,3,1,284,1.0,1.0,1.0,1,402,1.0,1.0,1.0,1,764,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q016,Post a comment on a GitHub issue,2,1,327,1.0,1.0,1.0,1,494,1.0,1.0,1.0,1,993,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q017,Merge the pull request,1,1,299,1.0,1.0,1.0,1,431,1.0,1.0,1.0,1,962,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q018,Trigger a GitHub Actions workflow,1,1,294,1.0,1.0,1.0,1,468,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q019,Get Dependabot alerts,1,1,286,1.0,1.0,1.0,1,463,1.0,1.0,1.0,1,779,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q020,List Sentry issues,1,1,238,1.0,1.0,1.0,1,396,1.0,1.0,1.0,1,806,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q021,Create a Stripe customer,2,1,223,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q022,Refund a Stripe payment,1,1,197,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,647,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q023,Send an email with Gmail,1,1,239,1.0,1.0,1.0,1,328,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q024,Search my Gmail inbox,1,1,193,1.0,1.0,1.0,1,300,1.0,1.0,1.0,1,607,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q025,Upload a file to Google Drive,4,2,496,0.0,1.0,1.0,1,418,1.0,1.0,1.0,1,765,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q026,Create a calendar event,1,1,300,1.0,1.0,1.0,1,480,1.0,1.0,1.0,1,882,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q027,List Kubernetes pods,1,1,180,1.0,1.0,1.0,1,314,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q028,Get logs from a k8s pod,1,1,262,1.0,1.0,1.0,1,429,1.0,1.0,1.0,1,754,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q029,Run a docker container,1,1,223,1.0,1.0,1.0,1,331,1.0,1.0,1.0,1,647,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q030,Show docker container logs,1,1,226,1.0,1.0,1.0,1,365,1.0,1.0,1.0,1,649,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q031,Navigate the browser to a URL,1,1,180,1.0,1.0,1.0,1,294,1.0,1.0,1.0,1,581,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q032,Take a screenshot of the page,1,1,212,1.0,1.0,1.0,1,314,1.0,1.0,1.0,1,623,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q033,Create a GitLab merge request,1,1,297,1.0,1.0,1.0,1,475,1.0,1.0,1.0,1,931,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q034,Search Notion pages,1,1,211,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,692,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q035,Create a Notion page,1,1,230,1.0,1.0,1.0,1,413,1.0,1.0,1.0,1,832,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q036,Query a Notion database,1,1,261,1.0,1.0,1.0,1,404,1.0,1.0,1.0,1,740,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q037,Create a Zendesk ticket,2,1,257,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,871,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q038,Create a HubSpot contact,1,1,216,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,826,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q039,Create a Salesforce lead,1,1,252,1.0,1.0,1.0,1,380,1.0,1.0,1.0,1,830,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q040,Run a SOQL query,1,1,186,1.0,1.0,1.0,1,294,1.0,1.0,1.0,1,601,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q041,List Shopify orders,1,1,193,1.0,1.0,1.0,1,333,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q042,Create a PagerDuty incident,1,1,255,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,831,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q043,Who is on call in PagerDuty,1,1,219,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,728,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q044,Query Datadog metrics,1,1,300,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,783,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q045,Search Datadog logs,1,1,242,1.0,1.0,1.0,1,363,1.0,1.0,1.0,1,751,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q046,List S3 buckets,1,1,186,1.0,1.0,1.0,1,327,1.0,1.0,1.0,1,656,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q047,Upload an object to S3,1,1,250,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,808,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q048,Purge Cloudflare cache,1,1,183,1.0,1.0,1.0,1,358,1.0,1.0,1.0,1,608,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q049,Create a Cloudflare DNS record,1,1,249,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,844,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q050,List Vercel deployments,1,1,177,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,558,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q051,Insert a row into Supabase,1,1,238,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,701,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q052,Export images from Figma,1,1,239,1.0,1.0,1.0,1,330,1.0,1.0,1.0,1,724,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q053,Send an SMS with Twilio,1,1,249,1.0,1.0,1.0,1,409,1.0,1.0,1.0,1,890,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q054,Create an Okta user,1,1,183,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,806,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q055,Instant query Prometheus,1,1,221,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,667,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q056,Search Grafana dashboards,1,1,185,1.0,1.0,1.0,1,287,1.0,1.0,1.0,1,680,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q057,Trigger a CircleCI pipeline,1,1,200,1.0,1.0,1.0,1,359,1.0,1.0,1.0,1,737,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q058,Run terraform plan,1,1,172,1.0,1.0,1.0,1,295,1.0,1.0,1.0,1,672,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q059,Create an Asana task,1,1,249,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,846,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q060,Create a Trello card,1,1,240,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,764,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q061,Create a ClickUp task,1,1,219,1.0,1.0,1.0,1,415,1.0,1.0,1.0,1,839,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q062,Create a Confluence page,1,1,260,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,922,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q063,Send a Discord message,1,1,224,1.0,1.0,1.0,1,350,1.0,1.0,1.0,1,665,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q064,Search Elasticsearch,1,1,208,1.0,1.0,1.0,1,315,1.0,1.0,1.0,1,633,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q065,Get a Redis key,1,1,182,1.0,1.0,1.0,1,283,1.0,1.0,1.0,1,548,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q066,Find MongoDB documents,1,1,258,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q067,List SQLite tables,1,1,151,1.0,1.0,1.0,1,240,1.0,1.0,1.0,1,537,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q068,Describe a postgres table,1,1,188,1.0,1.0,1.0,1,274,1.0,1.0,1.0,1,679,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q069,github_create_issue,1,1,320,1.0,1.0,1.0,1,536,1.0,1.0,1.0,1,998,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q070,createIssue,39,13,3380,0.0,0.0,0.0,8,3426,0.0,0.0,0.0,4,3354,0.0,0.0,0.0 +hybrid_rrf_trigram_embed,q071,list-directory,1,1,171,1.0,1.0,1.0,1,256,1.0,1.0,1.0,1,508,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q072,sequential thinking next thought,1,1,255,1.0,1.0,1.0,1,337,1.0,1.0,1.0,1,676,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q073,What time is it in Tokyo,1,1,217,1.0,1.0,1.0,1,391,1.0,1.0,1.0,1,780,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q074,Convert time between timezones,1,1,234,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,776,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q075,Star a GitHub repo,1,1,214,1.0,1.0,1.0,1,399,1.0,1.0,1.0,1,809,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q076,Fork a repository,1,1,239,1.0,1.0,1.0,1,378,1.0,1.0,1.0,1,783,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q077,Get the latest GitHub release,1,1,239,1.0,1.0,1.0,1,363,1.0,1.0,1.0,1,624,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q078,Create a gist,1,1,280,1.0,1.0,1.0,1,473,1.0,1.0,1.0,1,963,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q079,Search GitHub code,1,1,190,1.0,1.0,1.0,1,296,1.0,1.0,1.0,1,719,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q080,List GitHub notifications,1,1,176,1.0,1.0,1.0,1,279,1.0,1.0,1.0,1,673,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q081,Assign Copilot to an issue,1,1,267,1.0,1.0,1.0,1,466,1.0,1.0,1.0,1,838,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q082,Get secret scanning alerts,1,1,294,1.0,1.0,1.0,1,483,1.0,1.0,1.0,1,775,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q083,Create or update a file on GitHub,1,1,333,1.0,1.0,1.0,1,476,1.0,1.0,1.0,1,876,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q084,List collaborators on a repo,1,1,238,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,792,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q085,Reply in a Slack thread,2,1,242,1.0,1.0,1.0,1,365,1.0,1.0,1.0,1,688,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q086,Add an emoji reaction on Slack,1,1,302,1.0,1.0,1.0,1,454,1.0,1.0,1.0,1,890,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q087,Search Slack messages,1,1,251,1.0,1.0,1.0,1,379,1.0,1.0,1.0,1,661,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q088,Create a Slack channel,1,1,196,1.0,1.0,1.0,1,339,1.0,1.0,1.0,1,795,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q089,Transition a Jira issue,1,1,208,1.0,1.0,1.0,1,342,1.0,1.0,1.0,1,721,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q090,Search Jira with JQL,1,1,199,1.0,1.0,1.0,1,300,1.0,1.0,1.0,1,566,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q091,Add a worklog to Jira,1,1,233,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,660,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q092,List Linear teams,1,1,184,1.0,1.0,1.0,1,339,1.0,1.0,1.0,1,745,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q093,Comment on a Linear issue,1,1,195,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,781,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q094,Move a local file,2,1,225,1.0,1.0,1.0,1,349,1.0,1.0,1.0,1,708,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q095,Recursively search files by pattern,1,1,200,1.0,1.0,1.0,1,304,1.0,1.0,1.0,1,641,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q096,Edit a file in place with old and new string,1,1,255,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,812,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q097,Read the whole knowledge graph,1,1,154,1.0,1.0,1.0,1,259,1.0,1.0,1.0,1,513,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q098,Search memory nodes,1,1,195,1.0,1.0,1.0,1,302,1.0,1.0,1.0,1,624,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q099,Show unstaged git diff,1,1,190,1.0,1.0,1.0,1,299,1.0,1.0,1.0,1,615,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q100,Stage files with git add,1,1,198,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,773,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q101,Create a Shopify product,1,1,240,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,700,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q102,Cancel a Stripe subscription,1,1,183,1.0,1.0,1.0,1,301,1.0,1.0,1.0,1,663,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q103,Create a payment intent,2,1,234,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,866,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q104,Acknowledge a PagerDuty incident,1,1,223,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,690,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q105,Create a Datadog monitor,1,1,259,1.0,1.0,1.0,1,441,1.0,1.0,1.0,1,933,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q106,Resolve a Sentry issue,2,1,274,1.0,1.0,1.0,1,412,1.0,1.0,1.0,1,729,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q107,Scale a Kubernetes deployment,1,1,241,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q108,Apply a Kubernetes manifest,1,1,222,1.0,1.0,1.0,1,330,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q109,Stop a docker container,1,1,188,1.0,1.0,1.0,1,327,1.0,1.0,1.0,1,639,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q110,Build a docker image,1,1,211,1.0,1.0,1.0,1,319,1.0,1.0,1.0,1,635,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q111,Click an element in Playwright,1,1,216,1.0,1.0,1.0,1,318,1.0,1.0,1.0,1,566,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q112,Type text into a browser input,1,1,217,1.0,1.0,1.0,1,318,1.0,1.0,1.0,1,584,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q113,List GitLab projects,1,1,216,1.0,1.0,1.0,1,288,1.0,1.0,1.0,1,681,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q114,Create a Bitbucket pull request,1,1,313,1.0,1.0,1.0,1,606,1.0,1.0,1.0,1,1037,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q115,Create a Gmail draft,1,1,254,1.0,1.0,1.0,1,431,1.0,1.0,1.0,1,830,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q116,Share a Google Drive file,1,1,306,1.0,1.0,1.0,1,418,1.0,1.0,1.0,1,785,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q117,List Google Calendar events,1,1,219,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,716,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q118,Delete a calendar event,1,1,270,1.0,1.0,1.0,1,409,1.0,1.0,1.0,1,706,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q119,List Vercel projects,1,1,175,1.0,1.0,1.0,1,255,1.0,1.0,1.0,1,519,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q120,Select rows from Supabase,1,1,237,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,672,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q121,Post a Figma comment,1,1,201,1.0,1.0,1.0,1,308,1.0,1.0,1.0,1,653,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q122,List Okta users,1,1,163,1.0,1.0,1.0,1,271,1.0,1.0,1.0,1,622,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q123,Range query Prometheus,1,1,221,1.0,1.0,1.0,1,393,1.0,1.0,1.0,1,761,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q124,Create a Grafana annotation,1,1,259,1.0,1.0,1.0,1,461,1.0,1.0,1.0,1,960,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q125,Apply terraform,1,1,171,1.0,1.0,1.0,1,287,1.0,1.0,1.0,1,648,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q126,List Asana tasks,1,1,244,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,674,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q127,Comment on a Trello card,1,1,260,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,682,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q128,Update a ClickUp task,1,1,219,1.0,1.0,1.0,1,399,1.0,1.0,1.0,1,747,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q129,Search Confluence,1,1,221,1.0,1.0,1.0,1,369,1.0,1.0,1.0,1,651,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q130,Create a HubSpot deal,1,1,255,1.0,1.0,1.0,1,355,1.0,1.0,1.0,1,776,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q131,List Zendesk tickets,1,1,189,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,644,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q132,List Discord channels,1,1,184,1.0,1.0,1.0,1,300,1.0,1.0,1.0,1,637,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q133,Set a Redis key,2,1,182,1.0,1.0,1.0,1,284,1.0,1.0,1.0,1,611,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q134,Insert MongoDB documents,1,1,258,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,733,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q135,Run a MySQL query,1,1,179,1.0,1.0,1.0,1,294,1.0,1.0,1.0,1,591,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q136,Index a document in Elasticsearch,1,1,242,1.0,1.0,1.0,1,392,1.0,1.0,1.0,1,703,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q137,Get an S3 object,1,1,249,1.0,1.0,1.0,1,397,1.0,1.0,1.0,1,670,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q138,List Cloudflare zones,1,1,182,1.0,1.0,1.0,1,340,1.0,1.0,1.0,1,591,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q139,Create a Stripe invoice,1,1,185,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,694,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q140,List GitHub Actions workflows,1,1,293,1.0,1.0,1.0,1,452,1.0,1.0,1.0,1,803,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q141,Get GitHub Actions job logs,1,1,269,1.0,1.0,1.0,1,442,1.0,1.0,1.0,1,704,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q142,Create a GitHub repository,1,1,333,1.0,1.0,1.0,1,545,1.0,1.0,1.0,1,915,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q143,Checkout a git branch,1,1,182,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,710,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q144,Show git log,1,1,194,1.0,1.0,1.0,1,312,1.0,1.0,1.0,1,651,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q145,List postgres schemas,1,1,152,1.0,1.0,1.0,1,249,1.0,1.0,1.0,1,469,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q146,Open nodes in the knowledge graph,1,1,177,1.0,1.0,1.0,1,262,1.0,1.0,1.0,1,551,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q147,Get file metadata from the filesystem,1,1,190,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,685,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q148,Write a new local file,1,1,234,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,764,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q149,Search GitHub issues,1,1,209,1.0,1.0,1.0,1,314,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +hybrid_rrf_trigram_embed,q150,Get the authenticated GitHub user,2,1,179,1.0,1.0,1.0,1,317,1.0,1.0,1.0,1,605,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q001,Create a GitHub issue,1,1,394,1.0,1.0,1.0,1,598,1.0,1.0,1.0,1,1037,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q002,Create a Jira ticket,1,1,345,1.0,1.0,1.0,1,474,1.0,1.0,1.0,1,884,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q003,Send a Slack message,1,1,252,1.0,1.0,1.0,1,424,1.0,1.0,1.0,1,729,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q004,List open pull requests,1,1,311,1.0,1.0,1.0,1,477,1.0,1.0,1.0,1,1013,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q005,Open a pull request on GitHub,1,1,368,1.0,1.0,1.0,1,564,1.0,1.0,1.0,1,1144,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q006,Run a SQL query against postgres,1,1,183,1.0,1.0,1.0,1,272,1.0,1.0,1.0,1,596,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q007,Read a local file from disk,1,1,223,1.0,1.0,1.0,1,370,1.0,1.0,1.0,1,684,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q008,What files are in this folder,2,1,191,1.0,1.0,1.0,1,316,1.0,1.0,1.0,1,615,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q009,Remember this fact in the knowledge graph,3,1,164,1.0,1.0,1.0,1,261,1.0,1.0,1.0,1,585,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q010,Search the web with Brave,1,1,240,1.0,1.0,1.0,1,349,1.0,1.0,1.0,1,673,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q011,Fetch the contents of a URL,1,1,258,1.0,1.0,1.0,1,372,1.0,1.0,1.0,1,675,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q012,Show git status,1,1,196,1.0,1.0,1.0,1,312,1.0,1.0,1.0,1,600,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q013,Commit staged changes,1,1,182,1.0,1.0,1.0,1,327,1.0,1.0,1.0,1,706,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q014,Create a Linear issue,1,1,322,1.0,1.0,1.0,1,463,1.0,1.0,1.0,1,875,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q015,Make a task in my project tracker Linear,2,1,284,1.0,1.0,1.0,1,483,1.0,1.0,1.0,1,769,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q016,Post a comment on a GitHub issue,2,1,370,1.0,1.0,1.0,1,573,1.0,1.0,1.0,1,1087,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q017,Merge the pull request,1,1,283,1.0,1.0,1.0,1,528,1.0,1.0,1.0,1,1093,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q018,Trigger a GitHub Actions workflow,1,1,294,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,805,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q019,Get Dependabot alerts,1,1,287,1.0,1.0,1.0,1,422,1.0,1.0,1.0,1,708,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q020,List Sentry issues,1,1,238,1.0,1.0,1.0,1,396,1.0,1.0,1.0,1,812,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q021,Create a Stripe customer,2,1,186,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q022,Refund a Stripe payment,1,1,234,1.0,1.0,1.0,1,332,1.0,1.0,1.0,1,644,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q023,Send an email with Gmail,1,1,224,1.0,1.0,1.0,1,399,1.0,1.0,1.0,1,743,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q024,Search my Gmail inbox,1,1,178,1.0,1.0,1.0,1,315,1.0,1.0,1.0,1,661,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q025,Upload a file to Google Drive,3,1,307,1.0,1.0,1.0,1,418,1.0,1.0,1.0,1,810,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q026,Create a calendar event,1,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,945,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q027,List Kubernetes pods,1,1,184,1.0,1.0,1.0,1,314,1.0,1.0,1.0,1,706,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q028,Get logs from a k8s pod,1,1,262,1.0,1.0,1.0,1,455,1.0,1.0,1.0,1,741,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q029,Run a docker container,1,1,223,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,644,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q030,Show docker container logs,1,1,221,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,649,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q031,Navigate the browser to a URL,1,1,180,1.0,1.0,1.0,1,279,1.0,1.0,1.0,1,543,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q032,Take a screenshot of the page,1,1,212,1.0,1.0,1.0,1,316,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q033,Create a GitLab merge request,2,1,297,1.0,1.0,1.0,1,475,1.0,1.0,1.0,1,951,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q034,Search Notion pages,1,1,227,1.0,1.0,1.0,1,349,1.0,1.0,1.0,1,671,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q035,Create a Notion page,2,1,230,1.0,1.0,1.0,1,385,1.0,1.0,1.0,1,737,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q036,Query a Notion database,1,1,261,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q037,Create a Zendesk ticket,1,1,257,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,891,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q038,Create a HubSpot contact,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,925,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q039,Create a Salesforce lead,2,1,252,1.0,1.0,1.0,1,380,1.0,1.0,1.0,1,815,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q040,Run a SOQL query,1,1,187,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,694,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q041,List Shopify orders,1,1,208,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,647,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q042,Create a PagerDuty incident,1,1,259,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,874,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q043,Who is on call in PagerDuty,1,1,227,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,709,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q044,Query Datadog metrics,1,1,300,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,769,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q045,Search Datadog logs,1,1,242,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,802,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q046,List S3 buckets,1,1,185,1.0,1.0,1.0,1,318,1.0,1.0,1.0,1,690,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q047,Upload an object to S3,1,1,250,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,847,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q048,Purge Cloudflare cache,1,1,231,1.0,1.0,1.0,1,341,1.0,1.0,1.0,1,618,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q049,Create a Cloudflare DNS record,1,1,250,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,812,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q050,List Vercel deployments,1,1,177,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,650,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q051,Insert a row into Supabase,1,1,238,1.0,1.0,1.0,1,355,1.0,1.0,1.0,1,701,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q052,Export images from Figma,1,1,239,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,701,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q053,Send an SMS with Twilio,1,1,249,1.0,1.0,1.0,1,409,1.0,1.0,1.0,1,811,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q054,Create an Okta user,2,1,183,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,676,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q055,Instant query Prometheus,1,1,221,1.0,1.0,1.0,1,379,1.0,1.0,1.0,1,682,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q056,Search Grafana dashboards,1,1,217,1.0,1.0,1.0,1,319,1.0,1.0,1.0,1,677,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q057,Trigger a CircleCI pipeline,1,1,200,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,755,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q058,Run terraform plan,1,1,172,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,758,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q059,Create an Asana task,1,1,249,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,837,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q060,Create a Trello card,1,1,240,1.0,1.0,1.0,1,411,1.0,1.0,1.0,1,819,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q061,Create a ClickUp task,1,1,219,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,787,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q062,Create a Confluence page,1,1,260,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,791,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q063,Send a Discord message,1,1,224,1.0,1.0,1.0,1,343,1.0,1.0,1.0,1,718,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q064,Search Elasticsearch,1,1,208,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,680,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q065,Get a Redis key,1,1,182,1.0,1.0,1.0,1,280,1.0,1.0,1.0,1,566,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q066,Find MongoDB documents,1,1,219,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,729,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q067,List SQLite tables,1,1,151,1.0,1.0,1.0,1,240,1.0,1.0,1.0,1,590,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q068,Describe a postgres table,1,1,188,1.0,1.0,1.0,1,274,1.0,1.0,1.0,1,590,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q069,github_create_issue,1,1,378,1.0,1.0,1.0,1,573,1.0,1.0,1.0,1,1072,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q070,createIssue,39,13,3608,0.0,0.0,0.0,8,3654,0.0,0.0,0.0,4,3582,0.0,0.0,0.0 +hybrid_rrf_weighted_embed,q071,list-directory,1,1,171,1.0,1.0,1.0,1,301,1.0,1.0,1.0,1,623,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q072,sequential thinking next thought,1,1,239,1.0,1.0,1.0,1,344,1.0,1.0,1.0,1,650,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q073,What time is it in Tokyo,1,1,239,1.0,1.0,1.0,1,391,1.0,1.0,1.0,1,757,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q074,Convert time between timezones,1,1,216,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q075,Star a GitHub repo,1,1,242,1.0,1.0,1.0,1,425,1.0,1.0,1.0,1,801,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q076,Fork a repository,1,1,239,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,875,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q077,Get the latest GitHub release,1,1,239,1.0,1.0,1.0,1,344,1.0,1.0,1.0,1,728,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q078,Create a gist,1,1,229,1.0,1.0,1.0,1,429,1.0,1.0,1.0,1,925,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q079,Search GitHub code,1,1,283,1.0,1.0,1.0,1,385,1.0,1.0,1.0,1,751,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q080,List GitHub notifications,1,1,221,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,704,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q081,Assign Copilot to an issue,1,1,267,1.0,1.0,1.0,1,500,1.0,1.0,1.0,1,862,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q082,Get secret scanning alerts,1,1,295,1.0,1.0,1.0,1,470,1.0,1.0,1.0,1,776,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q083,Create or update a file on GitHub,1,1,333,1.0,1.0,1.0,1,476,1.0,1.0,1.0,1,1078,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q084,List collaborators on a repo,1,1,271,1.0,1.0,1.0,1,410,1.0,1.0,1.0,1,804,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q085,Reply in a Slack thread,1,1,257,1.0,1.0,1.0,1,416,1.0,1.0,1.0,1,748,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q086,Add an emoji reaction on Slack,1,1,271,1.0,1.0,1.0,1,436,1.0,1.0,1.0,1,785,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q087,Search Slack messages,1,1,251,1.0,1.0,1.0,1,393,1.0,1.0,1.0,1,748,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q088,Create a Slack channel,2,1,196,1.0,1.0,1.0,1,339,1.0,1.0,1.0,1,751,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q089,Transition a Jira issue,1,1,222,1.0,1.0,1.0,1,342,1.0,1.0,1.0,1,760,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q090,Search Jira with JQL,1,1,179,1.0,1.0,1.0,1,325,1.0,1.0,1.0,1,627,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q091,Add a worklog to Jira,1,1,214,1.0,1.0,1.0,1,439,1.0,1.0,1.0,1,821,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q092,List Linear teams,1,1,184,1.0,1.0,1.0,1,398,1.0,1.0,1.0,1,728,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q093,Comment on a Linear issue,1,1,218,1.0,1.0,1.0,1,464,1.0,1.0,1.0,1,781,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q094,Move a local file,1,1,224,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,715,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q095,Recursively search files by pattern,1,1,192,1.0,1.0,1.0,1,305,1.0,1.0,1.0,1,617,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q096,Edit a file in place with old and new string,1,1,255,1.0,1.0,1.0,1,413,1.0,1.0,1.0,1,779,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q097,Read the whole knowledge graph,1,1,154,1.0,1.0,1.0,1,258,1.0,1.0,1.0,1,510,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q098,Search memory nodes,1,1,202,1.0,1.0,1.0,1,288,1.0,1.0,1.0,1,564,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q099,Show unstaged git diff,1,1,190,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,588,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q100,Stage files with git add,1,1,211,1.0,1.0,1.0,1,326,1.0,1.0,1.0,1,651,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q101,Create a Shopify product,1,1,240,1.0,1.0,1.0,1,380,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q102,Cancel a Stripe subscription,1,1,167,1.0,1.0,1.0,1,297,1.0,1.0,1.0,1,622,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q103,Create a payment intent,1,1,234,1.0,1.0,1.0,1,441,1.0,1.0,1.0,1,800,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q104,Acknowledge a PagerDuty incident,1,1,261,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q105,Create a Datadog monitor,2,1,259,1.0,1.0,1.0,1,459,1.0,1.0,1.0,1,840,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q106,Resolve a Sentry issue,2,1,274,1.0,1.0,1.0,1,397,1.0,1.0,1.0,1,832,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q107,Scale a Kubernetes deployment,1,1,241,1.0,1.0,1.0,1,371,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q108,Apply a Kubernetes manifest,1,1,221,1.0,1.0,1.0,1,371,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q109,Stop a docker container,1,1,188,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,664,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q110,Build a docker image,1,1,211,1.0,1.0,1.0,1,319,1.0,1.0,1.0,1,642,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q111,Click an element in Playwright,1,1,216,1.0,1.0,1.0,1,341,1.0,1.0,1.0,1,566,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q112,Type text into a browser input,1,1,213,1.0,1.0,1.0,1,330,1.0,1.0,1.0,1,586,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q113,List GitLab projects,1,1,216,1.0,1.0,1.0,1,300,1.0,1.0,1.0,1,635,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q114,Create a Bitbucket pull request,1,1,313,1.0,1.0,1.0,1,553,1.0,1.0,1.0,1,1002,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q115,Create a Gmail draft,1,1,188,1.0,1.0,1.0,1,375,1.0,1.0,1.0,1,855,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q116,Share a Google Drive file,1,1,306,1.0,1.0,1.0,1,417,1.0,1.0,1.0,1,741,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q117,List Google Calendar events,1,1,219,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,769,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q118,Delete a calendar event,1,1,300,1.0,1.0,1.0,1,444,1.0,1.0,1.0,1,697,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q119,List Vercel projects,1,1,175,1.0,1.0,1.0,1,269,1.0,1.0,1.0,1,539,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q120,Select rows from Supabase,1,1,237,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,752,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q121,Post a Figma comment,1,1,204,1.0,1.0,1.0,1,350,1.0,1.0,1.0,1,729,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q122,List Okta users,1,1,177,1.0,1.0,1.0,1,271,1.0,1.0,1.0,1,606,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q123,Range query Prometheus,1,1,221,1.0,1.0,1.0,1,379,1.0,1.0,1.0,1,661,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q124,Create a Grafana annotation,1,1,218,1.0,1.0,1.0,1,421,1.0,1.0,1.0,1,915,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q125,Apply terraform,1,1,171,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,653,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q126,List Asana tasks,1,1,244,1.0,1.0,1.0,1,404,1.0,1.0,1.0,1,707,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q127,Comment on a Trello card,1,1,260,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,721,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q128,Update a ClickUp task,1,1,219,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q129,Search Confluence,1,1,221,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,694,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q130,Create a HubSpot deal,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,927,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q131,List Zendesk tickets,1,1,189,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,753,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q132,List Discord channels,1,1,166,1.0,1.0,1.0,1,300,1.0,1.0,1.0,1,705,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q133,Set a Redis key,1,1,182,1.0,1.0,1.0,1,280,1.0,1.0,1.0,1,621,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q134,Insert MongoDB documents,1,1,256,1.0,1.0,1.0,1,399,1.0,1.0,1.0,1,759,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q135,Run a MySQL query,2,1,182,1.0,1.0,1.0,1,294,1.0,1.0,1.0,1,678,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q136,Index a document in Elasticsearch,1,1,242,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,717,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q137,Get an S3 object,1,1,249,1.0,1.0,1.0,1,373,1.0,1.0,1.0,1,632,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q138,List Cloudflare zones,1,1,182,1.0,1.0,1.0,1,326,1.0,1.0,1.0,1,671,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q139,Create a Stripe invoice,1,1,204,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q140,List GitHub Actions workflows,1,1,293,1.0,1.0,1.0,1,452,1.0,1.0,1.0,1,821,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q141,Get GitHub Actions job logs,1,1,269,1.0,1.0,1.0,1,450,1.0,1.0,1.0,1,813,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q142,Create a GitHub repository,1,1,268,1.0,1.0,1.0,1,428,1.0,1.0,1.0,1,927,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q143,Checkout a git branch,1,1,189,1.0,1.0,1.0,1,325,1.0,1.0,1.0,1,718,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q144,Show git log,1,1,194,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,600,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q145,List postgres schemas,1,1,169,1.0,1.0,1.0,1,287,1.0,1.0,1.0,1,600,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q146,Open nodes in the knowledge graph,1,1,163,1.0,1.0,1.0,1,262,1.0,1.0,1.0,1,549,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q147,Get file metadata from the filesystem,2,1,192,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,649,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q148,Write a new local file,1,1,269,1.0,1.0,1.0,1,423,1.0,1.0,1.0,1,759,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q149,Search GitHub issues,1,1,258,1.0,1.0,1.0,1,363,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +hybrid_rrf_weighted_embed,q150,Get the authenticated GitHub user,2,1,179,1.0,1.0,1.0,1,285,1.0,1.0,1.0,1,700,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q001,Create a GitHub issue,1,1,388,1.0,1.0,1.0,1,598,1.0,1.0,1.0,1,1024,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q002,Create a Jira ticket,1,1,312,1.0,1.0,1.0,1,398,1.0,1.0,1.0,1,854,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q003,Send a Slack message,1,1,249,1.0,1.0,1.0,1,373,1.0,1.0,1.0,1,737,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q004,List open pull requests,1,1,311,1.0,1.0,1.0,1,506,1.0,1.0,1.0,1,1060,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q005,Open a pull request on GitHub,1,1,379,1.0,1.0,1.0,1,522,1.0,1.0,1.0,1,1144,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q006,Run a SQL query against postgres,1,1,183,1.0,1.0,1.0,1,272,1.0,1.0,1.0,1,590,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q007,Read a local file from disk,1,1,223,1.0,1.0,1.0,1,385,1.0,1.0,1.0,1,745,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q008,What files are in this folder,6,2,478,0.0,1.0,1.0,2,719,0.0,1.0,1.0,1,697,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q009,Remember this fact in the knowledge graph,2,1,155,1.0,1.0,1.0,1,260,1.0,1.0,1.0,1,510,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q010,Search the web with Brave,1,1,258,1.0,1.0,1.0,1,365,1.0,1.0,1.0,1,699,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q011,Fetch the contents of a URL,1,1,208,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,681,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q012,Show git status,1,1,196,1.0,1.0,1.0,1,293,1.0,1.0,1.0,1,584,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q013,Commit staged changes,1,1,215,1.0,1.0,1.0,1,327,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q014,Create a Linear issue,1,1,333,1.0,1.0,1.0,1,463,1.0,1.0,1.0,1,875,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q015,Make a task in my project tracker Linear,3,1,284,1.0,1.0,1.0,1,415,1.0,1.0,1.0,1,758,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q016,Post a comment on a GitHub issue,2,1,333,1.0,1.0,1.0,1,542,1.0,1.0,1.0,1,1063,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q017,Merge the pull request,1,1,283,1.0,1.0,1.0,1,523,1.0,1.0,1.0,1,1031,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q018,Trigger a GitHub Actions workflow,1,1,294,1.0,1.0,1.0,1,470,1.0,1.0,1.0,1,835,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q019,Get Dependabot alerts,1,1,287,1.0,1.0,1.0,1,418,1.0,1.0,1.0,1,730,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q020,List Sentry issues,1,1,238,1.0,1.0,1.0,1,396,1.0,1.0,1.0,1,813,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q021,Create a Stripe customer,2,1,186,1.0,1.0,1.0,1,315,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q022,Refund a Stripe payment,1,1,234,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,662,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q023,Send an email with Gmail,1,1,239,1.0,1.0,1.0,1,328,1.0,1.0,1.0,1,735,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q024,Search my Gmail inbox,1,1,265,1.0,1.0,1.0,1,375,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q025,Upload a file to Google Drive,2,1,307,1.0,1.0,1.0,1,418,1.0,1.0,1.0,1,794,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q026,Create a calendar event,1,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,885,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q027,List Kubernetes pods,1,1,184,1.0,1.0,1.0,1,314,1.0,1.0,1.0,1,718,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q028,Get logs from a k8s pod,1,1,275,1.0,1.0,1.0,1,447,1.0,1.0,1.0,1,911,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q029,Run a docker container,2,1,223,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,640,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q030,Show docker container logs,1,1,188,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,649,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q031,Navigate the browser to a URL,1,1,180,1.0,1.0,1.0,1,265,1.0,1.0,1.0,1,550,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q032,Take a screenshot of the page,1,1,242,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,671,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q033,Create a GitLab merge request,2,1,297,1.0,1.0,1.0,1,475,1.0,1.0,1.0,1,961,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q034,Search Notion pages,1,1,211,1.0,1.0,1.0,1,336,1.0,1.0,1.0,1,704,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q035,Create a Notion page,2,1,230,1.0,1.0,1.0,1,371,1.0,1.0,1.0,1,796,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q036,Query a Notion database,1,1,261,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,674,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q037,Create a Zendesk ticket,1,1,257,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,856,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q038,Create a HubSpot contact,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,877,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q039,Create a Salesforce lead,1,1,252,1.0,1.0,1.0,1,402,1.0,1.0,1.0,1,782,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q040,Run a SOQL query,1,1,187,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,672,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q041,List Shopify orders,1,1,208,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,695,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q042,Create a PagerDuty incident,1,1,259,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,836,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q043,Who is on call in PagerDuty,1,1,219,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,771,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q044,Query Datadog metrics,1,1,300,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,797,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q045,Search Datadog logs,1,1,294,1.0,1.0,1.0,1,423,1.0,1.0,1.0,1,784,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q046,List S3 buckets,1,1,206,1.0,1.0,1.0,1,363,1.0,1.0,1.0,1,699,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q047,Upload an object to S3,1,1,250,1.0,1.0,1.0,1,409,1.0,1.0,1.0,1,866,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q048,Purge Cloudflare cache,1,1,231,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,732,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q049,Create a Cloudflare DNS record,1,1,217,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,819,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q050,List Vercel deployments,1,1,177,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,615,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q051,Insert a row into Supabase,1,1,238,1.0,1.0,1.0,1,355,1.0,1.0,1.0,1,767,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q052,Export images from Figma,1,1,218,1.0,1.0,1.0,1,328,1.0,1.0,1.0,1,660,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q053,Send an SMS with Twilio,1,1,249,1.0,1.0,1.0,1,398,1.0,1.0,1.0,1,748,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q054,Create an Okta user,2,1,183,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,827,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q055,Instant query Prometheus,1,1,221,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,652,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q056,Search Grafana dashboards,1,1,217,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,705,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q057,Trigger a CircleCI pipeline,1,1,200,1.0,1.0,1.0,1,325,1.0,1.0,1.0,1,755,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q058,Run terraform plan,1,1,172,1.0,1.0,1.0,1,287,1.0,1.0,1.0,1,618,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q059,Create an Asana task,2,1,267,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,853,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q060,Create a Trello card,1,1,240,1.0,1.0,1.0,1,394,1.0,1.0,1.0,1,874,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q061,Create a ClickUp task,2,1,257,1.0,1.0,1.0,1,410,1.0,1.0,1.0,1,871,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q062,Create a Confluence page,1,1,260,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,876,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q063,Send a Discord message,1,1,224,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q064,Search Elasticsearch,1,1,239,1.0,1.0,1.0,1,404,1.0,1.0,1.0,1,686,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q065,Get a Redis key,1,1,182,1.0,1.0,1.0,1,312,1.0,1.0,1.0,1,631,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q066,Find MongoDB documents,1,1,258,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,788,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q067,List SQLite tables,1,1,151,1.0,1.0,1.0,1,252,1.0,1.0,1.0,1,567,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q068,Describe a postgres table,1,1,186,1.0,1.0,1.0,1,272,1.0,1.0,1.0,1,643,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q069,github_create_issue,1,1,361,1.0,1.0,1.0,1,588,1.0,1.0,1.0,1,1092,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q070,createIssue,3,1,332,1.0,1.0,1.0,1,520,1.0,1.0,1.0,1,988,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q071,list-directory,2,1,186,1.0,1.0,1.0,1,281,1.0,1.0,1.0,1,646,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q072,sequential thinking next thought,1,1,302,1.0,1.0,1.0,1,430,1.0,1.0,1.0,1,746,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q073,What time is it in Tokyo,1,1,217,1.0,1.0,1.0,1,380,1.0,1.0,1.0,1,674,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q074,Convert time between timezones,1,1,213,1.0,1.0,1.0,1,369,1.0,1.0,1.0,1,743,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q075,Star a GitHub repo,1,1,226,1.0,1.0,1.0,1,393,1.0,1.0,1.0,1,792,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q076,Fork a repository,1,1,235,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,792,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q077,Get the latest GitHub release,1,1,268,1.0,1.0,1.0,1,412,1.0,1.0,1.0,1,709,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q078,Create a gist,1,1,272,1.0,1.0,1.0,1,480,1.0,1.0,1.0,1,925,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q079,Search GitHub code,1,1,283,1.0,1.0,1.0,1,385,1.0,1.0,1.0,1,748,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q080,List GitHub notifications,1,1,190,1.0,1.0,1.0,1,311,1.0,1.0,1.0,1,687,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q081,Assign Copilot to an issue,1,1,259,1.0,1.0,1.0,1,464,1.0,1.0,1.0,1,838,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q082,Get secret scanning alerts,1,1,295,1.0,1.0,1.0,1,446,1.0,1.0,1.0,1,828,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q083,Create or update a file on GitHub,1,1,316,1.0,1.0,1.0,1,478,1.0,1.0,1.0,1,1038,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q084,List collaborators on a repo,1,1,278,1.0,1.0,1.0,1,420,1.0,1.0,1.0,1,893,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q085,Reply in a Slack thread,1,1,239,1.0,1.0,1.0,1,363,1.0,1.0,1.0,1,728,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q086,Add an emoji reaction on Slack,1,1,273,1.0,1.0,1.0,1,413,1.0,1.0,1.0,1,835,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q087,Search Slack messages,1,1,231,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,696,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q088,Create a Slack channel,2,1,196,1.0,1.0,1.0,1,339,1.0,1.0,1.0,1,777,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q089,Transition a Jira issue,1,1,208,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,763,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q090,Search Jira with JQL,1,1,237,1.0,1.0,1.0,1,394,1.0,1.0,1.0,1,806,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q091,Add a worklog to Jira,1,1,214,1.0,1.0,1.0,1,391,1.0,1.0,1.0,1,791,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q092,List Linear teams,1,1,184,1.0,1.0,1.0,1,398,1.0,1.0,1.0,1,722,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q093,Comment on a Linear issue,1,1,258,1.0,1.0,1.0,1,441,1.0,1.0,1.0,1,835,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q094,Move a local file,1,1,208,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,756,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q095,Recursively search files by pattern,1,1,219,1.0,1.0,1.0,1,335,1.0,1.0,1.0,1,690,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q096,Edit a file in place with old and new string,1,1,247,1.0,1.0,1.0,1,422,1.0,1.0,1.0,1,814,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q097,Read the whole knowledge graph,1,1,153,1.0,1.0,1.0,1,254,1.0,1.0,1.0,1,513,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q098,Search memory nodes,1,1,215,1.0,1.0,1.0,1,357,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q099,Show unstaged git diff,1,1,190,1.0,1.0,1.0,1,299,1.0,1.0,1.0,1,585,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q100,Stage files with git add,2,1,260,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,706,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q101,Create a Shopify product,1,1,240,1.0,1.0,1.0,1,380,1.0,1.0,1.0,1,700,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q102,Cancel a Stripe subscription,1,1,186,1.0,1.0,1.0,1,316,1.0,1.0,1.0,1,632,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q103,Create a payment intent,1,1,234,1.0,1.0,1.0,1,424,1.0,1.0,1.0,1,897,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q104,Acknowledge a PagerDuty incident,1,1,261,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,765,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q105,Create a Datadog monitor,2,1,259,1.0,1.0,1.0,1,459,1.0,1.0,1.0,1,928,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q106,Resolve a Sentry issue,2,1,251,1.0,1.0,1.0,1,397,1.0,1.0,1.0,1,837,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q107,Scale a Kubernetes deployment,1,1,252,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q108,Apply a Kubernetes manifest,1,1,253,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q109,Stop a docker container,1,1,188,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,645,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q110,Build a docker image,1,1,211,1.0,1.0,1.0,1,317,1.0,1.0,1.0,1,614,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q111,Click an element in Playwright,1,1,216,1.0,1.0,1.0,1,339,1.0,1.0,1.0,1,706,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q112,Type text into a browser input,1,1,217,1.0,1.0,1.0,1,316,1.0,1.0,1.0,1,624,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q113,List GitLab projects,1,1,209,1.0,1.0,1.0,1,316,1.0,1.0,1.0,1,676,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q114,Create a Bitbucket pull request,1,1,313,1.0,1.0,1.0,1,516,1.0,1.0,1.0,1,1097,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q115,Create a Gmail draft,1,1,228,1.0,1.0,1.0,1,375,1.0,1.0,1.0,1,836,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q116,Share a Google Drive file,1,1,254,1.0,1.0,1.0,1,418,1.0,1.0,1.0,1,802,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q117,List Google Calendar events,1,1,219,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,714,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q118,Delete a calendar event,1,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,739,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q119,List Vercel projects,1,1,175,1.0,1.0,1.0,1,269,1.0,1.0,1.0,1,567,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q120,Select rows from Supabase,1,1,237,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,688,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q121,Post a Figma comment,1,1,204,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,683,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q122,List Okta users,1,1,177,1.0,1.0,1.0,1,298,1.0,1.0,1.0,1,659,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q123,Range query Prometheus,1,1,221,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,668,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q124,Create a Grafana annotation,1,1,237,1.0,1.0,1.0,1,391,1.0,1.0,1.0,1,884,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q125,Apply terraform,1,1,171,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,673,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q126,List Asana tasks,1,1,244,1.0,1.0,1.0,1,404,1.0,1.0,1.0,1,707,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q127,Comment on a Trello card,1,1,260,1.0,1.0,1.0,1,422,1.0,1.0,1.0,1,759,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q128,Update a ClickUp task,1,1,257,1.0,1.0,1.0,1,399,1.0,1.0,1.0,1,739,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q129,Search Confluence,1,1,221,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q130,Create a HubSpot deal,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,862,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q131,List Zendesk tickets,1,1,189,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,697,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q132,List Discord channels,1,1,166,1.0,1.0,1.0,1,321,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q133,Set a Redis key,1,1,182,1.0,1.0,1.0,1,300,1.0,1.0,1.0,1,651,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q134,Insert MongoDB documents,1,1,256,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,766,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q135,Run a MySQL query,2,1,182,1.0,1.0,1.0,1,281,1.0,1.0,1.0,1,650,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q136,Index a document in Elasticsearch,1,1,242,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,810,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q137,Get an S3 object,1,1,249,1.0,1.0,1.0,1,375,1.0,1.0,1.0,1,705,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q138,List Cloudflare zones,1,1,182,1.0,1.0,1.0,1,341,1.0,1.0,1.0,1,719,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q139,Create a Stripe invoice,1,1,204,1.0,1.0,1.0,1,333,1.0,1.0,1.0,1,654,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q140,List GitHub Actions workflows,1,1,293,1.0,1.0,1.0,1,457,1.0,1.0,1.0,1,790,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q141,Get GitHub Actions job logs,1,1,269,1.0,1.0,1.0,1,471,1.0,1.0,1.0,1,884,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q142,Create a GitHub repository,1,1,250,1.0,1.0,1.0,1,388,1.0,1.0,1.0,1,949,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q143,Checkout a git branch,1,1,212,1.0,1.0,1.0,1,376,1.0,1.0,1.0,1,801,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q144,Show git log,1,1,206,1.0,1.0,1.0,1,304,1.0,1.0,1.0,1,619,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q145,List postgres schemas,1,1,158,1.0,1.0,1.0,1,277,1.0,1.0,1.0,1,595,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q146,Open nodes in the knowledge graph,1,1,163,1.0,1.0,1.0,1,267,1.0,1.0,1.0,1,504,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q147,Get file metadata from the filesystem,4,2,426,0.0,1.0,1.0,1,348,1.0,1.0,1.0,1,686,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q148,Write a new local file,1,1,199,1.0,1.0,1.0,1,342,1.0,1.0,1.0,1,737,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q149,Search GitHub issues,1,1,260,1.0,1.0,1.0,1,363,1.0,1.0,1.0,1,683,1.0,1.0,1.0 +hybrid_rrf_weighted_hashed,q150,Get the authenticated GitHub user,2,1,179,1.0,1.0,1.0,1,301,1.0,1.0,1.0,1,693,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q001,Create a GitHub issue,1,1,388,1.0,1.0,1.0,1,584,1.0,1.0,1.0,1,1038,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q002,Create a Jira ticket,1,1,333,1.0,1.0,1.0,1,528,1.0,1.0,1.0,1,1003,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q003,Send a Slack message,1,1,249,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,693,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q004,List open pull requests,1,1,234,1.0,1.0,1.0,1,474,1.0,1.0,1.0,1,993,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q005,Open a pull request on GitHub,1,1,368,1.0,1.0,1.0,1,522,1.0,1.0,1.0,1,1144,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q006,Run a SQL query against postgres,1,1,183,1.0,1.0,1.0,1,272,1.0,1.0,1.0,1,590,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q007,Read a local file from disk,1,1,223,1.0,1.0,1.0,1,428,1.0,1.0,1.0,1,781,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q008,What files are in this folder,9,3,664,0.0,0.0,1.0,2,703,0.0,1.0,1.0,1,681,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q009,Remember this fact in the knowledge graph,2,1,154,1.0,1.0,1.0,1,255,1.0,1.0,1.0,1,506,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q010,Search the web with Brave,1,1,258,1.0,1.0,1.0,1,410,1.0,1.0,1.0,1,705,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q011,Fetch the contents of a URL,1,1,222,1.0,1.0,1.0,1,380,1.0,1.0,1.0,1,722,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q012,Show git status,1,1,183,1.0,1.0,1.0,1,301,1.0,1.0,1.0,1,662,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q013,Commit staged changes,1,1,182,1.0,1.0,1.0,1,305,1.0,1.0,1.0,1,725,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q014,Create a Linear issue,1,1,340,1.0,1.0,1.0,1,463,1.0,1.0,1.0,1,925,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q015,Make a task in my project tracker Linear,3,1,284,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,638,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q016,Post a comment on a GitHub issue,2,1,333,1.0,1.0,1.0,1,510,1.0,1.0,1.0,1,1006,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q017,Merge the pull request,1,1,299,1.0,1.0,1.0,1,468,1.0,1.0,1.0,1,987,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q018,Trigger a GitHub Actions workflow,1,1,294,1.0,1.0,1.0,1,470,1.0,1.0,1.0,1,842,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q019,Get Dependabot alerts,1,1,269,1.0,1.0,1.0,1,418,1.0,1.0,1.0,1,675,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q020,List Sentry issues,1,1,238,1.0,1.0,1.0,1,396,1.0,1.0,1.0,1,796,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q021,Create a Stripe customer,2,1,186,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q022,Refund a Stripe payment,1,1,234,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,662,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q023,Send an email with Gmail,1,1,239,1.0,1.0,1.0,1,349,1.0,1.0,1.0,1,713,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q024,Search my Gmail inbox,1,1,190,1.0,1.0,1.0,1,294,1.0,1.0,1.0,1,617,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q025,Upload a file to Google Drive,5,2,496,0.0,1.0,1.0,1,418,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q026,Create a calendar event,2,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,909,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q027,List Kubernetes pods,1,1,184,1.0,1.0,1.0,1,314,1.0,1.0,1.0,1,738,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q028,Get logs from a k8s pod,1,1,262,1.0,1.0,1.0,1,441,1.0,1.0,1.0,1,848,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q029,Run a docker container,1,1,219,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,640,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q030,Show docker container logs,1,1,188,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,649,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q031,Navigate the browser to a URL,1,1,180,1.0,1.0,1.0,1,280,1.0,1.0,1.0,1,574,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q032,Take a screenshot of the page,1,1,212,1.0,1.0,1.0,1,372,1.0,1.0,1.0,1,675,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q033,Create a GitLab merge request,2,1,297,1.0,1.0,1.0,1,475,1.0,1.0,1.0,1,1054,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q034,Search Notion pages,1,1,211,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,635,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q035,Create a Notion page,2,1,230,1.0,1.0,1.0,1,371,1.0,1.0,1.0,1,832,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q036,Query a Notion database,1,1,261,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,688,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q037,Create a Zendesk ticket,1,1,257,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,882,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q038,Create a HubSpot contact,1,1,216,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,867,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q039,Create a Salesforce lead,2,1,252,1.0,1.0,1.0,1,402,1.0,1.0,1.0,1,787,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q040,Run a SOQL query,1,1,187,1.0,1.0,1.0,1,294,1.0,1.0,1.0,1,631,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q041,List Shopify orders,1,1,208,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,703,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q042,Create a PagerDuty incident,1,1,259,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,848,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q043,Who is on call in PagerDuty,1,1,227,1.0,1.0,1.0,1,363,1.0,1.0,1.0,1,764,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q044,Query Datadog metrics,1,1,300,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,799,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q045,Search Datadog logs,1,1,242,1.0,1.0,1.0,1,412,1.0,1.0,1.0,1,749,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q046,List S3 buckets,1,1,213,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,704,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q047,Upload an object to S3,1,1,250,1.0,1.0,1.0,1,426,1.0,1.0,1.0,1,850,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q048,Purge Cloudflare cache,1,1,231,1.0,1.0,1.0,1,335,1.0,1.0,1.0,1,548,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q049,Create a Cloudflare DNS record,1,1,217,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,810,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q050,List Vercel deployments,1,1,177,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,636,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q051,Insert a row into Supabase,1,1,238,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,672,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q052,Export images from Figma,1,1,239,1.0,1.0,1.0,1,328,1.0,1.0,1.0,1,701,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q053,Send an SMS with Twilio,1,1,215,1.0,1.0,1.0,1,396,1.0,1.0,1.0,1,740,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q054,Create an Okta user,2,1,183,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,753,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q055,Instant query Prometheus,1,1,221,1.0,1.0,1.0,1,327,1.0,1.0,1.0,1,666,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q056,Search Grafana dashboards,1,1,182,1.0,1.0,1.0,1,288,1.0,1.0,1.0,1,663,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q057,Trigger a CircleCI pipeline,1,1,200,1.0,1.0,1.0,1,325,1.0,1.0,1.0,1,755,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q058,Run terraform plan,1,1,172,1.0,1.0,1.0,1,287,1.0,1.0,1.0,1,593,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q059,Create an Asana task,2,1,267,1.0,1.0,1.0,1,429,1.0,1.0,1.0,1,841,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q060,Create a Trello card,1,1,240,1.0,1.0,1.0,1,411,1.0,1.0,1.0,1,888,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q061,Create a ClickUp task,2,1,257,1.0,1.0,1.0,1,415,1.0,1.0,1.0,1,878,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q062,Create a Confluence page,1,1,260,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,833,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q063,Send a Discord message,1,1,224,1.0,1.0,1.0,1,343,1.0,1.0,1.0,1,656,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q064,Search Elasticsearch,1,1,239,1.0,1.0,1.0,1,343,1.0,1.0,1.0,1,660,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q065,Get a Redis key,1,1,182,1.0,1.0,1.0,1,281,1.0,1.0,1.0,1,627,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q066,Find MongoDB documents,1,1,258,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,742,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q067,List SQLite tables,1,1,151,1.0,1.0,1.0,1,240,1.0,1.0,1.0,1,536,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q068,Describe a postgres table,1,1,182,1.0,1.0,1.0,1,290,1.0,1.0,1.0,1,640,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q069,github_create_issue,1,1,355,1.0,1.0,1.0,1,582,1.0,1.0,1.0,1,1050,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q070,createIssue,2,1,332,1.0,1.0,1.0,1,600,1.0,1.0,1.0,1,999,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q071,list-directory,1,1,186,1.0,1.0,1.0,1,281,1.0,1.0,1.0,1,643,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q072,sequential thinking next thought,1,1,278,1.0,1.0,1.0,1,434,1.0,1.0,1.0,1,862,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q073,What time is it in Tokyo,1,1,217,1.0,1.0,1.0,1,392,1.0,1.0,1.0,1,876,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q074,Convert time between timezones,1,1,204,1.0,1.0,1.0,1,319,1.0,1.0,1.0,1,687,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q075,Star a GitHub repo,1,1,226,1.0,1.0,1.0,1,379,1.0,1.0,1.0,1,789,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q076,Fork a repository,1,1,235,1.0,1.0,1.0,1,378,1.0,1.0,1.0,1,758,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q077,Get the latest GitHub release,1,1,239,1.0,1.0,1.0,1,383,1.0,1.0,1.0,1,725,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q078,Create a gist,1,1,272,1.0,1.0,1.0,1,473,1.0,1.0,1.0,1,966,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q079,Search GitHub code,1,1,283,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,746,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q080,List GitHub notifications,1,1,190,1.0,1.0,1.0,1,326,1.0,1.0,1.0,1,639,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q081,Assign Copilot to an issue,1,1,267,1.0,1.0,1.0,1,468,1.0,1.0,1.0,1,878,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q082,Get secret scanning alerts,1,1,295,1.0,1.0,1.0,1,470,1.0,1.0,1.0,1,733,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q083,Create or update a file on GitHub,1,1,333,1.0,1.0,1.0,1,527,1.0,1.0,1.0,1,1127,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q084,List collaborators on a repo,1,1,278,1.0,1.0,1.0,1,472,1.0,1.0,1.0,1,860,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q085,Reply in a Slack thread,1,1,239,1.0,1.0,1.0,1,349,1.0,1.0,1.0,1,668,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q086,Add an emoji reaction on Slack,1,1,302,1.0,1.0,1.0,1,465,1.0,1.0,1.0,1,816,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q087,Search Slack messages,1,1,231,1.0,1.0,1.0,1,373,1.0,1.0,1.0,1,661,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q088,Create a Slack channel,2,1,196,1.0,1.0,1.0,1,339,1.0,1.0,1.0,1,751,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q089,Transition a Jira issue,1,1,208,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,748,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q090,Search Jira with JQL,1,1,178,1.0,1.0,1.0,1,283,1.0,1.0,1.0,1,617,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q091,Add a worklog to Jira,1,1,249,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,610,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q092,List Linear teams,1,1,184,1.0,1.0,1.0,1,339,1.0,1.0,1.0,1,690,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q093,Comment on a Linear issue,1,1,213,1.0,1.0,1.0,1,383,1.0,1.0,1.0,1,831,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q094,Move a local file,3,1,224,1.0,1.0,1.0,1,335,1.0,1.0,1.0,1,686,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q095,Recursively search files by pattern,1,1,192,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,696,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q096,Edit a file in place with old and new string,1,1,230,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,776,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q097,Read the whole knowledge graph,1,1,152,1.0,1.0,1.0,1,253,1.0,1.0,1.0,1,561,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q098,Search memory nodes,1,1,176,1.0,1.0,1.0,1,331,1.0,1.0,1.0,1,625,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q099,Show unstaged git diff,1,1,190,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,613,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q100,Stage files with git add,1,1,193,1.0,1.0,1.0,1,332,1.0,1.0,1.0,1,694,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q101,Create a Shopify product,1,1,240,1.0,1.0,1.0,1,380,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q102,Cancel a Stripe subscription,1,1,188,1.0,1.0,1.0,1,321,1.0,1.0,1.0,1,663,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q103,Create a payment intent,1,1,234,1.0,1.0,1.0,1,424,1.0,1.0,1.0,1,906,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q104,Acknowledge a PagerDuty incident,1,1,261,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,665,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q105,Create a Datadog monitor,2,1,259,1.0,1.0,1.0,1,392,1.0,1.0,1.0,1,851,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q106,Resolve a Sentry issue,2,1,251,1.0,1.0,1.0,1,412,1.0,1.0,1.0,1,817,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q107,Scale a Kubernetes deployment,1,1,252,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q108,Apply a Kubernetes manifest,1,1,253,1.0,1.0,1.0,1,366,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q109,Stop a docker container,1,1,188,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,655,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q110,Build a docker image,1,1,211,1.0,1.0,1.0,1,317,1.0,1.0,1.0,1,598,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q111,Click an element in Playwright,1,1,216,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,564,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q112,Type text into a browser input,1,1,217,1.0,1.0,1.0,1,316,1.0,1.0,1.0,1,589,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q113,List GitLab projects,1,1,183,1.0,1.0,1.0,1,289,1.0,1.0,1.0,1,564,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q114,Create a Bitbucket pull request,1,1,313,1.0,1.0,1.0,1,553,1.0,1.0,1.0,1,1037,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q115,Create a Gmail draft,1,1,235,1.0,1.0,1.0,1,439,1.0,1.0,1.0,1,903,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q116,Share a Google Drive file,1,1,254,1.0,1.0,1.0,1,418,1.0,1.0,1.0,1,762,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q117,List Google Calendar events,1,1,219,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,754,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q118,Delete a calendar event,1,1,300,1.0,1.0,1.0,1,440,1.0,1.0,1.0,1,685,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q119,List Vercel projects,1,1,163,1.0,1.0,1.0,1,255,1.0,1.0,1.0,1,550,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q120,Select rows from Supabase,1,1,237,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,688,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q121,Post a Figma comment,1,1,201,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,639,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q122,List Okta users,1,1,198,1.0,1.0,1.0,1,288,1.0,1.0,1.0,1,618,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q123,Range query Prometheus,1,1,221,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,674,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q124,Create a Grafana annotation,1,1,237,1.0,1.0,1.0,1,435,1.0,1.0,1.0,1,923,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q125,Apply terraform,1,1,171,1.0,1.0,1.0,1,327,1.0,1.0,1.0,1,644,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q126,List Asana tasks,1,1,244,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,724,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q127,Comment on a Trello card,1,1,260,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,703,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q128,Update a ClickUp task,1,1,257,1.0,1.0,1.0,1,399,1.0,1.0,1.0,1,736,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q129,Search Confluence,1,1,221,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,708,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q130,Create a HubSpot deal,1,1,255,1.0,1.0,1.0,1,413,1.0,1.0,1.0,1,860,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q131,List Zendesk tickets,1,1,189,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q132,List Discord channels,1,1,166,1.0,1.0,1.0,1,321,1.0,1.0,1.0,1,666,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q133,Set a Redis key,1,1,182,1.0,1.0,1.0,1,281,1.0,1.0,1.0,1,586,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q134,Insert MongoDB documents,1,1,258,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,751,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q135,Run a MySQL query,2,1,182,1.0,1.0,1.0,1,294,1.0,1.0,1.0,1,585,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q136,Index a document in Elasticsearch,1,1,242,1.0,1.0,1.0,1,342,1.0,1.0,1.0,1,734,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q137,Get an S3 object,1,1,249,1.0,1.0,1.0,1,375,1.0,1.0,1.0,1,691,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q138,List Cloudflare zones,1,1,182,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,729,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q139,Create a Stripe invoice,1,1,204,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,732,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q140,List GitHub Actions workflows,1,1,293,1.0,1.0,1.0,1,459,1.0,1.0,1.0,1,783,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q141,Get GitHub Actions job logs,1,1,269,1.0,1.0,1.0,1,440,1.0,1.0,1.0,1,802,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q142,Create a GitHub repository,1,1,235,1.0,1.0,1.0,1,388,1.0,1.0,1.0,1,874,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q143,Checkout a git branch,1,1,212,1.0,1.0,1.0,1,376,1.0,1.0,1.0,1,864,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q144,Show git log,1,1,206,1.0,1.0,1.0,1,309,1.0,1.0,1.0,1,603,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q145,List postgres schemas,1,1,158,1.0,1.0,1.0,1,277,1.0,1.0,1.0,1,596,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q146,Open nodes in the knowledge graph,1,1,163,1.0,1.0,1.0,1,262,1.0,1.0,1.0,1,492,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q147,Get file metadata from the filesystem,3,1,220,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,694,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q148,Write a new local file,1,1,234,1.0,1.0,1.0,1,384,1.0,1.0,1.0,1,705,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q149,Search GitHub issues,1,1,258,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,683,1.0,1.0,1.0 +hybrid_rrf_weighted_trigram,q150,Get the authenticated GitHub user,2,1,179,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,694,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q001,Create a GitHub issue,1,1,388,1.0,1.0,1.0,1,598,1.0,1.0,1.0,1,955,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q002,Create a Jira ticket,1,1,312,1.0,1.0,1.0,1,412,1.0,1.0,1.0,1,763,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q003,Send a Slack message,1,1,232,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,707,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q004,List open pull requests,1,1,234,1.0,1.0,1.0,1,429,1.0,1.0,1.0,1,1035,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q005,Open a pull request on GitHub,1,1,368,1.0,1.0,1.0,1,653,1.0,1.0,1.0,1,1052,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q006,Run a SQL query against postgres,1,1,167,1.0,1.0,1.0,1,280,1.0,1.0,1.0,1,544,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q007,Read a local file from disk,1,1,223,1.0,1.0,1.0,1,385,1.0,1.0,1.0,1,782,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q008,What files are in this folder,6,2,473,0.0,1.0,1.0,2,730,0.0,1.0,1.0,1,708,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q009,Remember this fact in the knowledge graph,2,1,156,1.0,1.0,1.0,1,262,1.0,1.0,1.0,1,524,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q010,Search the web with Brave,1,1,245,1.0,1.0,1.0,1,412,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q011,Fetch the contents of a URL,1,1,258,1.0,1.0,1.0,1,372,1.0,1.0,1.0,1,707,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q012,Show git status,1,1,196,1.0,1.0,1.0,1,304,1.0,1.0,1.0,1,600,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q013,Commit staged changes,1,1,218,1.0,1.0,1.0,1,328,1.0,1.0,1.0,1,663,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q014,Create a Linear issue,1,1,340,1.0,1.0,1.0,1,463,1.0,1.0,1.0,1,765,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q015,Make a task in my project tracker Linear,2,1,352,1.0,1.0,1.0,1,432,1.0,1.0,1.0,1,769,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q016,Post a comment on a GitHub issue,2,1,333,1.0,1.0,1.0,1,573,1.0,1.0,1.0,1,1028,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q017,Merge the pull request,1,1,283,1.0,1.0,1.0,1,468,1.0,1.0,1.0,1,1121,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q018,Trigger a GitHub Actions workflow,1,1,294,1.0,1.0,1.0,1,450,1.0,1.0,1.0,1,737,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q019,Get Dependabot alerts,1,1,287,1.0,1.0,1.0,1,445,1.0,1.0,1.0,1,798,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q020,List Sentry issues,1,1,238,1.0,1.0,1.0,1,396,1.0,1.0,1.0,1,796,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q021,Create a Stripe customer,1,1,186,1.0,1.0,1.0,1,300,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q022,Refund a Stripe payment,1,1,234,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,662,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q023,Send an email with Gmail,1,1,224,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,685,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q024,Search my Gmail inbox,1,1,205,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,714,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q025,Upload a file to Google Drive,4,2,496,0.0,1.0,1.0,1,418,1.0,1.0,1.0,1,829,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q026,Create a calendar event,1,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,895,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q027,List Kubernetes pods,1,1,184,1.0,1.0,1.0,1,314,1.0,1.0,1.0,1,718,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q028,Get logs from a k8s pod,1,1,262,1.0,1.0,1.0,1,455,1.0,1.0,1.0,1,834,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q029,Run a docker container,1,1,219,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,644,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q030,Show docker container logs,1,1,188,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,686,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q031,Navigate the browser to a URL,1,1,180,1.0,1.0,1.0,1,294,1.0,1.0,1.0,1,536,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q032,Take a screenshot of the page,1,1,226,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,711,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q033,Create a GitLab merge request,3,1,297,1.0,1.0,1.0,1,475,1.0,1.0,1.0,1,792,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q034,Search Notion pages,1,1,195,1.0,1.0,1.0,1,366,1.0,1.0,1.0,1,686,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q035,Create a Notion page,3,1,230,1.0,1.0,1.0,1,371,1.0,1.0,1.0,1,687,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q036,Query a Notion database,1,1,279,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,687,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q037,Create a Zendesk ticket,1,1,257,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,856,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q038,Create a HubSpot contact,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,904,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q039,Create a Salesforce lead,2,1,252,1.0,1.0,1.0,1,402,1.0,1.0,1.0,1,787,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q040,Run a SOQL query,1,1,186,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,724,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q041,List Shopify orders,1,1,208,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,682,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q042,Create a PagerDuty incident,1,1,259,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,838,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q043,Who is on call in PagerDuty,1,1,227,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,723,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q044,Query Datadog metrics,1,1,300,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,728,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q045,Search Datadog logs,1,1,242,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,863,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q046,List S3 buckets,1,1,186,1.0,1.0,1.0,1,318,1.0,1.0,1.0,1,667,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q047,Upload an object to S3,1,1,250,1.0,1.0,1.0,1,437,1.0,1.0,1.0,1,950,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q048,Purge Cloudflare cache,1,1,231,1.0,1.0,1.0,1,340,1.0,1.0,1.0,1,746,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q049,Create a Cloudflare DNS record,1,1,217,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,738,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q050,List Vercel deployments,1,1,177,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,579,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q051,Insert a row into Supabase,1,1,238,1.0,1.0,1.0,1,355,1.0,1.0,1.0,1,665,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q052,Export images from Figma,1,1,239,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,658,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q053,Send an SMS with Twilio,1,1,249,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,730,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q054,Create an Okta user,3,1,183,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,656,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q055,Instant query Prometheus,1,1,221,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,680,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q056,Search Grafana dashboards,1,1,217,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,690,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q057,Trigger a CircleCI pipeline,1,1,200,1.0,1.0,1.0,1,390,1.0,1.0,1.0,1,755,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q058,Run terraform plan,1,1,172,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,717,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q059,Create an Asana task,2,1,267,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,825,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q060,Create a Trello card,1,1,240,1.0,1.0,1.0,1,394,1.0,1.0,1.0,1,884,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q061,Create a ClickUp task,2,1,257,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,751,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q062,Create a Confluence page,1,1,260,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,703,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q063,Send a Discord message,1,1,224,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,686,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q064,Search Elasticsearch,1,1,239,1.0,1.0,1.0,1,404,1.0,1.0,1.0,1,699,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q065,Get a Redis key,1,1,182,1.0,1.0,1.0,1,280,1.0,1.0,1.0,1,589,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q066,Find MongoDB documents,1,1,258,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,738,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q067,List SQLite tables,1,1,151,1.0,1.0,1.0,1,240,1.0,1.0,1.0,1,514,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q068,Describe a postgres table,1,1,186,1.0,1.0,1.0,1,272,1.0,1.0,1.0,1,597,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q069,github_create_issue,1,1,386,1.0,1.0,1.0,1,596,1.0,1.0,1.0,1,1084,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q070,createIssue,1,1,377,1.0,1.0,1.0,1,600,1.0,1.0,1.0,1,1003,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q071,list-directory,2,1,182,1.0,1.0,1.0,1,296,1.0,1.0,1.0,1,607,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q072,sequential thinking next thought,1,1,224,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,797,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q073,What time is it in Tokyo,1,1,217,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,783,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q074,Convert time between timezones,1,1,204,1.0,1.0,1.0,1,335,1.0,1.0,1.0,1,704,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q075,Star a GitHub repo,1,1,195,1.0,1.0,1.0,1,309,1.0,1.0,1.0,1,604,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q076,Fork a repository,1,1,235,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,792,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q077,Get the latest GitHub release,1,1,249,1.0,1.0,1.0,1,366,1.0,1.0,1.0,1,647,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q078,Create a gist,1,1,229,1.0,1.0,1.0,1,429,1.0,1.0,1.0,1,912,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q079,Search GitHub code,1,1,283,1.0,1.0,1.0,1,406,1.0,1.0,1.0,1,697,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q080,List GitHub notifications,1,1,169,1.0,1.0,1.0,1,292,1.0,1.0,1.0,1,585,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q081,Assign Copilot to an issue,1,1,267,1.0,1.0,1.0,1,434,1.0,1.0,1.0,1,928,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q082,Get secret scanning alerts,1,1,295,1.0,1.0,1.0,1,470,1.0,1.0,1.0,1,830,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q083,Create or update a file on GitHub,1,1,390,1.0,1.0,1.0,1,519,1.0,1.0,1.0,1,869,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q084,List collaborators on a repo,1,1,295,1.0,1.0,1.0,1,501,1.0,1.0,1.0,1,964,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q085,Reply in a Slack thread,1,1,254,1.0,1.0,1.0,1,358,1.0,1.0,1.0,1,717,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q086,Add an emoji reaction on Slack,1,1,216,1.0,1.0,1.0,1,357,1.0,1.0,1.0,1,709,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q087,Search Slack messages,1,1,213,1.0,1.0,1.0,1,337,1.0,1.0,1.0,1,689,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q088,Create a Slack channel,1,1,196,1.0,1.0,1.0,1,342,1.0,1.0,1.0,1,751,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q089,Transition a Jira issue,1,1,208,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,763,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q090,Search Jira with JQL,1,1,194,1.0,1.0,1.0,1,370,1.0,1.0,1.0,1,764,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q091,Add a worklog to Jira,1,1,249,1.0,1.0,1.0,1,350,1.0,1.0,1.0,1,765,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q092,List Linear teams,1,1,184,1.0,1.0,1.0,1,380,1.0,1.0,1.0,1,764,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q093,Comment on a Linear issue,1,1,334,1.0,1.0,1.0,1,464,1.0,1.0,1.0,1,831,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q094,Move a local file,1,1,214,1.0,1.0,1.0,1,358,1.0,1.0,1.0,1,699,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q095,Recursively search files by pattern,1,1,280,1.0,1.0,1.0,1,433,1.0,1.0,1.0,1,703,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q096,Edit a file in place with old and new string,1,1,227,1.0,1.0,1.0,1,394,1.0,1.0,1.0,1,682,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q097,Read the whole knowledge graph,1,1,152,1.0,1.0,1.0,1,259,1.0,1.0,1.0,1,529,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q098,Search memory nodes,1,1,202,1.0,1.0,1.0,1,301,1.0,1.0,1.0,1,546,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q099,Show unstaged git diff,1,1,190,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,602,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q100,Stage files with git add,2,1,213,1.0,1.0,1.0,1,336,1.0,1.0,1.0,1,677,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q101,Create a Shopify product,1,1,240,1.0,1.0,1.0,1,380,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q102,Cancel a Stripe subscription,1,1,188,1.0,1.0,1.0,1,301,1.0,1.0,1.0,1,632,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q103,Create a payment intent,1,1,234,1.0,1.0,1.0,1,417,1.0,1.0,1.0,1,897,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q104,Acknowledge a PagerDuty incident,1,1,261,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,717,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q105,Create a Datadog monitor,2,1,259,1.0,1.0,1.0,1,459,1.0,1.0,1.0,1,927,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q106,Resolve a Sentry issue,2,1,251,1.0,1.0,1.0,1,397,1.0,1.0,1.0,1,817,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q107,Scale a Kubernetes deployment,1,1,263,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q108,Apply a Kubernetes manifest,1,1,253,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q109,Stop a docker container,1,1,188,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,641,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q110,Build a docker image,1,1,224,1.0,1.0,1.0,1,317,1.0,1.0,1.0,1,601,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q111,Click an element in Playwright,1,1,216,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,566,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q112,Type text into a browser input,1,1,217,1.0,1.0,1.0,1,302,1.0,1.0,1.0,1,554,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q113,List GitLab projects,1,1,183,1.0,1.0,1.0,1,306,1.0,1.0,1.0,1,599,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q114,Create a Bitbucket pull request,1,1,313,1.0,1.0,1.0,1,471,1.0,1.0,1.0,1,1091,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q115,Create a Gmail draft,1,1,205,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,835,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q116,Share a Google Drive file,1,1,254,1.0,1.0,1.0,1,417,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q117,List Google Calendar events,1,1,269,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,784,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q118,Delete a calendar event,1,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,751,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q119,List Vercel projects,1,1,177,1.0,1.0,1.0,1,301,1.0,1.0,1.0,1,550,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q120,Select rows from Supabase,1,1,237,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,709,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q121,Post a Figma comment,1,1,204,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q122,List Okta users,1,1,177,1.0,1.0,1.0,1,306,1.0,1.0,1.0,1,606,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q123,Range query Prometheus,1,1,221,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,680,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q124,Create a Grafana annotation,1,1,218,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,881,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q125,Apply terraform,1,1,171,1.0,1.0,1.0,1,292,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q126,List Asana tasks,1,1,244,1.0,1.0,1.0,1,404,1.0,1.0,1.0,1,682,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q127,Comment on a Trello card,1,1,260,1.0,1.0,1.0,1,422,1.0,1.0,1.0,1,757,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q128,Update a ClickUp task,1,1,257,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,761,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q129,Search Confluence,1,1,221,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,731,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q130,Create a HubSpot deal,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,904,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q131,List Zendesk tickets,1,1,189,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,645,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q132,List Discord channels,1,1,166,1.0,1.0,1.0,1,321,1.0,1.0,1.0,1,625,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q133,Set a Redis key,1,1,182,1.0,1.0,1.0,1,280,1.0,1.0,1.0,1,600,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q134,Insert MongoDB documents,1,1,256,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q135,Run a MySQL query,2,1,182,1.0,1.0,1.0,1,266,1.0,1.0,1.0,1,543,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q136,Index a document in Elasticsearch,1,1,242,1.0,1.0,1.0,1,396,1.0,1.0,1.0,1,753,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q137,Get an S3 object,1,1,249,1.0,1.0,1.0,1,355,1.0,1.0,1.0,1,706,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q138,List Cloudflare zones,1,1,182,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,629,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q139,Create a Stripe invoice,1,1,204,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q140,List GitHub Actions workflows,1,1,293,1.0,1.0,1.0,1,433,1.0,1.0,1.0,1,736,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q141,Get GitHub Actions job logs,1,1,269,1.0,1.0,1.0,1,450,1.0,1.0,1.0,1,747,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q142,Create a GitHub repository,1,1,250,1.0,1.0,1.0,1,388,1.0,1.0,1.0,1,750,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q143,Checkout a git branch,1,1,212,1.0,1.0,1.0,1,373,1.0,1.0,1.0,1,666,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q144,Show git log,1,1,206,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,600,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q145,List postgres schemas,1,1,158,1.0,1.0,1.0,1,277,1.0,1.0,1.0,1,555,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q146,Open nodes in the knowledge graph,1,1,163,1.0,1.0,1.0,1,262,1.0,1.0,1.0,1,549,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q147,Get file metadata from the filesystem,4,2,462,0.0,1.0,1.0,1,348,1.0,1.0,1.0,1,688,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q148,Write a new local file,1,1,217,1.0,1.0,1.0,1,366,1.0,1.0,1.0,1,714,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q149,Search GitHub issues,1,1,270,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,670,1.0,1.0,1.0 +lexical_rerank_weighted_top20,q150,Get the authenticated GitHub user,6,2,382,0.0,1.0,1.0,2,615,0.0,1.0,1.0,1,593,1.0,1.0,1.0 +name_exact_substr,q001,Create a GitHub issue,29,10,2782,0.0,0.0,0.0,6,2698,0.0,0.0,0.0,3,2635,0.0,0.0,1.0 +name_exact_substr,q002,Create a Jira ticket,146,49,12183,0.0,0.0,0.0,30,11995,0.0,0.0,0.0,15,11695,0.0,0.0,0.0 +name_exact_substr,q003,Send a Slack message,121,41,10292,0.0,0.0,0.0,25,10122,0.0,0.0,0.0,13,10175,0.0,0.0,0.0 +name_exact_substr,q004,List open pull requests,37,13,3710,0.0,0.0,0.0,8,3752,0.0,0.0,0.0,4,3672,0.0,0.0,0.0 +name_exact_substr,q005,Open a pull request on GitHub,36,12,3495,0.0,0.0,0.0,8,3776,0.0,0.0,0.0,4,3684,0.0,0.0,0.0 +name_exact_substr,q006,Run a SQL query against postgres,98,33,8598,0.0,0.0,0.0,20,8376,0.0,0.0,0.0,10,8156,0.0,0.0,0.0 +name_exact_substr,q007,Read a local file from disk,76,26,7282,0.0,0.0,0.0,16,7224,0.0,0.0,0.0,8,7048,0.0,0.0,0.0 +name_exact_substr,q008,What files are in this folder,82,28,7711,0.0,0.0,0.0,17,7530,0.0,0.0,0.0,9,7616,0.0,0.0,0.0 +name_exact_substr,q009,Remember this fact in the knowledge graph,89,30,8108,0.0,0.0,0.0,18,7832,0.0,0.0,0.0,9,7625,0.0,0.0,0.0 +name_exact_substr,q010,Search the web with Brave,119,40,10063,0.0,0.0,0.0,24,9727,0.0,0.0,0.0,12,9475,0.0,0.0,0.0 +name_exact_substr,q011,Fetch the contents of a URL,103,35,8982,0.0,0.0,0.0,21,8674,0.0,0.0,0.0,11,8746,0.0,0.0,0.0 +name_exact_substr,q012,Show git status,104,35,8877,0.0,0.0,0.0,21,8611,0.0,0.0,0.0,11,8713,0.0,0.0,0.0 +name_exact_substr,q013,Commit staged changes,108,36,9075,0.0,0.0,0.0,22,8922,0.0,0.0,0.0,11,8713,0.0,0.0,0.0 +name_exact_substr,q014,Create a Linear issue,136,46,11528,0.0,0.0,0.0,28,11279,0.0,0.0,0.0,14,10999,0.0,0.0,0.0 +name_exact_substr,q015,Make a task in my project tracker Linear,136,46,11712,0.0,0.0,0.0,28,11391,0.0,0.0,0.0,14,11055,0.0,0.0,0.0 +name_exact_substr,q016,Post a comment on a GitHub issue,34,12,3507,0.0,0.0,0.0,7,3237,0.0,0.0,0.0,4,3688,0.0,0.0,0.0 +name_exact_substr,q017,Merge the pull request,41,14,4072,0.0,0.0,0.0,9,4373,0.0,0.0,0.0,5,4766,0.0,0.0,0.0 +name_exact_substr,q018,Trigger a GitHub Actions workflow,62,21,6123,0.0,0.0,0.0,13,6050,0.0,0.0,0.0,7,6293,0.0,0.0,0.0 +name_exact_substr,q019,Get Dependabot alerts,51,17,5097,0.0,0.0,0.0,11,5225,0.0,0.0,0.0,6,5482,0.0,0.0,0.0 +name_exact_substr,q020,List Sentry issues,228,76,18006,0.0,0.0,0.0,46,17548,0.0,0.0,0.0,23,17088,0.0,0.0,0.0 +name_exact_substr,q021,Create a Stripe customer,200,67,15974,0.0,0.0,0.0,40,15384,0.0,0.0,0.0,20,14984,0.0,0.0,0.0 +name_exact_substr,q022,Refund a Stripe payment,211,71,16820,0.0,0.0,0.0,43,16424,0.0,0.0,0.0,22,16307,0.0,0.0,0.0 +name_exact_substr,q023,Send an email with Gmail,244,82,19588,0.0,0.0,0.0,49,18841,0.0,0.0,0.0,25,18655,0.0,0.0,0.0 +name_exact_substr,q024,Search my Gmail inbox,245,82,19506,0.0,0.0,0.0,49,18792,0.0,0.0,0.0,25,18630,0.0,0.0,0.0 +name_exact_substr,q025,Upload a file to Google Drive,253,85,20366,0.0,0.0,0.0,51,19618,0.0,0.0,0.0,26,19523,0.0,0.0,0.0 +name_exact_substr,q026,Create a calendar event,258,86,20511,0.0,0.0,0.0,52,19991,0.0,0.0,0.0,26,19471,0.0,0.0,0.0 +name_exact_substr,q027,List Kubernetes pods,191,64,15221,0.0,0.0,0.0,39,14999,0.0,0.0,0.0,20,14964,0.0,0.0,0.0 +name_exact_substr,q028,Get logs from a k8s pod,193,65,15818,0.0,0.0,0.0,39,15194,0.0,0.0,0.0,20,15064,0.0,0.0,0.0 +name_exact_substr,q029,Run a docker container,185,62,14888,0.0,0.0,0.0,37,14342,0.0,0.0,0.0,19,14250,0.0,0.0,0.0 +name_exact_substr,q030,Show docker container logs,184,62,14888,0.0,0.0,0.0,37,14342,0.0,0.0,0.0,19,14250,0.0,0.0,0.0 +name_exact_substr,q031,Navigate the browser to a URL,166,56,13816,0.0,0.0,0.0,34,13458,0.0,0.0,0.0,17,13084,0.0,0.0,0.0 +name_exact_substr,q032,Take a screenshot of the page,171,57,14038,0.0,0.0,0.0,35,13740,0.0,0.0,0.0,18,13672,0.0,0.0,0.0 +name_exact_substr,q033,Create a GitLab merge request,265,89,21452,0.0,0.0,0.0,53,20508,0.0,0.0,0.0,27,20309,0.0,0.0,0.0 +name_exact_substr,q034,Search Notion pages,156,52,12874,0.0,0.0,0.0,32,12736,0.0,0.0,0.0,16,12416,0.0,0.0,0.0 +name_exact_substr,q035,Create a Notion page,158,53,13156,0.0,0.0,0.0,32,12768,0.0,0.0,0.0,16,12432,0.0,0.0,0.0 +name_exact_substr,q036,Query a Notion database,162,54,13386,0.0,0.0,0.0,33,13124,0.0,0.0,0.0,17,13067,0.0,0.0,0.0 +name_exact_substr,q037,Create a Zendesk ticket,357,119,28019,0.0,0.0,0.0,72,27234,0.0,0.0,0.0,36,26478,0.0,0.0,0.0 +name_exact_substr,q038,Create a HubSpot contact,352,118,27762,0.0,0.0,0.0,71,26866,0.0,0.0,0.0,36,26478,0.0,0.0,0.0 +name_exact_substr,q039,Create a Salesforce lead,226,76,18006,0.0,0.0,0.0,46,17548,0.0,0.0,0.0,23,17088,0.0,0.0,0.0 +name_exact_substr,q040,Run a SOQL query,220,74,17580,0.0,0.0,0.0,44,16791,0.0,0.0,0.0,22,16329,0.0,0.0,0.0 +name_exact_substr,q041,List Shopify orders,216,72,16998,0.0,0.0,0.0,44,16703,0.0,0.0,0.0,22,16285,0.0,0.0,0.0 +name_exact_substr,q042,Create a PagerDuty incident,240,80,19130,0.0,0.0,0.0,48,18458,0.0,0.0,0.0,24,17954,0.0,0.0,0.0 +name_exact_substr,q043,Who is on call in PagerDuty,243,81,19511,0.0,0.0,0.0,49,18939,0.0,0.0,0.0,25,18705,0.0,0.0,0.0 +name_exact_substr,q044,Query Datadog metrics,233,78,18569,0.0,0.0,0.0,47,17977,0.0,0.0,0.0,24,17954,0.0,0.0,0.0 +name_exact_substr,q045,Search Datadog logs,237,79,18863,0.0,0.0,0.0,48,18458,0.0,0.0,0.0,24,17954,0.0,0.0,0.0 +name_exact_substr,q046,List S3 buckets,293,98,23231,0.0,0.0,0.0,59,22522,0.0,0.0,0.0,30,22294,0.0,0.0,0.0 +name_exact_substr,q047,Upload an object to S3,296,99,23677,0.0,0.0,0.0,60,23014,0.0,0.0,0.0,30,22354,0.0,0.0,0.0 +name_exact_substr,q048,Purge Cloudflare cache,301,101,23981,0.0,0.0,0.0,61,23259,0.0,0.0,0.0,31,22962,0.0,0.0,0.0 +name_exact_substr,q049,Create a Cloudflare DNS record,300,100,23894,0.0,0.0,0.0,60,23014,0.0,0.0,0.0,30,22354,0.0,0.0,0.0 +name_exact_substr,q050,List Vercel deployments,303,101,23981,0.0,0.0,0.0,61,23259,0.0,0.0,0.0,31,22962,0.0,0.0,0.0 +name_exact_substr,q051,Insert a row into Supabase,308,103,24500,0.0,0.0,0.0,62,23675,0.0,0.0,0.0,31,22993,0.0,0.0,0.0 +name_exact_substr,q052,Export images from Figma,315,105,24863,0.0,0.0,0.0,63,23981,0.0,0.0,0.0,32,23662,0.0,0.0,0.0 +name_exact_substr,q053,Send an SMS with Twilio,316,106,25217,0.0,0.0,0.0,64,24398,0.0,0.0,0.0,32,23694,0.0,0.0,0.0 +name_exact_substr,q054,Create an Okta user,321,107,25289,0.0,0.0,0.0,65,24664,0.0,0.0,0.0,33,24298,0.0,0.0,0.0 +name_exact_substr,q055,Instant query Prometheus,323,108,25409,0.0,0.0,0.0,65,24599,0.0,0.0,0.0,33,24265,0.0,0.0,0.0 +name_exact_substr,q056,Search Grafana dashboards,326,109,25591,0.0,0.0,0.0,66,24925,0.0,0.0,0.0,33,24265,0.0,0.0,0.0 +name_exact_substr,q057,Trigger a CircleCI pipeline,331,111,26111,0.0,0.0,0.0,67,25340,0.0,0.0,0.0,34,25017,0.0,0.0,0.0 +name_exact_substr,q058,Run terraform plan,332,111,25889,0.0,0.0,0.0,67,25206,0.0,0.0,0.0,34,24949,0.0,0.0,0.0 +name_exact_substr,q059,Create an Asana task,335,112,26352,0.0,0.0,0.0,67,25340,0.0,0.0,0.0,34,25017,0.0,0.0,0.0 +name_exact_substr,q060,Create a Trello card,340,114,26830,0.0,0.0,0.0,68,25731,0.0,0.0,0.0,34,25017,0.0,0.0,0.0 +name_exact_substr,q061,Create a ClickUp task,344,115,27066,0.0,0.0,0.0,69,26100,0.0,0.0,0.0,35,25770,0.0,0.0,0.0 +name_exact_substr,q062,Create a Confluence page,349,117,27546,0.0,0.0,0.0,70,26505,0.0,0.0,0.0,35,25770,0.0,0.0,0.0 +name_exact_substr,q063,Send a Discord message,362,121,28315,0.0,0.0,0.0,73,27492,0.0,0.0,0.0,37,27133,0.0,0.0,0.0 +name_exact_substr,q064,Search Elasticsearch,290,97,22834,0.0,0.0,0.0,58,22047,0.0,0.0,0.0,29,21525,0.0,0.0,0.0 +name_exact_substr,q065,Get a Redis key,286,96,22810,0.0,0.0,0.0,58,22163,0.0,0.0,0.0,29,21583,0.0,0.0,0.0 +name_exact_substr,q066,Find MongoDB documents,281,94,22397,0.0,0.0,0.0,57,21868,0.0,0.0,0.0,29,21583,0.0,0.0,0.0 +name_exact_substr,q067,List SQLite tables,277,93,22087,0.0,0.0,0.0,56,21431,0.0,0.0,0.0,28,20899,0.0,0.0,0.0 +name_exact_substr,q068,Describe a postgres table,100,34,8708,0.0,0.0,0.0,20,8336,0.0,0.0,0.0,10,8136,0.0,0.0,0.0 +name_exact_substr,q069,github_create_issue,29,10,2762,0.0,0.0,0.0,6,2686,0.0,0.0,0.0,3,2629,0.0,0.0,1.0 +name_exact_substr,q070,createIssue,29,10,2752,0.0,0.0,0.0,6,2680,0.0,0.0,0.0,3,2626,0.0,0.0,1.0 +name_exact_substr,q071,list-directory,82,28,7599,0.0,0.0,0.0,17,7462,0.0,0.0,0.0,9,7580,0.0,0.0,0.0 +name_exact_substr,q072,sequential thinking next thought,116,39,9799,0.0,0.0,0.0,24,9727,0.0,0.0,0.0,12,9475,0.0,0.0,0.0 +name_exact_substr,q073,What time is it in Tokyo,117,39,9838,0.0,0.0,0.0,24,9751,0.0,0.0,0.0,12,9487,0.0,0.0,0.0 +name_exact_substr,q074,Convert time between timezones,118,40,10063,0.0,0.0,0.0,24,9727,0.0,0.0,0.0,12,9475,0.0,0.0,0.0 +name_exact_substr,q075,Star a GitHub repo,25,9,2387,0.0,0.0,0.0,5,2154,0.0,0.0,0.0,3,2635,0.0,0.0,1.0 +name_exact_substr,q076,Fork a repository,20,7,1831,0.0,0.0,0.0,4,1674,0.0,0.0,0.0,2,1636,0.0,1.0,1.0 +name_exact_substr,q077,Get the latest GitHub release,15,5,1242,0.0,0.0,0.0,3,1198,0.0,0.0,1.0,2,1642,0.0,1.0,1.0 +name_exact_substr,q078,Create a gist,66,22,6260,0.0,0.0,0.0,14,6405,0.0,0.0,0.0,7,6272,0.0,0.0,0.0 +name_exact_substr,q079,Search GitHub code,7,3,725,0.0,0.0,1.0,2,802,0.0,1.0,1.0,1,782,1.0,1.0,1.0 +name_exact_substr,q080,List GitHub notifications,53,18,5313,0.0,0.0,0.0,11,5225,0.0,0.0,0.0,6,5482,0.0,0.0,0.0 +name_exact_substr,q081,Assign Copilot to an issue,45,15,4523,0.0,0.0,0.0,9,4391,0.0,0.0,0.0,5,4776,0.0,0.0,0.0 +name_exact_substr,q082,Get secret scanning alerts,49,17,5097,0.0,0.0,0.0,10,4866,0.0,0.0,0.0,5,4766,0.0,0.0,0.0 +name_exact_substr,q083,Create or update a file on GitHub,17,6,1593,0.0,0.0,0.0,4,1694,0.0,0.0,0.0,2,1646,0.0,1.0,1.0 +name_exact_substr,q084,List collaborators on a repo,27,9,2387,0.0,0.0,0.0,6,2698,0.0,0.0,0.0,3,2635,0.0,0.0,1.0 +name_exact_substr,q085,Reply in a Slack thread,122,41,10333,0.0,0.0,0.0,25,10147,0.0,0.0,0.0,13,10188,0.0,0.0,0.0 +name_exact_substr,q086,Add an emoji reaction on Slack,123,41,10374,0.0,0.0,0.0,25,10172,0.0,0.0,0.0,13,10201,0.0,0.0,0.0 +name_exact_substr,q087,Search Slack messages,134,45,11132,0.0,0.0,0.0,27,10790,0.0,0.0,0.0,14,10985,0.0,0.0,0.0 +name_exact_substr,q088,Create a Slack channel,131,44,10940,0.0,0.0,0.0,27,10817,0.0,0.0,0.0,14,10999,0.0,0.0,0.0 +name_exact_substr,q089,Transition a Jira issue,151,51,12585,0.0,0.0,0.0,31,12385,0.0,0.0,0.0,16,12416,0.0,0.0,0.0 +name_exact_substr,q090,Search Jira with JQL,149,50,12445,0.0,0.0,0.0,30,12025,0.0,0.0,0.0,15,11710,0.0,0.0,0.0 +name_exact_substr,q091,Add a worklog to Jira,155,52,12978,0.0,0.0,0.0,31,12447,0.0,0.0,0.0,16,12448,0.0,0.0,0.0 +name_exact_substr,q092,List Linear teams,142,48,11860,0.0,0.0,0.0,29,11573,0.0,0.0,0.0,15,11680,0.0,0.0,0.0 +name_exact_substr,q093,Comment on a Linear issue,141,47,11773,0.0,0.0,0.0,29,11631,0.0,0.0,0.0,15,11710,0.0,0.0,0.0 +name_exact_substr,q094,Move a local file,85,29,7859,0.0,0.0,0.0,17,7496,0.0,0.0,0.0,9,7598,0.0,0.0,0.0 +name_exact_substr,q095,Recursively search files by pattern,86,29,7917,0.0,0.0,0.0,18,7814,0.0,0.0,0.0,9,7616,0.0,0.0,0.0 +name_exact_substr,q096,Edit a file in place with old and new string,80,27,7629,0.0,0.0,0.0,16,7288,0.0,0.0,0.0,8,7080,0.0,0.0,0.0 +name_exact_substr,q097,Read the whole knowledge graph,95,32,8392,0.0,0.0,0.0,19,8062,0.0,0.0,0.0,10,8146,0.0,0.0,0.0 +name_exact_substr,q098,Search memory nodes,96,32,8328,0.0,0.0,0.0,20,8316,0.0,0.0,0.0,10,8126,0.0,0.0,0.0 +name_exact_substr,q099,Show unstaged git diff,105,35,8947,0.0,0.0,0.0,21,8653,0.0,0.0,0.0,11,8735,0.0,0.0,0.0 +name_exact_substr,q100,Stage files with git add,109,37,9343,0.0,0.0,0.0,22,8966,0.0,0.0,0.0,11,8735,0.0,0.0,0.0 +name_exact_substr,q101,Create a Shopify product,214,72,17070,0.0,0.0,0.0,43,16424,0.0,0.0,0.0,22,16307,0.0,0.0,0.0 +name_exact_substr,q102,Cancel a Stripe subscription,210,70,16604,0.0,0.0,0.0,42,16044,0.0,0.0,0.0,21,15624,0.0,0.0,0.0 +name_exact_substr,q103,Create a payment intent,203,68,16204,0.0,0.0,0.0,41,15712,0.0,0.0,0.0,21,15624,0.0,0.0,0.0 +name_exact_substr,q104,Acknowledge a PagerDuty incident,241,81,19511,0.0,0.0,0.0,49,18939,0.0,0.0,0.0,25,18705,0.0,0.0,0.0 +name_exact_substr,q105,Create a Datadog monitor,236,79,18942,0.0,0.0,0.0,48,18506,0.0,0.0,0.0,24,17978,0.0,0.0,0.0 +name_exact_substr,q106,Resolve a Sentry issue,231,77,18333,0.0,0.0,0.0,47,17977,0.0,0.0,0.0,24,17954,0.0,0.0,0.0 +name_exact_substr,q107,Scale a Kubernetes deployment,197,66,15784,0.0,0.0,0.0,40,15384,0.0,0.0,0.0,20,14984,0.0,0.0,0.0 +name_exact_substr,q108,Apply a Kubernetes manifest,194,65,15558,0.0,0.0,0.0,39,15038,0.0,0.0,0.0,20,14984,0.0,0.0,0.0 +name_exact_substr,q109,Stop a docker container,186,62,14888,0.0,0.0,0.0,38,14630,0.0,0.0,0.0,19,14250,0.0,0.0,0.0 +name_exact_substr,q110,Build a docker image,189,63,15066,0.0,0.0,0.0,38,14630,0.0,0.0,0.0,19,14250,0.0,0.0,0.0 +name_exact_substr,q111,Click an element in Playwright,168,56,13816,0.0,0.0,0.0,34,13458,0.0,0.0,0.0,17,13084,0.0,0.0,0.0 +name_exact_substr,q112,Type text into a browser input,169,57,14038,0.0,0.0,0.0,34,13458,0.0,0.0,0.0,17,13084,0.0,0.0,0.0 +name_exact_substr,q113,List GitLab projects,269,90,21515,0.0,0.0,0.0,54,20795,0.0,0.0,0.0,27,20255,0.0,0.0,0.0 +name_exact_substr,q114,Create a Bitbucket pull request,272,91,21988,0.0,0.0,0.0,55,21340,0.0,0.0,0.0,28,20983,0.0,0.0,0.0 +name_exact_substr,q115,Create a Gmail draft,247,83,19734,0.0,0.0,0.0,50,19130,0.0,0.0,0.0,25,18630,0.0,0.0,0.0 +name_exact_substr,q116,Share a Google Drive file,256,86,20597,0.0,0.0,0.0,52,20043,0.0,0.0,0.0,26,19497,0.0,0.0,0.0 +name_exact_substr,q117,List Google Calendar events,257,86,20511,0.0,0.0,0.0,52,19991,0.0,0.0,0.0,26,19471,0.0,0.0,0.0 +name_exact_substr,q118,Delete a calendar event,260,87,20729,0.0,0.0,0.0,52,19991,0.0,0.0,0.0,26,19471,0.0,0.0,0.0 +name_exact_substr,q119,List Vercel projects,302,101,23981,0.0,0.0,0.0,61,23259,0.0,0.0,0.0,31,22962,0.0,0.0,0.0 +name_exact_substr,q120,Select rows from Supabase,307,103,24397,0.0,0.0,0.0,62,23613,0.0,0.0,0.0,31,22962,0.0,0.0,0.0 +name_exact_substr,q121,Post a Figma comment,314,105,24863,0.0,0.0,0.0,63,23981,0.0,0.0,0.0,32,23662,0.0,0.0,0.0 +name_exact_substr,q122,List Okta users,319,107,25182,0.0,0.0,0.0,64,24270,0.0,0.0,0.0,32,23630,0.0,0.0,0.0 +name_exact_substr,q123,Range query Prometheus,324,108,25409,0.0,0.0,0.0,65,24599,0.0,0.0,0.0,33,24265,0.0,0.0,0.0 +name_exact_substr,q124,Create a Grafana annotation,328,110,25915,0.0,0.0,0.0,66,24991,0.0,0.0,0.0,33,24298,0.0,0.0,0.0 +name_exact_substr,q125,Apply terraform,333,111,25778,0.0,0.0,0.0,67,25139,0.0,0.0,0.0,34,24915,0.0,0.0,0.0 +name_exact_substr,q126,List Asana tasks,336,112,26240,0.0,0.0,0.0,68,25663,0.0,0.0,0.0,34,24983,0.0,0.0,0.0 +name_exact_substr,q127,Comment on a Trello card,343,115,27181,0.0,0.0,0.0,69,26169,0.0,0.0,0.0,35,25805,0.0,0.0,0.0 +name_exact_substr,q128,Update a ClickUp task,347,116,27289,0.0,0.0,0.0,70,26505,0.0,0.0,0.0,35,25770,0.0,0.0,0.0 +name_exact_substr,q129,Search Confluence,351,117,27312,0.0,0.0,0.0,71,26724,0.0,0.0,0.0,36,26406,0.0,0.0,0.0 +name_exact_substr,q130,Create a HubSpot deal,355,119,28019,0.0,0.0,0.0,71,26866,0.0,0.0,0.0,36,26478,0.0,0.0,0.0 +name_exact_substr,q131,List Zendesk tickets,359,120,28122,0.0,0.0,0.0,72,27162,0.0,0.0,0.0,36,26442,0.0,0.0,0.0 +name_exact_substr,q132,List Discord channels,363,121,28194,0.0,0.0,0.0,73,27419,0.0,0.0,0.0,37,27096,0.0,0.0,0.0 +name_exact_substr,q133,Set a Redis key,287,96,22810,0.0,0.0,0.0,58,22163,0.0,0.0,0.0,29,21583,0.0,0.0,0.0 +name_exact_substr,q134,Insert MongoDB documents,282,94,22397,0.0,0.0,0.0,57,21868,0.0,0.0,0.0,29,21583,0.0,0.0,0.0 +name_exact_substr,q135,Run a MySQL query,279,93,22273,0.0,0.0,0.0,56,21543,0.0,0.0,0.0,28,20955,0.0,0.0,0.0 +name_exact_substr,q136,Index a document in Elasticsearch,291,97,23125,0.0,0.0,0.0,59,22581,0.0,0.0,0.0,30,22324,0.0,0.0,0.0 +name_exact_substr,q137,Get an S3 object,295,99,23578,0.0,0.0,0.0,59,22581,0.0,0.0,0.0,30,22324,0.0,0.0,0.0 +name_exact_substr,q138,List Cloudflare zones,298,100,23694,0.0,0.0,0.0,60,22894,0.0,0.0,0.0,30,22294,0.0,0.0,0.0 +name_exact_substr,q139,Create a Stripe invoice,205,69,16389,0.0,0.0,0.0,41,15712,0.0,0.0,0.0,21,15624,0.0,0.0,0.0 +name_exact_substr,q140,List GitHub Actions workflows,60,20,5782,0.0,0.0,0.0,12,5614,0.0,0.0,0.0,6,5488,0.0,0.0,0.0 +name_exact_substr,q141,Get GitHub Actions job logs,63,21,6123,0.0,0.0,0.0,13,6050,0.0,0.0,0.0,7,6293,0.0,0.0,0.0 +name_exact_substr,q142,Create a GitHub repository,18,6,1575,0.0,0.0,0.0,4,1682,0.0,0.0,0.0,2,1640,0.0,1.0,1.0 +name_exact_substr,q143,Checkout a git branch,113,38,9533,0.0,0.0,0.0,23,9279,0.0,0.0,0.0,12,9463,0.0,0.0,0.0 +name_exact_substr,q144,Show git log,111,37,9269,0.0,0.0,0.0,23,9256,0.0,0.0,0.0,12,9451,0.0,0.0,0.0 +name_exact_substr,q145,List postgres schemas,101,34,8674,0.0,0.0,0.0,21,8611,0.0,0.0,0.0,11,8713,0.0,0.0,0.0 +name_exact_substr,q146,Open nodes in the knowledge graph,97,33,8598,0.0,0.0,0.0,20,8376,0.0,0.0,0.0,10,8156,0.0,0.0,0.0 +name_exact_substr,q147,Get file metadata from the filesystem,87,29,7917,0.0,0.0,0.0,18,7814,0.0,0.0,0.0,9,7616,0.0,0.0,0.0 +name_exact_substr,q148,Write a new local file,79,27,7494,0.0,0.0,0.0,16,7208,0.0,0.0,0.0,8,7040,0.0,0.0,0.0 +name_exact_substr,q149,Search GitHub issues,32,11,3079,0.0,0.0,0.0,7,3209,0.0,0.0,0.0,4,3672,0.0,0.0,0.0 +name_exact_substr,q150,Get the authenticated GitHub user,1,1,181,1.0,1.0,1.0,1,379,1.0,1.0,1.0,1,784,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q001,Create a GitHub issue,1,1,388,1.0,1.0,1.0,1,598,1.0,1.0,1.0,1,1021,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q002,Create a Jira ticket,5,2,477,0.0,1.0,1.0,1,389,1.0,1.0,1.0,1,763,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q003,Send a Slack message,1,1,232,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,707,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q004,List open pull requests,1,1,234,1.0,1.0,1.0,1,429,1.0,1.0,1.0,1,1035,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q005,Open a pull request on GitHub,1,1,368,1.0,1.0,1.0,1,605,1.0,1.0,1.0,1,1069,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q006,Run a SQL query against postgres,1,1,167,1.0,1.0,1.0,1,280,1.0,1.0,1.0,1,590,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q007,Read a local file from disk,1,1,223,1.0,1.0,1.0,1,385,1.0,1.0,1.0,1,782,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q008,What files are in this folder,6,2,473,0.0,1.0,1.0,2,730,0.0,1.0,1.0,1,708,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q009,Remember this fact in the knowledge graph,2,1,154,1.0,1.0,1.0,1,261,1.0,1.0,1.0,1,524,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q010,Search the web with Brave,1,1,245,1.0,1.0,1.0,1,412,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q011,Fetch the contents of a URL,1,1,258,1.0,1.0,1.0,1,372,1.0,1.0,1.0,1,707,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q012,Show git status,1,1,196,1.0,1.0,1.0,1,304,1.0,1.0,1.0,1,600,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q013,Commit staged changes,1,1,218,1.0,1.0,1.0,1,328,1.0,1.0,1.0,1,663,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q014,Create a Linear issue,1,1,340,1.0,1.0,1.0,1,463,1.0,1.0,1.0,1,765,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q015,Make a task in my project tracker Linear,2,1,352,1.0,1.0,1.0,1,432,1.0,1.0,1.0,1,769,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q016,Post a comment on a GitHub issue,2,1,333,1.0,1.0,1.0,1,573,1.0,1.0,1.0,1,1067,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q017,Merge the pull request,1,1,283,1.0,1.0,1.0,1,523,1.0,1.0,1.0,1,1121,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q018,Trigger a GitHub Actions workflow,1,1,294,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,825,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q019,Get Dependabot alerts,1,1,287,1.0,1.0,1.0,1,445,1.0,1.0,1.0,1,798,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q020,List Sentry issues,1,1,238,1.0,1.0,1.0,1,396,1.0,1.0,1.0,1,796,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q021,Create a Stripe customer,2,1,186,1.0,1.0,1.0,1,300,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q022,Refund a Stripe payment,1,1,234,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,662,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q023,Send an email with Gmail,1,1,224,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,703,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q024,Search my Gmail inbox,1,1,205,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,714,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q025,Upload a file to Google Drive,4,2,496,0.0,1.0,1.0,1,418,1.0,1.0,1.0,1,791,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q026,Create a calendar event,2,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,895,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q027,List Kubernetes pods,1,1,184,1.0,1.0,1.0,1,314,1.0,1.0,1.0,1,718,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q028,Get logs from a k8s pod,1,1,252,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,732,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q029,Run a docker container,1,1,219,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,644,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q030,Show docker container logs,1,1,188,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,686,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q031,Navigate the browser to a URL,1,1,180,1.0,1.0,1.0,1,265,1.0,1.0,1.0,1,536,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q032,Take a screenshot of the page,1,1,226,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,711,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q033,Create a GitLab merge request,3,1,297,1.0,1.0,1.0,1,475,1.0,1.0,1.0,1,833,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q034,Search Notion pages,1,1,195,1.0,1.0,1.0,1,366,1.0,1.0,1.0,1,686,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q035,Create a Notion page,3,1,230,1.0,1.0,1.0,1,371,1.0,1.0,1.0,1,687,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q036,Query a Notion database,1,1,279,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,687,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q037,Create a Zendesk ticket,2,1,224,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,896,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q038,Create a HubSpot contact,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,832,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q039,Create a Salesforce lead,2,1,252,1.0,1.0,1.0,1,402,1.0,1.0,1.0,1,787,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q040,Run a SOQL query,1,1,186,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,724,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q041,List Shopify orders,1,1,208,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,682,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q042,Create a PagerDuty incident,1,1,259,1.0,1.0,1.0,1,403,1.0,1.0,1.0,1,838,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q043,Who is on call in PagerDuty,6,2,490,0.0,1.0,1.0,2,792,0.0,1.0,1.0,1,769,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q044,Query Datadog metrics,1,1,300,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,728,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q045,Search Datadog logs,1,1,242,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,877,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q046,List S3 buckets,1,1,186,1.0,1.0,1.0,1,318,1.0,1.0,1.0,1,667,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q047,Upload an object to S3,1,1,250,1.0,1.0,1.0,1,437,1.0,1.0,1.0,1,950,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q048,Purge Cloudflare cache,1,1,231,1.0,1.0,1.0,1,340,1.0,1.0,1.0,1,746,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q049,Create a Cloudflare DNS record,1,1,217,1.0,1.0,1.0,1,356,1.0,1.0,1.0,1,738,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q050,List Vercel deployments,1,1,177,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,579,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q051,Insert a row into Supabase,1,1,238,1.0,1.0,1.0,1,355,1.0,1.0,1.0,1,665,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q052,Export images from Figma,1,1,239,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,658,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q053,Send an SMS with Twilio,1,1,249,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,751,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q054,Create an Okta user,3,1,183,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,656,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q055,Instant query Prometheus,1,1,221,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,680,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q056,Search Grafana dashboards,1,1,217,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,690,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q057,Trigger a CircleCI pipeline,1,1,200,1.0,1.0,1.0,1,417,1.0,1.0,1.0,1,755,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q058,Run terraform plan,1,1,172,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,717,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q059,Create an Asana task,2,1,267,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,825,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q060,Create a Trello card,1,1,240,1.0,1.0,1.0,1,394,1.0,1.0,1.0,1,884,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q061,Create a ClickUp task,2,1,257,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,834,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q062,Create a Confluence page,1,1,260,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,703,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q063,Send a Discord message,1,1,224,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,682,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q064,Search Elasticsearch,1,1,239,1.0,1.0,1.0,1,404,1.0,1.0,1.0,1,699,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q065,Get a Redis key,1,1,182,1.0,1.0,1.0,1,280,1.0,1.0,1.0,1,589,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q066,Find MongoDB documents,1,1,258,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,738,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q067,List SQLite tables,1,1,156,1.0,1.0,1.0,1,252,1.0,1.0,1.0,1,514,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q068,Describe a postgres table,1,1,186,1.0,1.0,1.0,1,272,1.0,1.0,1.0,1,597,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q069,github_create_issue,1,1,386,1.0,1.0,1.0,1,596,1.0,1.0,1.0,1,1019,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q070,createIssue,29,10,2752,0.0,0.0,0.0,6,2680,0.0,0.0,0.0,3,2626,0.0,0.0,1.0 +query_rewrite_weighted_bm25,q071,list-directory,2,1,182,1.0,1.0,1.0,1,296,1.0,1.0,1.0,1,607,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q072,sequential thinking next thought,1,1,224,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,797,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q073,What time is it in Tokyo,1,1,217,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,783,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q074,Convert time between timezones,1,1,204,1.0,1.0,1.0,1,335,1.0,1.0,1.0,1,704,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q075,Star a GitHub repo,1,1,237,1.0,1.0,1.0,1,388,1.0,1.0,1.0,1,810,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q076,Fork a repository,1,1,235,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,792,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q077,Get the latest GitHub release,1,1,239,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,667,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q078,Create a gist,1,1,229,1.0,1.0,1.0,1,429,1.0,1.0,1.0,1,912,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q079,Search GitHub code,1,1,283,1.0,1.0,1.0,1,419,1.0,1.0,1.0,1,773,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q080,List GitHub notifications,1,1,204,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,764,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q081,Assign Copilot to an issue,1,1,267,1.0,1.0,1.0,1,434,1.0,1.0,1.0,1,928,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q082,Get secret scanning alerts,1,1,295,1.0,1.0,1.0,1,470,1.0,1.0,1.0,1,830,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q083,Create or update a file on GitHub,1,1,390,1.0,1.0,1.0,1,557,1.0,1.0,1.0,1,922,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q084,List collaborators on a repo,1,1,248,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,813,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q085,Reply in a Slack thread,1,1,254,1.0,1.0,1.0,1,358,1.0,1.0,1.0,1,717,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q086,Add an emoji reaction on Slack,1,1,216,1.0,1.0,1.0,1,357,1.0,1.0,1.0,1,709,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q087,Search Slack messages,1,1,213,1.0,1.0,1.0,1,337,1.0,1.0,1.0,1,689,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q088,Create a Slack channel,3,1,196,1.0,1.0,1.0,1,342,1.0,1.0,1.0,1,719,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q089,Transition a Jira issue,1,1,208,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,763,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q090,Search Jira with JQL,1,1,194,1.0,1.0,1.0,1,370,1.0,1.0,1.0,1,764,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q091,Add a worklog to Jira,1,1,249,1.0,1.0,1.0,1,350,1.0,1.0,1.0,1,765,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q092,List Linear teams,1,1,184,1.0,1.0,1.0,1,380,1.0,1.0,1.0,1,764,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q093,Comment on a Linear issue,1,1,334,1.0,1.0,1.0,1,464,1.0,1.0,1.0,1,760,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q094,Move a local file,1,1,214,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,699,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q095,Recursively search files by pattern,1,1,280,1.0,1.0,1.0,1,433,1.0,1.0,1.0,1,705,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q096,Edit a file in place with old and new string,1,1,227,1.0,1.0,1.0,1,394,1.0,1.0,1.0,1,682,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q097,Read the whole knowledge graph,1,1,152,1.0,1.0,1.0,1,259,1.0,1.0,1.0,1,529,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q098,Search memory nodes,1,1,202,1.0,1.0,1.0,1,301,1.0,1.0,1.0,1,546,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q099,Show unstaged git diff,1,1,190,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,602,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q100,Stage files with git add,3,1,213,1.0,1.0,1.0,1,336,1.0,1.0,1.0,1,677,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q101,Create a Shopify product,1,1,240,1.0,1.0,1.0,1,380,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q102,Cancel a Stripe subscription,1,1,188,1.0,1.0,1.0,1,301,1.0,1.0,1.0,1,632,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q103,Create a payment intent,1,1,234,1.0,1.0,1.0,1,417,1.0,1.0,1.0,1,897,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q104,Acknowledge a PagerDuty incident,1,1,261,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,717,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q105,Create a Datadog monitor,2,1,259,1.0,1.0,1.0,1,459,1.0,1.0,1.0,1,927,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q106,Resolve a Sentry issue,2,1,251,1.0,1.0,1.0,1,397,1.0,1.0,1.0,1,817,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q107,Scale a Kubernetes deployment,1,1,263,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q108,Apply a Kubernetes manifest,1,1,253,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q109,Stop a docker container,1,1,188,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,641,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q110,Build a docker image,1,1,224,1.0,1.0,1.0,1,317,1.0,1.0,1.0,1,601,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q111,Click an element in Playwright,1,1,216,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,566,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q112,Type text into a browser input,1,1,217,1.0,1.0,1.0,1,302,1.0,1.0,1.0,1,554,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q113,List GitLab projects,1,1,234,1.0,1.0,1.0,1,398,1.0,1.0,1.0,1,817,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q114,Create a Bitbucket pull request,1,1,313,1.0,1.0,1.0,1,471,1.0,1.0,1.0,1,1091,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q115,Create a Gmail draft,1,1,205,1.0,1.0,1.0,1,352,1.0,1.0,1.0,1,835,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q116,Share a Google Drive file,1,1,254,1.0,1.0,1.0,1,417,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q117,List Google Calendar events,1,1,269,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,784,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q118,Delete a calendar event,1,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,751,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q119,List Vercel projects,1,1,177,1.0,1.0,1.0,1,301,1.0,1.0,1.0,1,550,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q120,Select rows from Supabase,1,1,237,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,709,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q121,Post a Figma comment,1,1,204,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q122,List Okta users,1,1,177,1.0,1.0,1.0,1,306,1.0,1.0,1.0,1,606,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q123,Range query Prometheus,1,1,221,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,680,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q124,Create a Grafana annotation,1,1,218,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,881,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q125,Apply terraform,1,1,171,1.0,1.0,1.0,1,292,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q126,List Asana tasks,1,1,244,1.0,1.0,1.0,1,404,1.0,1.0,1.0,1,682,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q127,Comment on a Trello card,1,1,260,1.0,1.0,1.0,1,422,1.0,1.0,1.0,1,757,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q128,Update a ClickUp task,1,1,257,1.0,1.0,1.0,1,399,1.0,1.0,1.0,1,795,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q129,Search Confluence,1,1,221,1.0,1.0,1.0,1,401,1.0,1.0,1.0,1,731,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q130,Create a HubSpot deal,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,832,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q131,List Zendesk tickets,1,1,189,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,645,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q132,List Discord channels,1,1,166,1.0,1.0,1.0,1,321,1.0,1.0,1.0,1,625,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q133,Set a Redis key,1,1,182,1.0,1.0,1.0,1,280,1.0,1.0,1.0,1,600,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q134,Insert MongoDB documents,1,1,256,1.0,1.0,1.0,1,399,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q135,Run a MySQL query,1,1,174,1.0,1.0,1.0,1,289,1.0,1.0,1.0,1,686,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q136,Index a document in Elasticsearch,1,1,242,1.0,1.0,1.0,1,396,1.0,1.0,1.0,1,753,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q137,Get an S3 object,1,1,249,1.0,1.0,1.0,1,355,1.0,1.0,1.0,1,706,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q138,List Cloudflare zones,1,1,182,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,629,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q139,Create a Stripe invoice,1,1,204,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,632,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q140,List GitHub Actions workflows,1,1,293,1.0,1.0,1.0,1,468,1.0,1.0,1.0,1,845,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q141,Get GitHub Actions job logs,1,1,269,1.0,1.0,1.0,1,442,1.0,1.0,1.0,1,822,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q142,Create a GitHub repository,1,1,250,1.0,1.0,1.0,1,388,1.0,1.0,1.0,1,867,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q143,Checkout a git branch,1,1,212,1.0,1.0,1.0,1,373,1.0,1.0,1.0,1,666,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q144,Show git log,1,1,206,1.0,1.0,1.0,1,320,1.0,1.0,1.0,1,600,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q145,List postgres schemas,1,1,158,1.0,1.0,1.0,1,277,1.0,1.0,1.0,1,555,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q146,Open nodes in the knowledge graph,1,1,163,1.0,1.0,1.0,1,260,1.0,1.0,1.0,1,549,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q147,Get file metadata from the filesystem,4,2,462,0.0,1.0,1.0,1,348,1.0,1.0,1.0,1,688,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q148,Write a new local file,2,1,217,1.0,1.0,1.0,1,366,1.0,1.0,1.0,1,714,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q149,Search GitHub issues,1,1,263,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,742,1.0,1.0,1.0 +query_rewrite_weighted_bm25,q150,Get the authenticated GitHub user,9,3,585,0.0,0.0,1.0,2,628,0.0,1.0,1.0,1,606,1.0,1.0,1.0 +tfidf,q001,Create a GitHub issue,2,1,349,1.0,1.0,1.0,1,527,1.0,1.0,1.0,1,947,1.0,1.0,1.0 +tfidf,q002,Create a Jira ticket,2,1,283,1.0,1.0,1.0,1,466,1.0,1.0,1.0,1,951,1.0,1.0,1.0 +tfidf,q003,Send a Slack message,3,1,250,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,755,1.0,1.0,1.0 +tfidf,q004,List open pull requests,1,1,234,1.0,1.0,1.0,1,460,1.0,1.0,1.0,1,883,1.0,1.0,1.0 +tfidf,q005,Open a pull request on GitHub,1,1,364,1.0,1.0,1.0,1,611,1.0,1.0,1.0,1,1040,1.0,1.0,1.0 +tfidf,q006,Run a SQL query against postgres,1,1,180,1.0,1.0,1.0,1,295,1.0,1.0,1.0,1,553,1.0,1.0,1.0 +tfidf,q007,Read a local file from disk,1,1,223,1.0,1.0,1.0,1,428,1.0,1.0,1.0,1,780,1.0,1.0,1.0 +tfidf,q008,What files are in this folder,8,3,712,0.0,0.0,1.0,2,745,0.0,1.0,1.0,1,723,1.0,1.0,1.0 +tfidf,q009,Remember this fact in the knowledge graph,2,1,154,1.0,1.0,1.0,1,262,1.0,1.0,1.0,1,544,1.0,1.0,1.0 +tfidf,q010,Search the web with Brave,1,1,258,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,710,1.0,1.0,1.0 +tfidf,q011,Fetch the contents of a URL,1,1,208,1.0,1.0,1.0,1,372,1.0,1.0,1.0,1,707,1.0,1.0,1.0 +tfidf,q012,Show git status,2,1,183,1.0,1.0,1.0,1,299,1.0,1.0,1.0,1,645,1.0,1.0,1.0 +tfidf,q013,Commit staged changes,1,1,182,1.0,1.0,1.0,1,327,1.0,1.0,1.0,1,663,1.0,1.0,1.0 +tfidf,q014,Create a Linear issue,4,2,574,0.0,1.0,1.0,1,463,1.0,1.0,1.0,1,958,1.0,1.0,1.0 +tfidf,q015,Make a task in my project tracker Linear,145,49,12448,0.0,0.0,0.0,29,11769,0.0,0.0,0.0,15,11810,0.0,0.0,0.0 +tfidf,q016,Post a comment on a GitHub issue,3,1,307,1.0,1.0,1.0,1,444,1.0,1.0,1.0,1,732,1.0,1.0,1.0 +tfidf,q017,Merge the pull request,1,1,283,1.0,1.0,1.0,1,495,1.0,1.0,1.0,1,1121,1.0,1.0,1.0 +tfidf,q018,Trigger a GitHub Actions workflow,1,1,294,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,791,1.0,1.0,1.0 +tfidf,q019,Get Dependabot alerts,1,1,269,1.0,1.0,1.0,1,420,1.0,1.0,1.0,1,684,1.0,1.0,1.0 +tfidf,q020,List Sentry issues,1,1,236,1.0,1.0,1.0,1,410,1.0,1.0,1.0,1,796,1.0,1.0,1.0 +tfidf,q021,Create a Stripe customer,1,1,186,1.0,1.0,1.0,1,300,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +tfidf,q022,Refund a Stripe payment,1,1,234,1.0,1.0,1.0,1,370,1.0,1.0,1.0,1,647,1.0,1.0,1.0 +tfidf,q023,Send an email with Gmail,1,1,224,1.0,1.0,1.0,1,344,1.0,1.0,1.0,1,711,1.0,1.0,1.0 +tfidf,q024,Search my Gmail inbox,1,1,178,1.0,1.0,1.0,1,315,1.0,1.0,1.0,1,652,1.0,1.0,1.0 +tfidf,q025,Upload a file to Google Drive,4,2,523,0.0,1.0,1.0,1,415,1.0,1.0,1.0,1,750,1.0,1.0,1.0 +tfidf,q026,Create a calendar event,3,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,962,1.0,1.0,1.0 +tfidf,q027,List Kubernetes pods,1,1,184,1.0,1.0,1.0,1,324,1.0,1.0,1.0,1,699,1.0,1.0,1.0 +tfidf,q028,Get logs from a k8s pod,1,1,299,1.0,1.0,1.0,1,455,1.0,1.0,1.0,1,871,1.0,1.0,1.0 +tfidf,q029,Run a docker container,1,1,237,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,597,1.0,1.0,1.0 +tfidf,q030,Show docker container logs,1,1,188,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,645,1.0,1.0,1.0 +tfidf,q031,Navigate the browser to a URL,1,1,180,1.0,1.0,1.0,1,294,1.0,1.0,1.0,1,596,1.0,1.0,1.0 +tfidf,q032,Take a screenshot of the page,1,1,226,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,678,1.0,1.0,1.0 +tfidf,q033,Create a GitLab merge request,3,1,297,1.0,1.0,1.0,1,475,1.0,1.0,1.0,1,939,1.0,1.0,1.0 +tfidf,q034,Search Notion pages,1,1,180,1.0,1.0,1.0,1,283,1.0,1.0,1.0,1,672,1.0,1.0,1.0 +tfidf,q035,Create a Notion page,3,1,230,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,703,1.0,1.0,1.0 +tfidf,q036,Query a Notion database,1,1,279,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,709,1.0,1.0,1.0 +tfidf,q037,Create a Zendesk ticket,3,1,257,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,875,1.0,1.0,1.0 +tfidf,q038,Create a HubSpot contact,2,1,216,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,899,1.0,1.0,1.0 +tfidf,q039,Create a Salesforce lead,2,1,252,1.0,1.0,1.0,1,416,1.0,1.0,1.0,1,796,1.0,1.0,1.0 +tfidf,q040,Run a SOQL query,1,1,189,1.0,1.0,1.0,1,359,1.0,1.0,1.0,1,662,1.0,1.0,1.0 +tfidf,q041,List Shopify orders,1,1,193,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,662,1.0,1.0,1.0 +tfidf,q042,Create a PagerDuty incident,2,1,259,1.0,1.0,1.0,1,387,1.0,1.0,1.0,1,857,1.0,1.0,1.0 +tfidf,q043,Who is on call in PagerDuty,1,1,237,1.0,1.0,1.0,1,419,1.0,1.0,1.0,1,772,1.0,1.0,1.0 +tfidf,q044,Query Datadog metrics,1,1,300,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,797,1.0,1.0,1.0 +tfidf,q045,Search Datadog logs,1,1,242,1.0,1.0,1.0,1,458,1.0,1.0,1.0,1,824,1.0,1.0,1.0 +tfidf,q046,List S3 buckets,1,1,170,1.0,1.0,1.0,1,312,1.0,1.0,1.0,1,650,1.0,1.0,1.0 +tfidf,q047,Upload an object to S3,1,1,250,1.0,1.0,1.0,1,437,1.0,1.0,1.0,1,905,1.0,1.0,1.0 +tfidf,q048,Purge Cloudflare cache,1,1,231,1.0,1.0,1.0,1,340,1.0,1.0,1.0,1,746,1.0,1.0,1.0 +tfidf,q049,Create a Cloudflare DNS record,1,1,249,1.0,1.0,1.0,1,360,1.0,1.0,1.0,1,757,1.0,1.0,1.0 +tfidf,q050,List Vercel deployments,1,1,173,1.0,1.0,1.0,1,291,1.0,1.0,1.0,1,618,1.0,1.0,1.0 +tfidf,q051,Insert a row into Supabase,1,1,203,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,665,1.0,1.0,1.0 +tfidf,q052,Export images from Figma,1,1,196,1.0,1.0,1.0,1,328,1.0,1.0,1.0,1,777,1.0,1.0,1.0 +tfidf,q053,Send an SMS with Twilio,1,1,254,1.0,1.0,1.0,1,409,1.0,1.0,1.0,1,763,1.0,1.0,1.0 +tfidf,q054,Create an Okta user,2,1,183,1.0,1.0,1.0,1,329,1.0,1.0,1.0,1,683,1.0,1.0,1.0 +tfidf,q055,Instant query Prometheus,1,1,221,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,679,1.0,1.0,1.0 +tfidf,q056,Search Grafana dashboards,1,1,217,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,635,1.0,1.0,1.0 +tfidf,q057,Trigger a CircleCI pipeline,1,1,216,1.0,1.0,1.0,1,324,1.0,1.0,1.0,1,755,1.0,1.0,1.0 +tfidf,q058,Run terraform plan,1,1,172,1.0,1.0,1.0,1,280,1.0,1.0,1.0,1,684,1.0,1.0,1.0 +tfidf,q059,Create an Asana task,2,1,267,1.0,1.0,1.0,1,405,1.0,1.0,1.0,1,825,1.0,1.0,1.0 +tfidf,q060,Create a Trello card,1,1,240,1.0,1.0,1.0,1,413,1.0,1.0,1.0,1,903,1.0,1.0,1.0 +tfidf,q061,Create a ClickUp task,1,1,257,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,770,1.0,1.0,1.0 +tfidf,q062,Create a Confluence page,3,1,260,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,755,1.0,1.0,1.0 +tfidf,q063,Send a Discord message,1,1,224,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,740,1.0,1.0,1.0 +tfidf,q064,Search Elasticsearch,1,1,239,1.0,1.0,1.0,1,375,1.0,1.0,1.0,1,657,1.0,1.0,1.0 +tfidf,q065,Get a Redis key,1,1,182,1.0,1.0,1.0,1,281,1.0,1.0,1.0,1,579,1.0,1.0,1.0 +tfidf,q066,Find MongoDB documents,1,1,258,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,738,1.0,1.0,1.0 +tfidf,q067,List SQLite tables,1,1,147,1.0,1.0,1.0,1,245,1.0,1.0,1.0,1,543,1.0,1.0,1.0 +tfidf,q068,Describe a postgres table,1,1,215,1.0,1.0,1.0,1,349,1.0,1.0,1.0,1,597,1.0,1.0,1.0 +tfidf,q069,github_create_issue,2,1,315,1.0,1.0,1.0,1,472,1.0,1.0,1.0,1,900,1.0,1.0,1.0 +tfidf,q070,createIssue,2,1,314,1.0,1.0,1.0,1,433,1.0,1.0,1.0,1,864,1.0,1.0,1.0 +tfidf,q071,list-directory,2,1,234,1.0,1.0,1.0,1,348,1.0,1.0,1.0,1,654,1.0,1.0,1.0 +tfidf,q072,sequential thinking next thought,1,1,224,1.0,1.0,1.0,1,382,1.0,1.0,1.0,1,797,1.0,1.0,1.0 +tfidf,q073,What time is it in Tokyo,1,1,222,1.0,1.0,1.0,1,378,1.0,1.0,1.0,1,699,1.0,1.0,1.0 +tfidf,q074,Convert time between timezones,1,1,204,1.0,1.0,1.0,1,360,1.0,1.0,1.0,1,704,1.0,1.0,1.0 +tfidf,q075,Star a GitHub repo,1,1,201,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,780,1.0,1.0,1.0 +tfidf,q076,Fork a repository,1,1,248,1.0,1.0,1.0,1,386,1.0,1.0,1.0,1,825,1.0,1.0,1.0 +tfidf,q077,Get the latest GitHub release,1,1,201,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,753,1.0,1.0,1.0 +tfidf,q078,Create a gist,3,1,229,1.0,1.0,1.0,1,421,1.0,1.0,1.0,1,911,1.0,1.0,1.0 +tfidf,q079,Search GitHub code,1,1,283,1.0,1.0,1.0,1,419,1.0,1.0,1.0,1,725,1.0,1.0,1.0 +tfidf,q080,List GitHub notifications,1,1,192,1.0,1.0,1.0,1,328,1.0,1.0,1.0,1,687,1.0,1.0,1.0 +tfidf,q081,Assign Copilot to an issue,1,1,267,1.0,1.0,1.0,1,434,1.0,1.0,1.0,1,879,1.0,1.0,1.0 +tfidf,q082,Get secret scanning alerts,1,1,295,1.0,1.0,1.0,1,470,1.0,1.0,1.0,1,733,1.0,1.0,1.0 +tfidf,q083,Create or update a file on GitHub,1,1,333,1.0,1.0,1.0,1,545,1.0,1.0,1.0,1,957,1.0,1.0,1.0 +tfidf,q084,List collaborators on a repo,1,1,307,1.0,1.0,1.0,1,462,1.0,1.0,1.0,1,948,1.0,1.0,1.0 +tfidf,q085,Reply in a Slack thread,1,1,240,1.0,1.0,1.0,1,346,1.0,1.0,1.0,1,694,1.0,1.0,1.0 +tfidf,q086,Add an emoji reaction on Slack,1,1,268,1.0,1.0,1.0,1,438,1.0,1.0,1.0,1,802,1.0,1.0,1.0 +tfidf,q087,Search Slack messages,1,1,211,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,633,1.0,1.0,1.0 +tfidf,q088,Create a Slack channel,1,1,196,1.0,1.0,1.0,1,339,1.0,1.0,1.0,1,718,1.0,1.0,1.0 +tfidf,q089,Transition a Jira issue,1,1,207,1.0,1.0,1.0,1,342,1.0,1.0,1.0,1,816,1.0,1.0,1.0 +tfidf,q090,Search Jira with JQL,1,1,202,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,602,1.0,1.0,1.0 +tfidf,q091,Add a worklog to Jira,1,1,285,1.0,1.0,1.0,1,417,1.0,1.0,1.0,1,839,1.0,1.0,1.0 +tfidf,q092,List Linear teams,1,1,137,1.0,1.0,1.0,1,283,1.0,1.0,1.0,1,591,1.0,1.0,1.0 +tfidf,q093,Comment on a Linear issue,1,1,244,1.0,1.0,1.0,1,360,1.0,1.0,1.0,1,798,1.0,1.0,1.0 +tfidf,q094,Move a local file,1,1,224,1.0,1.0,1.0,1,335,1.0,1.0,1.0,1,684,1.0,1.0,1.0 +tfidf,q095,Recursively search files by pattern,1,1,212,1.0,1.0,1.0,1,322,1.0,1.0,1.0,1,716,1.0,1.0,1.0 +tfidf,q096,Edit a file in place with old and new string,1,1,247,1.0,1.0,1.0,1,353,1.0,1.0,1.0,1,735,1.0,1.0,1.0 +tfidf,q097,Read the whole knowledge graph,1,1,152,1.0,1.0,1.0,1,260,1.0,1.0,1.0,1,561,1.0,1.0,1.0 +tfidf,q098,Search memory nodes,1,1,202,1.0,1.0,1.0,1,338,1.0,1.0,1.0,1,620,1.0,1.0,1.0 +tfidf,q099,Show unstaged git diff,1,1,190,1.0,1.0,1.0,1,303,1.0,1.0,1.0,1,621,1.0,1.0,1.0 +tfidf,q100,Stage files with git add,1,1,220,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,774,1.0,1.0,1.0 +tfidf,q101,Create a Shopify product,3,1,240,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,702,1.0,1.0,1.0 +tfidf,q102,Cancel a Stripe subscription,1,1,206,1.0,1.0,1.0,1,321,1.0,1.0,1.0,1,610,1.0,1.0,1.0 +tfidf,q103,Create a payment intent,2,1,234,1.0,1.0,1.0,1,426,1.0,1.0,1.0,1,916,1.0,1.0,1.0 +tfidf,q104,Acknowledge a PagerDuty incident,1,1,261,1.0,1.0,1.0,1,389,1.0,1.0,1.0,1,717,1.0,1.0,1.0 +tfidf,q105,Create a Datadog monitor,2,1,259,1.0,1.0,1.0,1,459,1.0,1.0,1.0,1,948,1.0,1.0,1.0 +tfidf,q106,Resolve a Sentry issue,2,1,251,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,745,1.0,1.0,1.0 +tfidf,q107,Scale a Kubernetes deployment,1,1,230,1.0,1.0,1.0,1,374,1.0,1.0,1.0,1,726,1.0,1.0,1.0 +tfidf,q108,Apply a Kubernetes manifest,1,1,200,1.0,1.0,1.0,1,344,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +tfidf,q109,Stop a docker container,1,1,188,1.0,1.0,1.0,1,345,1.0,1.0,1.0,1,641,1.0,1.0,1.0 +tfidf,q110,Build a docker image,1,1,211,1.0,1.0,1.0,1,317,1.0,1.0,1.0,1,601,1.0,1.0,1.0 +tfidf,q111,Click an element in Playwright,1,1,216,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,654,1.0,1.0,1.0 +tfidf,q112,Type text into a browser input,1,1,232,1.0,1.0,1.0,1,337,1.0,1.0,1.0,1,647,1.0,1.0,1.0 +tfidf,q113,List GitLab projects,1,1,171,1.0,1.0,1.0,1,255,1.0,1.0,1.0,1,609,1.0,1.0,1.0 +tfidf,q114,Create a Bitbucket pull request,1,1,313,1.0,1.0,1.0,1,456,1.0,1.0,1.0,1,1091,1.0,1.0,1.0 +tfidf,q115,Create a Gmail draft,1,1,188,1.0,1.0,1.0,1,315,1.0,1.0,1.0,1,844,1.0,1.0,1.0 +tfidf,q116,Share a Google Drive file,1,1,254,1.0,1.0,1.0,1,417,1.0,1.0,1.0,1,684,1.0,1.0,1.0 +tfidf,q117,List Google Calendar events,1,1,269,1.0,1.0,1.0,1,441,1.0,1.0,1.0,1,753,1.0,1.0,1.0 +tfidf,q118,Delete a calendar event,1,1,300,1.0,1.0,1.0,1,427,1.0,1.0,1.0,1,796,1.0,1.0,1.0 +tfidf,q119,List Vercel projects,1,1,175,1.0,1.0,1.0,1,269,1.0,1.0,1.0,1,550,1.0,1.0,1.0 +tfidf,q120,Select rows from Supabase,1,1,237,1.0,1.0,1.0,1,354,1.0,1.0,1.0,1,815,1.0,1.0,1.0 +tfidf,q121,Post a Figma comment,1,1,218,1.0,1.0,1.0,1,350,1.0,1.0,1.0,1,658,1.0,1.0,1.0 +tfidf,q122,List Okta users,1,1,177,1.0,1.0,1.0,1,306,1.0,1.0,1.0,1,582,1.0,1.0,1.0 +tfidf,q123,Range query Prometheus,1,1,221,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,679,1.0,1.0,1.0 +tfidf,q124,Create a Grafana annotation,1,1,218,1.0,1.0,1.0,1,410,1.0,1.0,1.0,1,900,1.0,1.0,1.0 +tfidf,q125,Apply terraform,1,1,194,1.0,1.0,1.0,1,292,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +tfidf,q126,List Asana tasks,1,1,211,1.0,1.0,1.0,1,368,1.0,1.0,1.0,1,708,1.0,1.0,1.0 +tfidf,q127,Comment on a Trello card,1,1,241,1.0,1.0,1.0,1,422,1.0,1.0,1.0,1,695,1.0,1.0,1.0 +tfidf,q128,Update a ClickUp task,1,1,257,1.0,1.0,1.0,1,399,1.0,1.0,1.0,1,740,1.0,1.0,1.0 +tfidf,q129,Search Confluence,1,1,221,1.0,1.0,1.0,1,362,1.0,1.0,1.0,1,677,1.0,1.0,1.0 +tfidf,q130,Create a HubSpot deal,1,1,255,1.0,1.0,1.0,1,364,1.0,1.0,1.0,1,899,1.0,1.0,1.0 +tfidf,q131,List Zendesk tickets,1,1,189,1.0,1.0,1.0,1,367,1.0,1.0,1.0,1,694,1.0,1.0,1.0 +tfidf,q132,List Discord channels,1,1,184,1.0,1.0,1.0,1,300,1.0,1.0,1.0,1,639,1.0,1.0,1.0 +tfidf,q133,Set a Redis key,1,1,182,1.0,1.0,1.0,1,302,1.0,1.0,1.0,1,595,1.0,1.0,1.0 +tfidf,q134,Insert MongoDB documents,1,1,256,1.0,1.0,1.0,1,381,1.0,1.0,1.0,1,698,1.0,1.0,1.0 +tfidf,q135,Run a MySQL query,1,1,179,1.0,1.0,1.0,1,281,1.0,1.0,1.0,1,619,1.0,1.0,1.0 +tfidf,q136,Index a document in Elasticsearch,1,1,242,1.0,1.0,1.0,1,344,1.0,1.0,1.0,1,621,1.0,1.0,1.0 +tfidf,q137,Get an S3 object,1,1,249,1.0,1.0,1.0,1,355,1.0,1.0,1.0,1,729,1.0,1.0,1.0 +tfidf,q138,List Cloudflare zones,1,1,215,1.0,1.0,1.0,1,319,1.0,1.0,1.0,1,685,1.0,1.0,1.0 +tfidf,q139,Create a Stripe invoice,1,1,204,1.0,1.0,1.0,1,319,1.0,1.0,1.0,1,669,1.0,1.0,1.0 +tfidf,q140,List GitHub Actions workflows,1,1,268,1.0,1.0,1.0,1,456,1.0,1.0,1.0,1,812,1.0,1.0,1.0 +tfidf,q141,Get GitHub Actions job logs,1,1,237,1.0,1.0,1.0,1,377,1.0,1.0,1.0,1,859,1.0,1.0,1.0 +tfidf,q142,Create a GitHub repository,2,1,268,1.0,1.0,1.0,1,406,1.0,1.0,1.0,1,837,1.0,1.0,1.0 +tfidf,q143,Checkout a git branch,1,1,266,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,1001,1.0,1.0,1.0 +tfidf,q144,Show git log,1,1,194,1.0,1.0,1.0,1,299,1.0,1.0,1.0,1,619,1.0,1.0,1.0 +tfidf,q145,List postgres schemas,1,1,158,1.0,1.0,1.0,1,277,1.0,1.0,1.0,1,604,1.0,1.0,1.0 +tfidf,q146,Open nodes in the knowledge graph,1,1,163,1.0,1.0,1.0,1,260,1.0,1.0,1.0,1,531,1.0,1.0,1.0 +tfidf,q147,Get file metadata from the filesystem,1,1,190,1.0,1.0,1.0,1,334,1.0,1.0,1.0,1,769,1.0,1.0,1.0 +tfidf,q148,Write a new local file,1,1,234,1.0,1.0,1.0,1,428,1.0,1.0,1.0,1,727,1.0,1.0,1.0 +tfidf,q149,Search GitHub issues,1,1,263,1.0,1.0,1.0,1,407,1.0,1.0,1.0,1,742,1.0,1.0,1.0 +tfidf,q150,Get the authenticated GitHub user,3,1,179,1.0,1.0,1.0,1,281,1.0,1.0,1.0,1,606,1.0,1.0,1.0 diff --git a/experiments/mcp-tool-search/results/cost_per_query.json b/experiments/mcp-tool-search/results/cost_per_query.json new file mode 100644 index 00000000..6eb7df41 --- /dev/null +++ b/experiments/mcp-tool-search/results/cost_per_query.json @@ -0,0 +1,103562 @@ +{ + "bm25_desc_only": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 380, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 584, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1011, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 281, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 481, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 966, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 771, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 423, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 379, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 612, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1163, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 671, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 425, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 832, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 413, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 727, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 154, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 261, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 537, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 430, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 723, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 346, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 305, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 357, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 498, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 956, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "hit_rank": 146, + "k3_calls": 49, + "k3_tokens": 12370, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 30, + "k5_tokens": 12110, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11750, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 307, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 482, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 887, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 321, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 524, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1103, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 831, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 287, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 420, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 739, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 395, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 395, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 305, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 666, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 555, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 446, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 747, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 461, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 813, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 741, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 423, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 822, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 594, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 293, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 568, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 326, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 833, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 376, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 822, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 279, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 430, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 752, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 426, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 791, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 838, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 773, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 809, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 419, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 863, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 263, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 490, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 423, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 788, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 732, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 173, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 527, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 357, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 743, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 735, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 415, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 745, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 692, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 766, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 285, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 610, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 156, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 243, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 494, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "hit_rank": 39, + "k3_calls": 13, + "k3_tokens": 3683, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 8, + "k5_tokens": 3724, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3648, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q070", + "query": "createIssue", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 332, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 579, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1039, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "hit_rank": 90, + "k3_calls": 30, + "k3_tokens": 7901, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7685, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7523, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "hit_rank": 116, + "k3_calls": 39, + "k3_tokens": 9799, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 24, + "k5_tokens": 9727, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9475, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 636, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 394, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 772, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 822, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 291, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 383, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 383, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 419, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 326, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 421, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 819, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 804, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 369, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 610, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1088, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 411, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 767, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 461, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 827, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 773, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 222, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 822, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 734, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 440, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 874, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 318, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 264, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 451, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 434, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 712, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 741, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 419, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 259, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 561, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 241, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 313, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 628, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 495, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 710, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 815, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 826, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 411, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 581, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 636, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 209, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 289, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 615, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 519, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1050, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 306, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 808, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 317, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 441, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 784, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 477, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 787, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 248, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 559, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 720, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 174, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 618, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 372, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 722, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 659, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 438, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 787, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 785, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 838, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 603, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 590, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 307, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 564, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 670, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 723, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 361, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 554, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 453, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 812, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 442, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 839, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 920, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "hit_rank": 7, + "k3_calls": 3, + "k3_tokens": 766, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 852, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 832, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 382, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 311, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 626, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 258, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 485, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 260, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 505, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 447, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 750, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 376, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 803, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 683, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "bm25_ident_split": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 380, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 584, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1035, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 281, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 466, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 951, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 760, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 460, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 836, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 364, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 612, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1144, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 592, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 809, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "hit_rank": 9, + "k3_calls": 3, + "k3_tokens": 717, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 756, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 734, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 154, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 261, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 507, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 410, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 613, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 917, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "hit_rank": 146, + "k3_calls": 49, + "k3_tokens": 12414, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 30, + "k5_tokens": 12154, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11794, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 307, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 478, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 922, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 468, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1121, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 768, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 715, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 395, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 691, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 523, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 446, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 801, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 830, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 649, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 346, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 617, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 608, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1031, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 671, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 822, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 668, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 899, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 710, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 332, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 696, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 855, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 258, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 474, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 850, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 708, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 173, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 529, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 203, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 347, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 637, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 622, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 679, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 415, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 672, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 589, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 156, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 243, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 496, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 378, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 568, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1091, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 332, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 579, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 971, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 607, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 225, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 771, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 792, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 383, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 649, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 311, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 833, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 369, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 546, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1001, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 278, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 433, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 932, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 271, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 811, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 664, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 778, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 241, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 619, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 236, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 874, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 137, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 646, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 259, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 561, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 635, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 313, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 602, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 197, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 872, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 167, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 285, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 610, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 654, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 308, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 612, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 648, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 255, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 609, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 546, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1139, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 735, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 248, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 646, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 271, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 638, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 642, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 422, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 761, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 691, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 899, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 605, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 592, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 611, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 346, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 605, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 668, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 579, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 457, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 790, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 442, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 786, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 406, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 874, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 266, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1001, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 160, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 515, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 260, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 531, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 220, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 766, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 791, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 233, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 683, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 304, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 622, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "bm25_name_only": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 369, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 575, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 911, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 481, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 808, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 434, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1009, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 405, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 560, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 978, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 398, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 798, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "hit_rank": 85, + "k3_calls": 29, + "k3_tokens": 7932, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 17, + "k5_tokens": 7554, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7648, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "hit_rank": 90, + "k3_calls": 30, + "k3_tokens": 8091, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7815, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7608, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 325, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 310, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 322, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 528, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 974, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "hit_rank": 146, + "k3_calls": 49, + "k3_tokens": 12480, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 30, + "k5_tokens": 12220, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11860, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 307, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 732, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 354, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 586, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1108, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 719, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 287, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 421, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 801, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 289, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 425, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 716, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "hit_rank": 254, + "k3_calls": 85, + "k3_tokens": 20407, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 51, + "k5_tokens": 19659, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 26, + "k10_tokens": 19548, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 480, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 844, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 393, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 750, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 803, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 241, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 648, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 169, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 309, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 756, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 323, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 552, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 881, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 351, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 748, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 813, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 476, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 813, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 784, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 408, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 818, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "hit_rank": 244, + "k3_calls": 82, + "k3_tokens": 19786, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 49, + "k5_tokens": 18956, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 25, + "k10_tokens": 18737, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 225, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 619, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 280, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 444, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 871, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 236, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 735, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 839, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 756, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 415, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 879, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 173, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 710, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 679, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 325, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 651, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 631, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 365, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 704, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 376, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 685, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 308, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 430, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 701, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 275, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 430, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 712, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 821, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 809, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 496, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 800, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 835, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 664, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "hit_rank": 290, + "k3_calls": 97, + "k3_tokens": 22834, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 58, + "k5_tokens": 22047, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 29, + "k10_tokens": 21525, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 628, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 187, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 398, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 580, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1139, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 397, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 579, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1138, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "hit_rank": 116, + "k3_calls": 39, + "k3_tokens": 9799, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 24, + "k5_tokens": 9727, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9475, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 309, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 731, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 732, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 612, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 248, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 800, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 612, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 175, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 457, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1083, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 818, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 508, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 861, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 274, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 773, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 414, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 426, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 792, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 685, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 270, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 334, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 914, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 176, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 334, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 464, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 885, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 271, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 398, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 760, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 280, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 772, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 398, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 387, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 642, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 361, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 670, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 604, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 477, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 814, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 787, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 904, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 935, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 307, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 174, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 581, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 332, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 581, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 298, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 720, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 236, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 598, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 326, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 268, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 573, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "hit_rank": 7, + "k3_calls": 3, + "k3_tokens": 1123, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 1177, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1155, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 265, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 814, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 278, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 440, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 771, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 233, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 408, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 789, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 798, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 263, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 576, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 711, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 209, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 398, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 762, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 305, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 264, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 423, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 788, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 712, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 292, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 502, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 856, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 786, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 603, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 408, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 734, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 198, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 771, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 735, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 374, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 731, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 710, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 822, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 730, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 293, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 503, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 782, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 959, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 233, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 387, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 690, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "hit_rank": 11, + "k3_calls": 4, + "k3_tokens": 796, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 3, + "k5_tokens": 1039, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 1.0, + "k10_calls": 2, + "k10_tokens": 1422, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "bm25_plain": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 380, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 570, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1093, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 281, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 466, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 951, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 760, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 460, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 836, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 364, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 564, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1180, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 592, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 809, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "hit_rank": 9, + "k3_calls": 3, + "k3_tokens": 717, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 756, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 734, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 154, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 261, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 507, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 410, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 613, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 917, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "hit_rank": 146, + "k3_calls": 49, + "k3_tokens": 12414, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 30, + "k5_tokens": 12154, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11794, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 307, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 442, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 876, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 468, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1121, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 715, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 395, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 691, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 523, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 446, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 801, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 830, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 649, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 346, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 617, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 608, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 975, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 671, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 822, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 668, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 710, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 332, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 696, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 855, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 258, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 474, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 850, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 708, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 173, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 529, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 203, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 347, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 637, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 622, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 679, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 414, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 719, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 672, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 589, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 142, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 240, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 496, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 378, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 568, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1091, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "hit_rank": 29, + "k3_calls": 10, + "k3_tokens": 2752, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 6, + "k5_tokens": 2680, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 3, + "k10_tokens": 2626, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 607, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 394, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 779, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 792, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 361, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 712, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 649, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 833, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 306, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 481, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 889, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 278, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 433, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 932, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 271, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 811, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 664, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 778, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 241, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 619, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 236, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 874, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 137, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 646, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 259, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 561, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 635, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 313, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 602, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 197, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 872, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 167, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 285, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 610, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 654, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 308, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 612, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 648, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 143, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 240, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 623, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 546, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1139, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 735, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 248, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 646, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 271, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 638, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 642, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 422, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 398, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 789, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 691, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 605, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 592, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 181, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 286, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 608, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 346, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 605, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 668, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 579, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 468, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 845, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 421, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 822, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 406, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 949, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 266, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1001, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 160, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 515, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 260, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 531, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 220, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 766, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 791, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 233, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 700, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 288, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 569, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "bm25_tfidf_hybrid": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 349, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 521, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 972, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 281, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 466, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 951, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 460, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 836, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 364, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 611, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1144, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 553, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 780, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "hit_rank": 8, + "k3_calls": 3, + "k3_tokens": 712, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 745, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 723, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 154, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 544, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 710, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 372, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 640, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 574, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 921, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "hit_rank": 145, + "k3_calls": 49, + "k3_tokens": 12448, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 29, + "k5_tokens": 11769, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11810, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 307, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 444, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 819, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 495, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1121, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 795, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 420, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 236, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 711, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 652, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 523, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 446, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 735, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 962, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 834, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 596, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 678, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 939, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 286, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 639, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 279, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 875, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 899, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 310, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 857, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 419, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 772, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 804, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 824, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 170, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 616, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 862, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 173, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 618, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 203, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 683, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 636, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 325, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 413, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 903, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 770, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 589, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 147, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 245, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 543, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 347, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 511, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 931, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 332, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 472, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 880, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 654, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 222, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 248, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 201, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 361, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 421, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 911, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 419, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 434, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 875, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 341, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 545, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1020, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 307, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 445, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 962, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 346, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 467, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 812, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 207, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 605, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 285, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 406, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 839, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 137, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 283, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 639, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 798, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 716, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 735, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 260, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 561, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 621, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 621, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 220, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 610, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 426, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 916, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 948, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 745, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 374, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 654, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 255, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 609, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 508, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1091, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 844, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 175, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 269, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 815, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 306, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 410, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 900, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 708, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 241, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 422, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 695, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 677, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 899, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 639, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 595, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 619, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 607, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 457, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 782, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 822, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 406, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 837, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 266, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1001, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 619, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 158, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 604, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 260, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 531, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 220, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 769, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 752, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 263, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 710, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 284, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "bm25_weighted": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 388, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 598, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 955, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 312, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1035, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 368, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 653, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1052, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 167, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 590, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 782, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 473, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 730, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 708, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 154, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 261, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 524, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 372, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 304, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 340, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 352, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 432, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 769, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 601, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 938, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 523, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1121, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 450, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 287, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 445, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 798, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 496, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 791, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 895, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 834, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 644, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 265, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 536, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 711, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 792, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 279, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 856, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 904, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 787, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 838, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 877, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 318, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 950, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 579, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 656, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 690, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 394, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 884, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 589, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 240, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 514, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 272, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 386, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 596, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1019, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 340, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 546, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 961, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 607, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 783, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 309, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 604, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 792, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 912, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 406, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 697, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 169, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 585, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 434, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 928, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 830, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 347, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 519, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 869, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 278, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 433, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 816, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 357, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 689, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 719, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 334, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 464, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 760, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 280, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 433, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 394, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 259, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 529, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 546, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 602, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 336, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 677, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 897, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 927, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 817, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 263, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 253, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 566, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 554, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 304, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 599, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 471, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1091, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 835, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 784, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 306, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 881, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 422, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 761, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 731, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 904, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 166, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 625, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 268, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 543, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 629, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 433, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 736, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 450, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 747, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 750, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 666, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 158, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 555, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 260, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 549, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 462, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 688, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 270, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 670, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "hit_rank": 11, + "k3_calls": 4, + "k3_tokens": 771, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 3, + "k5_tokens": 1050, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 1.0, + "k10_calls": 2, + "k10_tokens": 1402, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "boolean_and": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 343, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 448, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 870, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 800, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "hit_rank": 121, + "k3_calls": 41, + "k3_tokens": 10292, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 25, + "k5_tokens": 10122, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 13, + "k10_tokens": 10175, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "hit_rank": 37, + "k3_calls": 13, + "k3_tokens": 3710, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 8, + "k5_tokens": 3752, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3672, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 265, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 423, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 838, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 167, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 325, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "hit_rank": 76, + "k3_calls": 26, + "k3_tokens": 7282, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 16, + "k5_tokens": 7224, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 8, + "k10_tokens": 7048, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "hit_rank": 82, + "k3_calls": 28, + "k3_tokens": 7711, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 17, + "k5_tokens": 7530, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7616, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "hit_rank": 89, + "k3_calls": 30, + "k3_tokens": 8108, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7832, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7625, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "hit_rank": 119, + "k3_calls": 40, + "k3_tokens": 10063, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 24, + "k5_tokens": 9727, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9475, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "hit_rank": 103, + "k3_calls": 35, + "k3_tokens": 8982, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 21, + "k5_tokens": 8674, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 11, + "k10_tokens": 8746, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q012", + "query": "Show git status", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 736, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 175, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 748, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 273, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 800, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "hit_rank": 136, + "k3_calls": 46, + "k3_tokens": 11712, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 28, + "k5_tokens": 11391, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 14, + "k10_tokens": 11055, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "hit_rank": 34, + "k3_calls": 12, + "k3_tokens": 3507, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 7, + "k5_tokens": 3237, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3688, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "hit_rank": 41, + "k3_calls": 14, + "k3_tokens": 4072, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 9, + "k5_tokens": 4373, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 5, + "k10_tokens": 4766, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 233, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 806, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "hit_rank": 51, + "k3_calls": 17, + "k3_tokens": 5097, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 11, + "k5_tokens": 5225, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 6, + "k10_tokens": 5482, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 207, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 365, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 780, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 181, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 754, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "hit_rank": 244, + "k3_calls": 82, + "k3_tokens": 19588, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 49, + "k5_tokens": 18841, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 25, + "k10_tokens": 18655, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "hit_rank": 245, + "k3_calls": 82, + "k3_tokens": 19506, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 49, + "k5_tokens": 18792, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 25, + "k10_tokens": 18630, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "hit_rank": 253, + "k3_calls": 85, + "k3_tokens": 20366, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 51, + "k5_tokens": 19618, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 26, + "k10_tokens": 19523, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 805, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 168, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 326, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 741, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 372, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 787, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 336, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "hit_rank": 184, + "k3_calls": 62, + "k3_tokens": 14888, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 37, + "k5_tokens": 14342, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 19, + "k10_tokens": 14250, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 165, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 323, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 346, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 761, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 804, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 181, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 754, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 773, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 198, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 771, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 201, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 359, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 198, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 771, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 220, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 793, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 173, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 750, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 790, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "hit_rank": 243, + "k3_calls": 81, + "k3_tokens": 19511, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 49, + "k5_tokens": 18939, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 25, + "k10_tokens": 18705, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 372, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 787, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 785, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 153, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 311, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 201, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 359, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 756, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 790, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 744, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "hit_rank": 308, + "k3_calls": 103, + "k3_tokens": 24500, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 62, + "k5_tokens": 23675, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 31, + "k10_tokens": 22993, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 201, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 359, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "hit_rank": 316, + "k3_calls": 106, + "k3_tokens": 25217, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 64, + "k5_tokens": 24398, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 32, + "k10_tokens": 23694, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 165, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 323, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 164, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 744, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 756, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 155, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 313, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 376, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 791, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 752, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 176, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 749, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 157, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 730, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 346, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 761, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 147, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 305, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 720, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 176, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 749, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 341, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 446, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 868, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 418, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 600, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 941, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 290, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 712, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "hit_rank": 117, + "k3_calls": 39, + "k3_tokens": 9838, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 24, + "k5_tokens": 9751, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9487, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 191, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 351, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 766, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 347, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 762, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 782, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 750, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 376, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 791, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "hit_rank": 49, + "k3_calls": 17, + "k3_tokens": 5097, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 10, + "k5_tokens": 4866, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 5, + "k10_tokens": 4766, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "hit_rank": 17, + "k3_calls": 6, + "k3_tokens": 1593, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 4, + "k5_tokens": 1694, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 2, + "k10_tokens": 1646, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 191, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "hit_rank": 122, + "k3_calls": 41, + "k3_tokens": 10333, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 25, + "k5_tokens": 10147, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 13, + "k10_tokens": 10188, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "hit_rank": 123, + "k3_calls": 41, + "k3_tokens": 10374, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 25, + "k5_tokens": 10172, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 13, + "k10_tokens": 10201, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 181, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 754, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 145, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "hit_rank": 141, + "k3_calls": 47, + "k3_tokens": 11773, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 29, + "k5_tokens": 11631, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11710, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q094", + "query": "Move a local file", + "hit_rank": 85, + "k3_calls": 29, + "k3_tokens": 7859, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 17, + "k5_tokens": 7496, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7598, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "hit_rank": 86, + "k3_calls": 29, + "k3_tokens": 7917, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7814, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7616, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "hit_rank": 80, + "k3_calls": 27, + "k3_tokens": 7629, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 16, + "k5_tokens": 7288, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 8, + "k10_tokens": 7080, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "hit_rank": 95, + "k3_calls": 32, + "k3_tokens": 8392, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 19, + "k5_tokens": 8062, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 10, + "k10_tokens": 8146, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 168, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 326, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 741, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 167, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 325, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "hit_rank": 109, + "k3_calls": 37, + "k3_tokens": 9343, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 22, + "k5_tokens": 8966, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 11, + "k10_tokens": 8735, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 775, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 162, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 735, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 387, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 802, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "hit_rank": 231, + "k3_calls": 77, + "k3_tokens": 18333, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 47, + "k5_tokens": 17977, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 24, + "k10_tokens": 17954, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 210, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 783, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 752, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 158, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 731, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 750, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "hit_rank": 168, + "k3_calls": 56, + "k3_tokens": 13816, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 34, + "k5_tokens": 13458, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 17, + "k10_tokens": 13084, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "hit_rank": 169, + "k3_calls": 57, + "k3_tokens": 14038, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 34, + "k5_tokens": 13458, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 17, + "k10_tokens": 13084, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 736, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 811, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 351, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 766, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 199, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 357, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 772, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 201, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 359, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 309, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 736, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 369, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 784, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 198, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 771, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 744, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 752, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 165, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 323, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 166, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 739, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 160, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 318, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 166, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 739, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 198, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 771, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 149, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 307, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 722, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 160, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 318, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 346, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 532, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 938, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 174, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 332, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 747, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 173, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 148, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 306, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 164, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "hit_rank": 87, + "k3_calls": 29, + "k3_tokens": 7917, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7814, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7616, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "hit_rank": 79, + "k3_calls": 27, + "k3_tokens": 7494, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 16, + "k5_tokens": 7208, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 8, + "k10_tokens": 7040, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 198, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 771, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 181, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 784, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "boolean_or": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 434, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 614, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1216, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 528, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1058, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 507, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 924, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 334, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 423, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 808, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 364, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 611, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1194, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 604, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 444, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 881, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "hit_rank": 85, + "k3_calls": 29, + "k3_tokens": 7919, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 17, + "k5_tokens": 7541, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7663, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 285, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 506, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 410, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 369, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 671, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 176, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 613, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 351, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 601, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1019, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 353, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 488, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 823, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 338, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 519, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1190, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 370, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 602, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1196, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 442, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 848, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 243, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 792, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 302, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 425, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 809, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 749, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 859, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 446, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 801, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 371, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 588, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1186, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 314, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 481, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 881, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 170, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 304, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 609, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 642, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 552, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1075, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 248, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 387, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 856, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 439, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 749, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 282, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 517, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1115, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 480, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1141, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 477, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1138, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 676, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 922, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 220, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 490, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 800, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 279, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 735, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 768, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 303, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 479, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 954, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 474, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1135, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 173, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 690, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 347, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 292, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 477, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 844, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 836, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 748, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 383, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 681, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 657, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 210, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 908, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 465, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1126, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 415, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1008, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 438, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 934, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 320, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 549, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 981, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 243, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 627, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 156, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 252, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 604, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 272, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 432, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 612, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1214, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 418, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 600, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1107, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 198, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 408, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 241, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 456, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 993, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 298, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 479, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 849, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 311, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 480, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 932, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 383, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 823, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 322, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 554, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1115, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 722, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 792, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 332, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 527, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 864, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 446, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 839, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 389, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 612, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1243, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 274, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 965, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 411, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 304, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 461, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 950, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 685, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 414, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 270, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 914, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 700, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 285, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 495, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 874, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 298, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 534, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 936, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 506, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 907, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 236, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 359, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 360, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 489, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 875, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 154, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 561, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 304, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 615, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 306, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 538, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 836, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 914, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 643, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1120, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 341, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 521, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1160, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 519, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1064, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 253, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 222, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 408, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 236, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 236, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 422, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 772, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 410, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 209, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 439, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 671, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1243, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 396, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 576, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1149, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 271, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 936, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 199, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 481, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 895, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 256, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 607, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 844, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 304, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 528, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 883, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 203, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 383, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 681, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 331, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 563, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1154, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 207, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 271, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 438, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1028, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1022, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 651, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 342, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 522, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1161, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 198, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 750, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 701, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 392, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 639, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 411, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 973, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 731, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 897, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 400, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 795, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 361, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 805, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 346, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 585, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1012, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 330, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 545, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 960, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 187, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 613, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 347, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 284, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 505, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 463, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 750, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 927, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 265, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 420, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "char_trigram": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 380, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 557, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1021, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 318, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 513, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1004, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 653, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 477, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1034, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 327, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 505, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1144, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 592, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 849, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "hit_rank": 91, + "k3_calls": 31, + "k3_tokens": 8311, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 19, + "k5_tokens": 8176, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 10, + "k10_tokens": 8240, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 335, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 255, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 543, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 222, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 634, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 664, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 925, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "hit_rank": 146, + "k3_calls": 49, + "k3_tokens": 12254, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 30, + "k5_tokens": 11994, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11634, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 307, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 468, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 892, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 299, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 859, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 847, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 286, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 671, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 579, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 523, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 786, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 484, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 939, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 441, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 858, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 640, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 298, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 649, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 282, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 574, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 646, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 967, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 693, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 413, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 860, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 730, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 892, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 710, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 592, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 628, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 853, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 780, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 248, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 646, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 187, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 616, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 426, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 830, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 548, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 306, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 710, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 681, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 692, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 719, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 287, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 666, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 325, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 593, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 822, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 411, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 745, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 415, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 802, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 870, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 701, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 630, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 283, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 582, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 240, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 487, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 274, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 697, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 347, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 516, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1032, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 332, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 579, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 982, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 586, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 278, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 434, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 862, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 804, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 778, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 817, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 191, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 719, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 176, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 467, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 883, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 483, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 318, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 469, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 881, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 271, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 413, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 813, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 347, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 621, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 302, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 496, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 919, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 392, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 801, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 627, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 233, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 831, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 422, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 689, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 253, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 574, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 621, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 174, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 614, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 209, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 745, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 203, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 361, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 439, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 931, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 448, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 805, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 848, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 400, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 222, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 655, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 598, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 566, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 233, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 629, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 209, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 609, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 489, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1072, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 431, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 780, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 255, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 530, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 351, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 670, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 201, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 612, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 607, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 347, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 776, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 275, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 432, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 893, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 644, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 631, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 712, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 760, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 369, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 805, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 593, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 598, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 284, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 554, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 369, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 638, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 696, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 711, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 762, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 374, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 406, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 844, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 814, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 311, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 646, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 269, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 525, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 525, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 220, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 635, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 209, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 611, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "codedb_find": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 326, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 521, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1013, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "hit_rank": 7, + "k3_calls": 3, + "k3_tokens": 963, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 1040, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1020, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "hit_rank": 130, + "k3_calls": 44, + "k3_tokens": 11040, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 26, + "k5_tokens": 10501, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 13, + "k10_tokens": 10241, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "hit_rank": 47, + "k3_calls": 16, + "k3_tokens": 4347, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 10, + "k5_tokens": 4459, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 5, + "k10_tokens": 4359, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "hit_rank": 36, + "k3_calls": 12, + "k3_tokens": 3495, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 8, + "k5_tokens": 3776, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3684, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "hit_rank": 103, + "k3_calls": 35, + "k3_tokens": 8997, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 21, + "k5_tokens": 8689, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 11, + "k10_tokens": 8745, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "hit_rank": 77, + "k3_calls": 26, + "k3_tokens": 7264, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 16, + "k5_tokens": 7172, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 8, + "k10_tokens": 6996, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "hit_rank": 82, + "k3_calls": 28, + "k3_tokens": 7711, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 17, + "k5_tokens": 7530, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7616, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "hit_rank": 89, + "k3_calls": 30, + "k3_tokens": 8108, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7832, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7625, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "hit_rank": 124, + "k3_calls": 42, + "k3_tokens": 10467, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 25, + "k5_tokens": 10040, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 13, + "k10_tokens": 10187, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "hit_rank": 110, + "k3_calls": 37, + "k3_tokens": 9458, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 22, + "k5_tokens": 9076, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 11, + "k10_tokens": 8834, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q012", + "query": "Show git status", + "hit_rank": 112, + "k3_calls": 38, + "k3_tokens": 9466, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 23, + "k5_tokens": 9245, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9359, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "hit_rank": 110, + "k3_calls": 37, + "k3_tokens": 9257, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 22, + "k5_tokens": 8907, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 11, + "k10_tokens": 8698, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 303, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 461, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 820, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "hit_rank": 136, + "k3_calls": 46, + "k3_tokens": 11712, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 28, + "k5_tokens": 11391, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 14, + "k10_tokens": 11055, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "hit_rank": 38, + "k3_calls": 13, + "k3_tokens": 3791, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 8, + "k5_tokens": 3759, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3663, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 365, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 560, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 988, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "hit_rank": 62, + "k3_calls": 21, + "k3_tokens": 6123, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 13, + "k5_tokens": 6050, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 7, + "k10_tokens": 6293, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 671, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 716, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 450, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 892, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "hit_rank": 217, + "k3_calls": 73, + "k3_tokens": 17304, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 44, + "k5_tokens": 16814, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 22, + "k10_tokens": 16374, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "hit_rank": 245, + "k3_calls": 82, + "k3_tokens": 19583, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 49, + "k5_tokens": 18819, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 25, + "k10_tokens": 18648, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "hit_rank": 245, + "k3_calls": 82, + "k3_tokens": 19506, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 49, + "k5_tokens": 18792, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 25, + "k10_tokens": 18630, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "hit_rank": 253, + "k3_calls": 85, + "k3_tokens": 20366, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 51, + "k5_tokens": 19618, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 26, + "k10_tokens": 19523, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 324, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 484, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 984, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 198, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 630, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "hit_rank": 193, + "k3_calls": 65, + "k3_tokens": 15818, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 39, + "k5_tokens": 15194, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 20, + "k10_tokens": 15064, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "hit_rank": 194, + "k3_calls": 65, + "k3_tokens": 15640, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 39, + "k5_tokens": 15120, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 20, + "k10_tokens": 15019, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "hit_rank": 184, + "k3_calls": 62, + "k3_tokens": 14888, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 37, + "k5_tokens": 14342, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 19, + "k10_tokens": 14250, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "hit_rank": 171, + "k3_calls": 57, + "k3_tokens": 13984, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 35, + "k5_tokens": 13740, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 18, + "k10_tokens": 13672, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "hit_rank": 171, + "k3_calls": 57, + "k3_tokens": 14038, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 35, + "k5_tokens": 13740, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 18, + "k10_tokens": 13672, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 397, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 592, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 930, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "hit_rank": 165, + "k3_calls": 55, + "k3_tokens": 13537, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 33, + "k5_tokens": 13097, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 17, + "k10_tokens": 13110, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 430, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 773, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 233, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 752, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 756, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 301, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 508, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 832, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 445, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 895, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 398, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 837, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 210, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 692, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 823, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "hit_rank": 243, + "k3_calls": 81, + "k3_tokens": 19511, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 49, + "k5_tokens": 18939, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 25, + "k10_tokens": 18705, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 277, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 835, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "hit_rank": 241, + "k3_calls": 81, + "k3_tokens": 19283, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 49, + "k5_tokens": 18785, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 25, + "k10_tokens": 18646, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 153, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 311, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "hit_rank": 296, + "k3_calls": 99, + "k3_tokens": 23677, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 60, + "k5_tokens": 23014, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 30, + "k10_tokens": 22354, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 844, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 642, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 336, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "hit_rank": 309, + "k3_calls": 103, + "k3_tokens": 24489, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 62, + "k5_tokens": 23669, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 31, + "k10_tokens": 22987, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "hit_rank": 316, + "k3_calls": 106, + "k3_tokens": 25071, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 64, + "k5_tokens": 24328, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 32, + "k10_tokens": 23656, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "hit_rank": 316, + "k3_calls": 106, + "k3_tokens": 25217, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 64, + "k5_tokens": 24398, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 32, + "k10_tokens": 23694, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "hit_rank": 325, + "k3_calls": 109, + "k3_tokens": 25800, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 65, + "k5_tokens": 24768, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 33, + "k10_tokens": 24366, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "hit_rank": 329, + "k3_calls": 110, + "k3_tokens": 25880, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 66, + "k5_tokens": 25000, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 33, + "k10_tokens": 24340, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 225, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "hit_rank": 332, + "k3_calls": 111, + "k3_tokens": 26129, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 67, + "k5_tokens": 25314, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 34, + "k10_tokens": 25005, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 365, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 337, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 467, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 843, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 394, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 895, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 304, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 467, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 854, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 450, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 889, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 752, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 700, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "hit_rank": 291, + "k3_calls": 97, + "k3_tokens": 22977, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 59, + "k5_tokens": 22447, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 30, + "k10_tokens": 22206, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "hit_rank": 281, + "k3_calls": 94, + "k3_tokens": 22397, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 57, + "k5_tokens": 21868, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 29, + "k10_tokens": 21583, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 651, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 677, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 324, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 520, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 998, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 397, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 579, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1039, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 599, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "hit_rank": 116, + "k3_calls": 39, + "k3_tokens": 9799, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 24, + "k5_tokens": 9727, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9475, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "hit_rank": 119, + "k3_calls": 40, + "k3_tokens": 10114, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 24, + "k5_tokens": 9762, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9498, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "hit_rank": 122, + "k3_calls": 41, + "k3_tokens": 10273, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 25, + "k5_tokens": 10083, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 13, + "k10_tokens": 10227, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "hit_rank": 30, + "k3_calls": 10, + "k3_tokens": 2448, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 6, + "k5_tokens": 2372, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 3, + "k10_tokens": 2315, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 810, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 290, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 496, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1021, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 277, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 369, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 783, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 175, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 325, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 643, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 312, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 824, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 465, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 884, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "hit_rank": 20, + "k3_calls": 7, + "k3_tokens": 1850, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 4, + "k5_tokens": 1698, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 2, + "k10_tokens": 1650, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "hit_rank": 33, + "k3_calls": 11, + "k3_tokens": 2773, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 7, + "k5_tokens": 2927, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3392, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "hit_rank": 130, + "k3_calls": 44, + "k3_tokens": 11132, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 26, + "k5_tokens": 10591, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 13, + "k10_tokens": 10318, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "hit_rank": 125, + "k3_calls": 42, + "k3_tokens": 10605, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 25, + "k5_tokens": 10158, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 13, + "k10_tokens": 10233, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 357, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "hit_rank": 151, + "k3_calls": 51, + "k3_tokens": 12663, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 31, + "k5_tokens": 12344, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 16, + "k10_tokens": 12438, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "hit_rank": 155, + "k3_calls": 52, + "k3_tokens": 12978, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 31, + "k5_tokens": 12447, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 16, + "k10_tokens": 12448, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 199, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "hit_rank": 146, + "k3_calls": 49, + "k3_tokens": 12189, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 30, + "k5_tokens": 11992, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11677, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q094", + "query": "Move a local file", + "hit_rank": 95, + "k3_calls": 32, + "k3_tokens": 8648, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 19, + "k5_tokens": 8320, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 10, + "k10_tokens": 8418, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "hit_rank": 88, + "k3_calls": 30, + "k3_tokens": 8094, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7830, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7632, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "hit_rank": 81, + "k3_calls": 27, + "k3_tokens": 7623, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 17, + "k5_tokens": 7586, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7650, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "hit_rank": 105, + "k3_calls": 35, + "k3_tokens": 9234, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 21, + "k5_tokens": 8940, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 11, + "k10_tokens": 9003, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "hit_rank": 105, + "k3_calls": 35, + "k3_tokens": 8947, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 21, + "k5_tokens": 8653, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 11, + "k10_tokens": 8735, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "hit_rank": 111, + "k3_calls": 37, + "k3_tokens": 9333, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 23, + "k5_tokens": 9295, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9410, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 286, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 414, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 820, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 400, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 809, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 299, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 489, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 779, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 296, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 423, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "hit_rank": 232, + "k3_calls": 78, + "k3_tokens": 18609, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 47, + "k5_tokens": 18021, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 24, + "k10_tokens": 17953, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 395, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 804, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "hit_rank": 200, + "k3_calls": 67, + "k3_tokens": 16099, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 40, + "k5_tokens": 15471, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 20, + "k10_tokens": 15071, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "hit_rank": 196, + "k3_calls": 66, + "k3_tokens": 15806, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 40, + "k5_tokens": 15415, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 20, + "k10_tokens": 15015, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "hit_rank": 195, + "k3_calls": 65, + "k3_tokens": 15591, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 39, + "k5_tokens": 15071, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 20, + "k10_tokens": 15055, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "hit_rank": 175, + "k3_calls": 59, + "k3_tokens": 14528, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 35, + "k5_tokens": 13874, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 18, + "k10_tokens": 13763, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "hit_rank": 169, + "k3_calls": 57, + "k3_tokens": 14038, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 34, + "k5_tokens": 13458, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 17, + "k10_tokens": 13084, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 653, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "hit_rank": 272, + "k3_calls": 91, + "k3_tokens": 21988, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 55, + "k5_tokens": 21340, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 28, + "k10_tokens": 20983, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 494, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1037, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 316, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 816, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 301, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 440, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 156, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 614, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "hit_rank": 307, + "k3_calls": 103, + "k3_tokens": 24397, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 62, + "k5_tokens": 23613, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 31, + "k10_tokens": 22962, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 289, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 817, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 233, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 361, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 656, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 266, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 472, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 835, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 197, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 287, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 681, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 395, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 767, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "hit_rank": 345, + "k3_calls": 115, + "k3_tokens": 27183, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 69, + "k5_tokens": 26171, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 35, + "k10_tokens": 25769, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 277, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 840, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 536, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 982, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 644, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 596, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "hit_rank": 293, + "k3_calls": 98, + "k3_tokens": 23335, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 59, + "k5_tokens": 22602, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 30, + "k10_tokens": 22356, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "hit_rank": 283, + "k3_calls": 95, + "k3_tokens": 22628, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 57, + "k5_tokens": 21868, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 29, + "k10_tokens": 21583, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "hit_rank": 280, + "k3_calls": 94, + "k3_tokens": 22516, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 56, + "k5_tokens": 21595, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 28, + "k10_tokens": 21007, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "hit_rank": 291, + "k3_calls": 97, + "k3_tokens": 23125, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 59, + "k5_tokens": 22581, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 30, + "k10_tokens": 22324, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "hit_rank": 295, + "k3_calls": 99, + "k3_tokens": 23578, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 59, + "k5_tokens": 22581, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 30, + "k10_tokens": 22324, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 161, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 613, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 770, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "hit_rank": 60, + "k3_calls": 20, + "k3_tokens": 5782, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 12, + "k5_tokens": 5614, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 6, + "k10_tokens": 5488, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 502, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1026, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "hit_rank": 117, + "k3_calls": 39, + "k3_tokens": 9806, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 24, + "k5_tokens": 9724, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9484, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q144", + "query": "Show git log", + "hit_rank": 119, + "k3_calls": 40, + "k3_tokens": 9928, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 24, + "k5_tokens": 9624, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9396, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 207, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 668, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "hit_rank": 100, + "k3_calls": 34, + "k3_tokens": 8817, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 20, + "k5_tokens": 8407, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 10, + "k10_tokens": 8187, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "hit_rank": 90, + "k3_calls": 30, + "k3_tokens": 8177, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7913, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7715, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "hit_rank": 88, + "k3_calls": 30, + "k3_tokens": 8232, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7980, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7791, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 712, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "hit_rank": 10, + "k3_calls": 4, + "k3_tokens": 1056, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 2, + "k5_tokens": 907, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 885, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "codedb_search": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 335, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 582, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1111, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 481, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 966, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 460, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 836, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 368, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 653, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1144, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 553, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 482, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 712, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 690, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 156, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 493, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 374, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 264, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 672, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 596, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 974, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "hit_rank": 146, + "k3_calls": 49, + "k3_tokens": 12453, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 30, + "k5_tokens": 12193, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11833, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 307, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 510, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1103, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 299, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 495, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1121, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 642, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 395, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 332, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 629, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 716, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 668, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 523, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 446, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 836, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 282, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 603, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 711, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 922, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 558, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 822, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 279, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 426, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 646, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 899, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 661, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 187, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 293, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 628, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 284, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 449, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 767, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 821, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 203, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 361, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 523, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 820, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 286, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 643, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 173, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 449, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 203, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 347, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 598, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 743, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 605, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 664, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 615, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 415, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 654, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 628, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 617, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 626, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 631, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 142, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 240, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 417, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 274, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 543, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "hit_rank": 29, + "k3_calls": 10, + "k3_tokens": 2762, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 6, + "k5_tokens": 2686, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 3, + "k10_tokens": 2629, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "hit_rank": 29, + "k3_calls": 10, + "k3_tokens": 2752, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 6, + "k5_tokens": 2680, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 3, + "k10_tokens": 2626, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "hit_rank": 82, + "k3_calls": 28, + "k3_tokens": 7599, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 17, + "k5_tokens": 7462, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7580, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 222, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 633, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 650, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 376, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 772, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 783, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 653, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 307, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 655, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 875, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 681, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 327, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 532, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1080, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 264, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 776, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 365, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 642, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 271, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 433, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 903, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 609, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 778, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 318, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 599, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 334, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 478, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 878, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 646, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 298, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 886, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 736, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 170, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 305, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 562, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 290, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 546, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 313, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 557, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 519, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 923, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 176, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 275, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 610, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 590, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 626, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 674, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 308, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 587, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 547, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 571, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 589, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 595, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 439, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 634, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1155, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 284, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 411, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 760, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 253, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 697, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 646, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 174, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 618, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 275, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 574, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 238, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 595, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 566, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 438, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 636, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 899, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 529, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 513, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 526, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 607, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 174, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 611, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 538, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 634, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 285, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 482, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 453, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 772, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 442, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 769, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 562, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 927, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1001, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 555, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 160, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 439, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 531, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 209, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 715, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 846, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 233, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 288, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 622, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "codedb_symbol": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 341, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 466, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 802, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "hit_rank": 8, + "k3_calls": 3, + "k3_tokens": 719, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 809, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 789, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 506, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1038, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "hit_rank": 46, + "k3_calls": 16, + "k3_tokens": 4346, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 10, + "k5_tokens": 4440, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 5, + "k10_tokens": 4325, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "hit_rank": 104, + "k3_calls": 35, + "k3_tokens": 9005, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 21, + "k5_tokens": 8697, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 11, + "k10_tokens": 8763, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "hit_rank": 85, + "k3_calls": 29, + "k3_tokens": 7857, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 17, + "k5_tokens": 7449, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7634, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "hit_rank": 7, + "k3_calls": 3, + "k3_tokens": 648, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 691, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "hit_rank": 99, + "k3_calls": 33, + "k3_tokens": 8732, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 20, + "k5_tokens": 8484, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 10, + "k10_tokens": 8254, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 500, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 810, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "hit_rank": 111, + "k3_calls": 37, + "k3_tokens": 9389, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 23, + "k5_tokens": 9302, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9383, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q012", + "query": "Show git status", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 602, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "hit_rank": 117, + "k3_calls": 39, + "k3_tokens": 9686, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 24, + "k5_tokens": 9576, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9348, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "hit_rank": 143, + "k3_calls": 48, + "k3_tokens": 12006, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 29, + "k5_tokens": 11681, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11704, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "hit_rank": 9, + "k3_calls": 3, + "k3_tokens": 741, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 831, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 807, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "hit_rank": 41, + "k3_calls": 14, + "k3_tokens": 3866, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 9, + "k5_tokens": 4037, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 5, + "k10_tokens": 4516, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 303, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 595, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1038, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "hit_rank": 65, + "k3_calls": 22, + "k3_tokens": 6335, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 13, + "k5_tokens": 6086, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 7, + "k10_tokens": 6281, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 747, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 303, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 507, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 841, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 203, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 720, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "hit_rank": 9, + "k3_calls": 3, + "k3_tokens": 678, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 706, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "hit_rank": 251, + "k3_calls": 84, + "k3_tokens": 19977, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 51, + "k5_tokens": 19522, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 26, + "k10_tokens": 19419, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 496, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 383, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 671, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 317, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 461, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 834, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "hit_rank": 9, + "k3_calls": 3, + "k3_tokens": 758, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 807, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 783, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "hit_rank": 189, + "k3_calls": 63, + "k3_tokens": 15143, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 38, + "k5_tokens": 14689, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 19, + "k10_tokens": 14309, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 274, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 387, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "hit_rank": 174, + "k3_calls": 58, + "k3_tokens": 14269, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 35, + "k5_tokens": 13802, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 18, + "k10_tokens": 13742, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "hit_rank": 178, + "k3_calls": 60, + "k3_tokens": 14674, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 36, + "k5_tokens": 14146, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 18, + "k10_tokens": 13750, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "hit_rank": 10, + "k3_calls": 4, + "k3_tokens": 1156, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 2, + "k5_tokens": 1020, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 998, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "hit_rank": 10, + "k3_calls": 4, + "k3_tokens": 782, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 2, + "k5_tokens": 650, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 630, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 266, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 365, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 722, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 741, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 413, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 293, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 210, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 644, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 802, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 347, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 801, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 276, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 780, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 191, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 752, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 676, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 287, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 477, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 917, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "hit_rank": 301, + "k3_calls": 101, + "k3_tokens": 24119, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 61, + "k5_tokens": 23357, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 31, + "k10_tokens": 23011, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 618, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 739, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 627, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 739, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "hit_rank": 8, + "k3_calls": 3, + "k3_tokens": 605, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 640, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 619, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "hit_rank": 327, + "k3_calls": 109, + "k3_tokens": 25677, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 66, + "k5_tokens": 25014, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 33, + "k10_tokens": 24354, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 209, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 359, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 732, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "hit_rank": 9, + "k3_calls": 3, + "k3_tokens": 667, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 695, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 674, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 387, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 747, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "hit_rank": 7, + "k3_calls": 3, + "k3_tokens": 736, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 785, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 488, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 376, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 397, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 289, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 602, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "hit_rank": 286, + "k3_calls": 96, + "k3_tokens": 22835, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 58, + "k5_tokens": 22198, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 29, + "k10_tokens": 21618, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 260, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 626, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 160, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 258, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 577, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 357, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 522, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1073, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 356, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 521, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1072, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 629, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "hit_rank": 116, + "k3_calls": 39, + "k3_tokens": 9799, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 24, + "k5_tokens": 9727, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9475, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "hit_rank": 125, + "k3_calls": 42, + "k3_tokens": 10496, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 25, + "k5_tokens": 10044, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 13, + "k10_tokens": 10182, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "hit_rank": 127, + "k3_calls": 43, + "k3_tokens": 10789, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 26, + "k5_tokens": 10502, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 13, + "k10_tokens": 10229, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 241, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 766, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 248, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 831, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "hit_rank": 7, + "k3_calls": 3, + "k3_tokens": 731, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 767, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 745, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 413, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 814, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 283, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 620, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "hit_rank": 49, + "k3_calls": 17, + "k3_tokens": 5013, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 10, + "k5_tokens": 4762, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 5, + "k10_tokens": 4652, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 276, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 446, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 789, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "hit_rank": 27, + "k3_calls": 9, + "k3_tokens": 2203, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 6, + "k5_tokens": 2360, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 3, + "k10_tokens": 2288, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "hit_rank": 31, + "k3_calls": 11, + "k3_tokens": 3009, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 7, + "k5_tokens": 3110, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3597, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 747, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 471, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 210, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 666, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 405, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 681, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "hit_rank": 153, + "k3_calls": 51, + "k3_tokens": 12624, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 31, + "k5_tokens": 12368, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 16, + "k10_tokens": 12436, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "hit_rank": 155, + "k3_calls": 52, + "k3_tokens": 12765, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 31, + "k5_tokens": 12256, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 16, + "k10_tokens": 12345, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 478, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 814, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 158, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 708, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "hit_rank": 146, + "k3_calls": 49, + "k3_tokens": 12231, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 30, + "k5_tokens": 12034, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11719, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q094", + "query": "Move a local file", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 780, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "hit_rank": 96, + "k3_calls": 32, + "k3_tokens": 8562, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 20, + "k5_tokens": 8492, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 10, + "k10_tokens": 8272, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "hit_rank": 89, + "k3_calls": 30, + "k3_tokens": 8310, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7998, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7764, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 627, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 652, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 719, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 678, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 187, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 631, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 272, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 780, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 481, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 804, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 786, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "hit_rank": 232, + "k3_calls": 78, + "k3_tokens": 18579, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 47, + "k5_tokens": 17991, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 24, + "k10_tokens": 17923, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 652, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "hit_rank": 203, + "k3_calls": 68, + "k3_tokens": 16305, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 41, + "k5_tokens": 15821, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 21, + "k10_tokens": 15741, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "hit_rank": 190, + "k3_calls": 64, + "k3_tokens": 15302, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 38, + "k5_tokens": 14689, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 19, + "k10_tokens": 14309, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "hit_rank": 10, + "k3_calls": 4, + "k3_tokens": 797, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 2, + "k5_tokens": 665, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "hit_rank": 171, + "k3_calls": 57, + "k3_tokens": 14078, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 35, + "k5_tokens": 13845, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 18, + "k10_tokens": 13724, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 266, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 284, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 610, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "hit_rank": 272, + "k3_calls": 91, + "k3_tokens": 21988, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 55, + "k5_tokens": 21340, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 28, + "k10_tokens": 20983, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "hit_rank": 261, + "k3_calls": 87, + "k3_tokens": 20738, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 53, + "k5_tokens": 20393, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 27, + "k10_tokens": 20238, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 301, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 461, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 771, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 576, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 461, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 834, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 139, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 241, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 585, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 711, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "hit_rank": 8, + "k3_calls": 3, + "k3_tokens": 704, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 748, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 646, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "hit_rank": 10, + "k3_calls": 4, + "k3_tokens": 940, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 2, + "k5_tokens": 808, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 788, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 209, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 602, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "hit_rank": 334, + "k3_calls": 112, + "k3_tokens": 26006, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 67, + "k5_tokens": 25091, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 34, + "k10_tokens": 24881, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "hit_rank": 8, + "k3_calls": 3, + "k3_tokens": 746, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 819, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 448, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 785, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 187, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 369, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 848, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 676, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 220, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 308, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 622, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 397, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 289, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 602, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "hit_rank": 287, + "k3_calls": 96, + "k3_tokens": 22835, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 58, + "k5_tokens": 22198, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 29, + "k10_tokens": 21618, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 293, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "hit_rank": 293, + "k3_calls": 98, + "k3_tokens": 23327, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 59, + "k5_tokens": 22549, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 30, + "k10_tokens": 22277, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 392, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 630, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 164, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 361, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 701, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "hit_rank": 62, + "k3_calls": 21, + "k3_tokens": 6073, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 13, + "k5_tokens": 6075, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 7, + "k10_tokens": 6292, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 276, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 515, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 773, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "hit_rank": 7, + "k3_calls": 3, + "k3_tokens": 714, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 770, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 750, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 274, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 762, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 615, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "hit_rank": 106, + "k3_calls": 36, + "k3_tokens": 9236, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 22, + "k5_tokens": 9029, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 11, + "k10_tokens": 8787, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "hit_rank": 97, + "k3_calls": 33, + "k3_tokens": 8765, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 20, + "k5_tokens": 8530, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 10, + "k10_tokens": 8310, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "hit_rank": 87, + "k3_calls": 29, + "k3_tokens": 7978, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7948, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7759, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 280, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 484, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 844, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "hit_rank": 11, + "k3_calls": 4, + "k3_tokens": 802, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 3, + "k5_tokens": 1045, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 1.0, + "k10_calls": 2, + "k10_tokens": 1428, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "codedb_trigram_tokens": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 270, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "hit_rank": 10, + "k3_calls": 4, + "k3_tokens": 839, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 2, + "k5_tokens": 707, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 839, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 393, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 536, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 885, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 395, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 598, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 962, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 272, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 546, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "hit_rank": 10, + "k3_calls": 4, + "k3_tokens": 937, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 2, + "k5_tokens": 801, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 779, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "hit_rank": 91, + "k3_calls": 31, + "k3_tokens": 8425, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 19, + "k5_tokens": 8277, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 10, + "k10_tokens": 8326, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "hit_rank": 99, + "k3_calls": 33, + "k3_tokens": 8757, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 20, + "k5_tokens": 8509, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 10, + "k10_tokens": 8279, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 169, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 298, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 671, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 681, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 134, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 236, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 611, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "hit_rank": 7, + "k3_calls": 3, + "k3_tokens": 860, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 910, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 890, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "hit_rank": 145, + "k3_calls": 49, + "k3_tokens": 12381, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 29, + "k5_tokens": 11702, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11743, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "hit_rank": 44, + "k3_calls": 15, + "k3_tokens": 4304, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 9, + "k5_tokens": 4160, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 5, + "k10_tokens": 4542, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 321, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 543, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1093, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 272, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 722, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 795, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 201, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 309, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 593, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 587, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 264, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 710, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "hit_rank": 251, + "k3_calls": 84, + "k3_tokens": 19910, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 51, + "k5_tokens": 19469, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 26, + "k10_tokens": 19411, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "hit_rank": 10, + "k3_calls": 4, + "k3_tokens": 1083, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 2, + "k5_tokens": 947, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 925, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 748, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 627, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 273, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 450, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 801, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 646, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 747, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 423, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 731, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 246, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 285, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 635, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 431, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 831, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 743, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "hit_rank": 166, + "k3_calls": 56, + "k3_tokens": 13846, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 34, + "k5_tokens": 13474, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 17, + "k10_tokens": 13117, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 263, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 387, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 711, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 642, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 801, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 423, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 634, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 173, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 376, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 626, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 486, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 736, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 715, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 712, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 287, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 449, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 315, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 528, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 828, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 115, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 220, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 642, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 346, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 505, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 822, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 286, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 643, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 173, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 541, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 620, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 325, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 161, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 308, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 289, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 657, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 275, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 681, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 271, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 677, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 271, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 634, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 227, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 584, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 824, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 293, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 288, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 553, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 156, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 252, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 451, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 170, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 286, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 543, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 634, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 542, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 920, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "hit_rank": 34, + "k3_calls": 12, + "k3_tokens": 3256, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 7, + "k5_tokens": 3080, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3574, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q071", + "query": "list-directory", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 653, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 452, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 650, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 220, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 773, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 425, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 306, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 637, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 464, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 681, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 661, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 284, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 347, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 537, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 858, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 720, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 286, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 423, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 768, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 392, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 411, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 434, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 800, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 652, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "hit_rank": 135, + "k3_calls": 45, + "k3_tokens": 11143, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 27, + "k5_tokens": 10783, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 14, + "k10_tokens": 10993, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 383, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 772, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 670, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 338, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 479, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 815, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 494, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "hit_rank": 147, + "k3_calls": 49, + "k3_tokens": 12287, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 30, + "k5_tokens": 12042, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11727, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q094", + "query": "Move a local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 395, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 419, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 748, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 265, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 410, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 801, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 158, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 271, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 539, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 305, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 507, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 187, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 297, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 508, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "hit_rank": 113, + "k3_calls": 38, + "k3_tokens": 9484, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 23, + "k5_tokens": 9231, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9385, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 546, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 288, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 626, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 174, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 622, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 246, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 608, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 563, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 374, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 734, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 170, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 267, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 524, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "hit_rank": 9, + "k3_calls": 3, + "k3_tokens": 943, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 1016, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 994, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 288, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 398, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 510, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 778, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 376, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 770, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 748, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 170, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 529, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 766, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 406, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 873, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 417, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 617, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 275, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 681, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 272, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 678, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 146, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 238, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 595, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 616, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 438, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 934, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 423, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 772, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 873, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 617, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 547, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 288, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 553, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 607, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 173, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 568, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 318, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 661, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 613, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 547, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 408, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 296, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 723, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 421, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 712, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 691, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 739, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 364, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 169, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 527, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 308, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 570, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "hit_rank": 8, + "k3_calls": 3, + "k3_tokens": 723, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 752, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 730, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "hit_rank": 87, + "k3_calls": 29, + "k3_tokens": 7985, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7922, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7733, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "hit_rank": 8, + "k3_calls": 3, + "k3_tokens": 679, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 744, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "hit_rank": 8, + "k3_calls": 3, + "k3_tokens": 614, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 644, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 622, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "codedb_word": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 403, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 639, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1048, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 265, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 813, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 530, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1113, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 417, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 713, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1200, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 165, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 546, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 444, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 848, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 427, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 664, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "hit_rank": 89, + "k3_calls": 30, + "k3_tokens": 8108, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7832, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7625, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 365, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 671, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "hit_rank": 7, + "k3_calls": 3, + "k3_tokens": 664, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 693, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 671, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 263, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 482, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 838, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 305, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 322, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 871, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "hit_rank": 146, + "k3_calls": 49, + "k3_tokens": 12388, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 30, + "k5_tokens": 12128, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11768, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "hit_rank": 9, + "k3_calls": 3, + "k3_tokens": 1027, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 1107, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1083, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 337, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 531, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 936, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 398, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 578, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1011, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 421, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 827, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 813, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 407, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 678, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 392, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 775, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "hit_rank": 9, + "k3_calls": 3, + "k3_tokens": 642, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 676, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 656, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "hit_rank": 257, + "k3_calls": 86, + "k3_tokens": 20636, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 52, + "k5_tokens": 20030, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 26, + "k10_tokens": 19458, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 481, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 820, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 719, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 913, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 561, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 442, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 730, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 291, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 441, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 833, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 393, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 734, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 279, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 450, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 821, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 873, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 423, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 696, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 173, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 376, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 486, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 387, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 715, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 442, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 833, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 810, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 287, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 449, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 492, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 828, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 153, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 311, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 423, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 648, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 332, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 768, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 431, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 789, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 625, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 313, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 749, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 309, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 745, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 747, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 166, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 824, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 776, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 710, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 265, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 767, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 310, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 639, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 156, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 252, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 542, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 170, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 286, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 392, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 596, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1087, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 616, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 976, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 958, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 653, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 369, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 685, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 345, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 525, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 963, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 287, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 426, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 845, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 210, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 536, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 995, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 473, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 913, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 301, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 448, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 887, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 827, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "hit_rank": 9, + "k3_calls": 3, + "k3_tokens": 798, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 844, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 820, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 342, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 550, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1085, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 769, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 507, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 414, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 414, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 823, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 636, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 411, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 773, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 778, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 754, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 351, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 693, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 323, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 565, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 950, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "hit_rank": 8, + "k3_calls": 3, + "k3_tokens": 511, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 563, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 542, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 305, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 546, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 447, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "hit_rank": 115, + "k3_calls": 39, + "k3_tokens": 9731, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 23, + "k5_tokens": 9285, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9371, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 326, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 762, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 816, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 266, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 420, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 246, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 760, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 296, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 862, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 398, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 365, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 669, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1202, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 344, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 494, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 744, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 270, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 791, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 770, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 481, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 820, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 170, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 567, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 425, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 830, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 871, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 625, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 313, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 749, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 310, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 444, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 835, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 423, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 772, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 873, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 638, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 310, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 639, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 158, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 271, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 523, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 432, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 712, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 745, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 191, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 672, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 558, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1032, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 349, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 559, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1033, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "hit_rank": 24, + "k3_calls": 8, + "k3_tokens": 2112, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 5, + "k5_tokens": 2116, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 3, + "k10_tokens": 2504, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 504, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1049, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 338, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 492, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 900, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 169, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 634, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 297, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 523, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 525, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 810, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 847, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 322, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 558, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 998, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "decision_heuristic": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 388, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 598, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 955, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 312, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1035, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 368, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 653, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1052, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 167, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 590, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 782, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 473, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 730, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 708, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 154, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 261, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 524, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 372, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 304, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 340, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 352, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 432, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 769, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 601, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 938, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 523, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1121, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 450, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 287, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 445, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 798, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 496, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 791, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 895, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 834, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 644, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 265, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 536, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 711, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 792, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 279, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 856, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 904, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 787, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 838, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 877, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 318, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 950, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 579, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 656, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 690, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 394, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 884, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 589, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 240, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 514, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 272, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 386, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 596, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1019, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 340, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 546, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 961, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 607, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 783, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 309, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 604, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 792, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 912, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 406, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 697, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 169, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 585, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 434, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 928, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 830, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 347, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 519, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 869, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 278, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 433, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 816, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 357, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 689, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 719, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 334, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 464, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 760, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 280, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 433, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 394, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 259, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 529, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 546, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 602, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 336, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 677, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 897, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 927, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 817, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 263, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 253, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 566, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 554, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 304, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 599, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 471, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1091, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 835, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 784, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 306, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 881, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 422, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 761, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 731, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 904, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 166, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 625, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 268, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 543, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 629, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 433, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 736, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 450, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 747, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 750, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 666, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 158, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 555, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 260, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 549, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 462, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 688, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 270, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 670, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "hit_rank": 11, + "k3_calls": 4, + "k3_tokens": 771, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 3, + "k5_tokens": 1050, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 1.0, + "k10_calls": 2, + "k10_tokens": 1402, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "embed_bge_small": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 375, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 560, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1034, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 345, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 493, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 902, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 795, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 477, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 828, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 314, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 547, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1094, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 596, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 677, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 656, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 164, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 266, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 585, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 642, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 369, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 643, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 696, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 640, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 275, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 875, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 284, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 483, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 870, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 388, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 617, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1118, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 528, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1003, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 457, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 780, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 286, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 481, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 819, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 766, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 197, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 743, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 678, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 307, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 806, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 317, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 858, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 661, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 594, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 678, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 613, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 346, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 633, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 943, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 383, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 672, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 855, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 786, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 915, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 187, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 210, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 823, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 253, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 719, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 284, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 756, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 298, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 773, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 624, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 734, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 618, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 808, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 691, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 670, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 820, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 806, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 773, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 332, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 710, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 264, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 856, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 752, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 782, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 866, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 635, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 142, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 240, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 526, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 274, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 593, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 367, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 536, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1012, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "hit_rank": 39, + "k3_calls": 13, + "k3_tokens": 3554, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 8, + "k5_tokens": 3600, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3528, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q071", + "query": "list-directory", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 271, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 489, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 650, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 785, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 950, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 277, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 468, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 950, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 383, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 887, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 722, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 176, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 301, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 434, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 826, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 483, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 327, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 476, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 878, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 365, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 696, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 277, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 289, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 440, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 795, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 720, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 745, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 618, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 909, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 398, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 771, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 225, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 304, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 609, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 785, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 162, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 270, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 582, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 285, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 620, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 201, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 723, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 365, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 670, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 629, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 799, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 274, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 800, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 241, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 636, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 318, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 594, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 624, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 372, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 735, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 372, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 606, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1091, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 739, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 271, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 318, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 651, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 175, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 260, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 524, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 623, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 271, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 613, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 736, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 911, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 653, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 731, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 652, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 715, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 674, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 642, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 374, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 595, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 372, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 681, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 326, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 570, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 452, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 859, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 442, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 805, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 274, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 519, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1091, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 693, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 668, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 254, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 493, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 599, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 685, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 280, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 421, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 850, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 670, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "hashed_ngram_embed": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 334, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 509, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1047, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 345, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 535, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 940, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 394, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 363, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 517, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1044, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 327, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 520, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1144, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 575, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 766, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 464, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 715, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 343, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 272, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 554, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 236, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 359, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 336, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 637, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 754, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 471, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 895, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 580, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 849, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 337, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 509, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1018, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 474, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 933, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 491, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 862, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 286, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 773, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 845, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 272, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 826, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 837, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 749, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 783, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 411, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 298, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 609, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 265, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 408, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 883, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 708, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 419, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 815, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 691, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 881, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 805, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 766, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 187, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 691, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 818, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 749, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 422, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 736, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 613, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 732, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 782, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 637, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 741, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 848, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 744, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 708, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 198, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 677, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 325, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 676, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 293, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 582, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 872, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 411, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 895, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 858, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 943, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 692, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 696, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 622, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 788, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 156, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 243, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 591, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 290, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 347, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 567, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1062, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 319, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 501, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 917, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 621, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 302, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 430, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 779, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 414, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 861, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 824, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 236, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 387, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 824, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 175, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 468, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 852, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 440, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 756, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 314, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 491, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 909, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 272, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 413, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 789, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 289, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 500, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 945, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 248, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 791, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 271, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 772, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 305, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 749, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 233, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 862, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 824, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 209, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 778, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 225, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 649, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 282, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 524, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 864, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 162, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 631, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 174, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 629, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 400, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 719, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 203, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 640, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 466, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 917, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 461, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 905, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 892, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 438, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 263, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 604, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 590, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 646, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 693, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 209, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 564, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1080, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 374, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 814, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 833, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 175, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 269, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 593, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 659, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 715, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 284, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 636, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 347, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 434, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 813, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 228, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 712, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 787, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 787, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 369, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 819, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 222, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 739, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 635, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 779, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 625, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 392, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 869, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 392, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 722, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 676, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 435, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 824, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 460, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 815, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 928, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 306, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 158, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 267, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 575, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 410, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 683, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 199, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 392, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 875, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 830, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 603, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "hybrid_codedb_embed": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 380, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 584, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1088, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 345, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 540, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 924, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 408, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 778, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 477, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 769, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 368, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 564, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1144, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 551, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 406, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 191, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 164, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 568, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 691, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 369, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 659, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 275, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 867, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 502, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 426, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 857, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 388, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 628, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1039, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 299, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 495, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1093, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 735, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 464, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 748, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 393, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 809, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 197, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 644, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 743, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 668, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 277, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 446, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 806, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 801, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 408, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 587, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 383, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 683, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 603, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 983, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 640, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 419, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 834, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 656, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 426, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 770, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 187, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 293, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 622, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 618, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 284, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 739, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 778, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 203, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 361, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 577, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 795, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 567, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 808, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 531, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 636, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 771, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 795, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 650, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 627, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 795, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 627, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 617, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 283, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 549, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 672, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 142, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 240, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 433, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 274, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 590, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 367, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 536, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1012, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "hit_rank": 39, + "k3_calls": 13, + "k3_tokens": 3554, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 8, + "k5_tokens": 3600, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3528, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q071", + "query": "list-directory", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 271, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 489, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 579, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 838, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 883, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 722, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 174, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 307, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 671, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 500, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 873, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 723, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 320, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 547, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 975, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 745, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 708, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 271, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 454, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 833, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 374, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 668, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 222, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 760, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 203, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 357, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 633, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 483, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 892, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 732, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 464, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 845, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 225, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 695, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 413, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 693, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 154, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 533, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 160, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 288, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 568, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 313, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 523, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 840, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 176, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 290, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 610, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 282, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 636, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 649, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 700, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 274, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 664, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 642, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 326, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 566, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 636, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 439, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 606, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1091, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 805, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 271, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 263, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 406, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 253, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 689, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 308, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 688, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 161, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 634, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 674, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 326, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 692, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 238, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 562, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 624, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 715, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 636, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 573, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 594, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 563, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 689, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 286, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 594, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 567, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 624, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 285, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 525, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 453, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 778, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 442, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 730, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 346, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 556, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 968, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 771, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 629, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 169, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 455, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 564, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 856, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 285, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 625, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "hybrid_codedb_trigram_bm25": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 264, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 495, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 884, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 471, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 383, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 734, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 519, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1035, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 458, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 643, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1013, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 167, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 272, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 544, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 813, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "hit_rank": 9, + "k3_calls": 3, + "k3_tokens": 690, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 753, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 731, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 383, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 311, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 612, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 653, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 653, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 283, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 639, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 611, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 543, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 471, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 840, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 290, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 451, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 537, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 414, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 971, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 516, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1093, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 296, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 449, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 287, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 426, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 813, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 347, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 613, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 668, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "hit_rank": 7, + "k3_calls": 3, + "k3_tokens": 642, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 711, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 691, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 526, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 415, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 907, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 805, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 688, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 747, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 265, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 609, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 841, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "hit_rank": 8, + "k3_calls": 3, + "k3_tokens": 695, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 727, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 387, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 730, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 761, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 779, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 273, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 423, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 376, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 838, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 346, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 749, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 449, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 668, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 284, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 493, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 863, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 133, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 590, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 487, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 857, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 286, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 643, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 513, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 620, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 325, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 730, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 285, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 657, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 274, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 595, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 306, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 624, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 664, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 227, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 598, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 783, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 784, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 776, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 646, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 586, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 153, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 252, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 458, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 170, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 272, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 543, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 303, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 487, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1053, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 598, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 481, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 917, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 644, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 748, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 650, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 425, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 720, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 640, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 770, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 246, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 282, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 577, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 457, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 983, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 747, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 276, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 446, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 864, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 837, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 741, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "hit_rank": 10, + "k3_calls": 4, + "k3_tokens": 837, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 2, + "k5_tokens": 705, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 685, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 789, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 734, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 821, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 552, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 426, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 871, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 744, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 280, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 419, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 228, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 369, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 772, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 155, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 255, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 523, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 305, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 512, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 297, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 540, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "hit_rank": 9, + "k3_calls": 3, + "k3_tokens": 634, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 675, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 654, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 621, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 802, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 626, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 832, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 823, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 374, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 273, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 670, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 587, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 571, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 614, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 565, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 509, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1020, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 730, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 778, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 441, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 770, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 754, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 286, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 298, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 595, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 274, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 595, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 786, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 238, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 595, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 892, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 761, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 731, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 872, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 604, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 573, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 605, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 607, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 266, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 543, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 575, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 408, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 296, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 449, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 634, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 169, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 514, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 308, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 507, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 191, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 692, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "hit_rank": 7, + "k3_calls": 3, + "k3_tokens": 667, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 737, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 716, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 459, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "hit_rank": 11, + "k3_calls": 4, + "k3_tokens": 751, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 3, + "k5_tokens": 994, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 1.0, + "k10_calls": 2, + "k10_tokens": 1377, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "hybrid_codedb_weighted_bm25": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 388, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 598, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1111, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 795, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 489, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 836, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 368, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 653, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1144, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 272, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 551, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 473, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 714, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 692, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 156, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 261, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 511, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 374, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 712, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 264, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 615, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 596, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 876, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 326, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 494, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 806, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 569, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1072, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 468, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1121, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 332, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 716, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 700, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 523, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 446, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 827, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 810, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 822, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 536, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 711, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 844, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 318, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 639, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 784, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 279, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 723, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 899, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 187, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 332, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 616, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 284, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 449, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 767, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 860, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 598, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 868, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 286, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 643, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 173, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 506, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 347, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 598, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 695, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 668, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 676, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 681, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 664, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 700, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 415, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 652, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 628, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 672, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 589, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 631, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 240, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 420, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 278, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 543, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 386, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 596, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1019, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 340, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 546, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 961, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 607, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 222, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 678, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 650, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 789, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 739, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 395, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 286, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 595, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 896, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 367, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 548, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 966, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 264, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 824, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 359, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 655, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 768, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 336, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 652, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 491, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 823, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 276, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 464, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 888, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 688, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 280, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 735, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 260, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 562, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 304, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 546, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 537, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 433, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 834, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 168, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 282, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 610, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 626, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 747, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 754, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 587, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 547, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 573, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 577, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 209, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 587, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 401, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 582, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1091, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 448, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 730, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 731, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 697, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 688, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 659, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 238, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 595, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 627, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 745, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 695, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 899, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 590, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 166, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 555, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 530, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 607, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 545, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 623, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 635, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 542, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 453, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 442, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 927, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 835, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 158, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 500, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 549, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 770, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 198, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 784, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 263, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 670, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 592, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "hybrid_rrf_bm25_embed": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 380, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 584, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1042, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 330, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 535, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 929, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 374, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 719, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 477, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 976, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 406, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 564, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1144, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 596, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 406, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 747, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 191, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 648, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 164, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 261, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 568, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 351, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 637, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 372, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 700, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 616, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 275, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 925, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 515, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 440, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 801, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 302, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 542, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1028, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 495, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1093, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 734, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 393, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 772, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 332, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 743, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 678, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 307, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 806, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 824, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 587, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 683, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 309, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 624, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 629, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 929, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 656, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 782, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 866, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 798, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 821, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 785, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 785, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 128, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 233, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 482, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 420, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 835, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 618, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 819, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 542, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 701, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 685, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 806, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 676, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 661, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 677, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 820, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 734, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 734, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 310, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 653, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 566, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 240, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 517, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 590, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 378, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 564, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1048, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "hit_rank": 39, + "k3_calls": 13, + "k3_tokens": 3322, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 8, + "k5_tokens": 3368, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3296, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q071", + "query": "list-directory", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 269, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 508, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 650, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 696, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 400, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 801, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 875, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 877, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 719, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 174, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 435, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 822, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 756, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 389, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 536, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1012, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 410, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 804, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 270, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 414, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 636, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 222, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 286, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 549, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 332, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 689, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 794, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 623, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 413, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 841, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 154, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 258, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 546, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 311, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 570, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 313, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 588, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 411, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 789, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 365, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 167, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 610, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 739, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 760, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 274, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 241, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 664, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 642, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 326, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 569, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 336, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 586, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 288, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 640, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 606, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1139, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 789, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 306, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 444, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 690, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 260, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 539, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 736, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 308, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 683, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 271, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 788, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 653, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 635, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 732, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 636, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 772, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 565, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 166, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 562, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 642, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 637, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 640, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 326, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 558, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 438, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 845, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 450, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 804, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 556, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 915, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 169, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 248, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 491, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 531, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 775, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 656, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "hybrid_rrf_bm25_trigram": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 380, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 557, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1038, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 318, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 513, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1003, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 460, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 887, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 364, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 522, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1144, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 590, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 806, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "hit_rank": 9, + "k3_calls": 3, + "k3_tokens": 679, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 712, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 690, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 154, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 255, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 507, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 410, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 688, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 222, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 305, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 921, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "hit_rank": 146, + "k3_calls": 49, + "k3_tokens": 12289, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 30, + "k5_tokens": 12029, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11669, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 307, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 442, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 912, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 299, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 468, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 957, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 691, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 332, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 365, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 174, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 615, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 523, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 736, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 446, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 840, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 441, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 830, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 649, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 282, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 536, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 612, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1054, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 614, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 832, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 688, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 780, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 710, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 592, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 638, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 799, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 395, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 129, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 229, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 476, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 426, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 850, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 548, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 497, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 203, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 672, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 666, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 288, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 670, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 325, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 287, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 593, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 841, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 374, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 415, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 681, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 633, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 567, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 243, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 480, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 274, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 640, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 568, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1120, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 332, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 579, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 963, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 580, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 278, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 434, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 862, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 750, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 383, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 191, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 311, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 864, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 369, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 585, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1047, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 278, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 443, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 827, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 627, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 654, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 635, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 756, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 690, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 767, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 655, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 253, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 561, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 176, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 618, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 313, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 613, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 220, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 387, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 762, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 690, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 374, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 711, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 655, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 598, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 564, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 609, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 209, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 289, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 546, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1085, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 225, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 739, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 697, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 440, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 685, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 248, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 664, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 201, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 308, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 649, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 288, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 618, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 782, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 644, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 612, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 736, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 413, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 696, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 575, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 166, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 577, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 585, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 678, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 554, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 438, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 783, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 442, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 802, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 406, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 853, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 926, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 603, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 160, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 488, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 492, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 220, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 228, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "hybrid_rrf_trigram_embed": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 380, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 538, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1014, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 330, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 535, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 981, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 253, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 477, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 976, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 314, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 509, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1144, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 560, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 387, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 290, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 587, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 334, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 263, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 637, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 369, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 668, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 695, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 275, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 875, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 284, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 327, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 494, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 993, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 299, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 431, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 962, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 468, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 286, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 779, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 806, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 197, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 607, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 496, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 480, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 882, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 754, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 365, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 649, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 581, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 623, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 931, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 692, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 413, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 832, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 871, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 826, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 830, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 831, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 783, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 656, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 808, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 608, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 844, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 558, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 701, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 890, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 806, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 287, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 359, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 672, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 846, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 415, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 839, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 922, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 633, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 283, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 548, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 240, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 537, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 274, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 679, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 320, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 536, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 998, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "hit_rank": 39, + "k3_calls": 13, + "k3_tokens": 3380, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 8, + "k5_tokens": 3426, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3354, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q071", + "query": "list-directory", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 256, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 508, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 676, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 780, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 776, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 809, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 783, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 624, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 280, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 473, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 963, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 719, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 176, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 279, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 466, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 838, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 483, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 775, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 476, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 876, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 792, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 365, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 688, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 302, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 454, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 890, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 661, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 795, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 199, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 566, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 233, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 745, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 225, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 708, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 304, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 812, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 154, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 259, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 513, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 624, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 615, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 198, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 773, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 700, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 866, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 690, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 441, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 933, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 274, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 241, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 222, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 639, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 635, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 318, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 566, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 318, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 584, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 288, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 681, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 606, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1037, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 431, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 830, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 306, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 785, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 716, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 270, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 175, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 255, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 519, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 672, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 201, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 308, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 653, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 271, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 622, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 393, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 761, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 461, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 960, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 287, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 648, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 674, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 747, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 369, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 651, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 776, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 644, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 637, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 284, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 611, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 591, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 392, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 670, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 591, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 452, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 803, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 442, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 545, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 915, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 710, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 651, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 249, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 469, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 551, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 685, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 209, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 605, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "hybrid_rrf_weighted_embed": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 394, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 598, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1037, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 345, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 474, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 884, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 424, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 311, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 477, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1013, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 368, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 564, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1144, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 272, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 596, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 191, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 615, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 164, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 261, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 585, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 372, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 322, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 875, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 284, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 483, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 769, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 370, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 573, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1087, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 528, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1093, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 805, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 287, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 422, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 708, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 812, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 332, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 644, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 743, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 661, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 307, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 810, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 945, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 741, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 644, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 649, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 279, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 543, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 951, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 671, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 891, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 925, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 815, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 187, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 874, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 769, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 802, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 318, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 690, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 847, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 618, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 812, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 650, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 701, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 701, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 811, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 676, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 677, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 837, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 411, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 819, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 787, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 791, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 566, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 240, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 590, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 274, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 590, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 378, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 573, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1072, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "hit_rank": 39, + "k3_calls": 13, + "k3_tokens": 3608, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 8, + "k5_tokens": 3654, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3582, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q071", + "query": "list-directory", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 623, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 650, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 425, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 801, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 875, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 925, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 500, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 862, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 776, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 476, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1078, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 271, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 410, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 804, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 748, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 271, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 436, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 785, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 393, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 748, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 222, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 760, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 325, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 627, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 439, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 821, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 398, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 464, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 715, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 305, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 617, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 413, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 779, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 154, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 258, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 510, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 288, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 564, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 588, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 326, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 651, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 167, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 297, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 622, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 441, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 800, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 840, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 274, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 832, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 241, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 664, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 642, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 566, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 586, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 635, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 553, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1002, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 855, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 306, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 741, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 769, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 444, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 697, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 175, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 269, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 539, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 752, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 271, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 661, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 421, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 915, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 653, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 927, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 166, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 621, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 678, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 326, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 671, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 452, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 821, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 450, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 813, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 927, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 325, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 169, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 287, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 549, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 649, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 423, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 285, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 700, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "hybrid_rrf_weighted_hashed": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 388, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 598, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1024, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 312, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 398, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 854, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 311, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 506, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1060, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 379, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 522, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1144, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 272, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 590, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 745, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 478, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 719, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 697, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 155, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 260, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 510, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 365, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 681, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 293, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 584, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 875, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 284, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 415, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 542, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1063, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 523, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1031, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 835, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 287, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 730, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 813, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 735, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 265, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 307, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 794, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 885, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 275, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 447, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 911, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 640, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 649, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 265, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 671, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 961, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 336, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 674, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 856, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 877, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 782, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 187, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 672, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 695, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 836, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 771, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 423, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 784, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 866, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 732, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 819, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 615, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 767, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 398, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 748, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 827, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 652, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 325, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 287, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 618, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 853, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 394, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 874, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 410, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 871, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 876, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 631, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 788, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 252, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 567, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 272, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 643, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 361, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 588, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1092, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 332, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 520, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 988, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 646, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 302, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 430, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 674, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 369, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 743, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 393, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 792, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 792, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 272, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 480, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 925, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 748, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 311, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 464, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 838, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 446, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 828, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 316, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 478, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1038, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 278, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 420, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 893, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 273, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 413, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 835, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 696, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 394, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 806, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 791, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 398, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 722, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 441, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 835, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 756, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 690, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 422, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 814, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 153, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 254, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 513, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 357, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 585, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 700, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 424, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 897, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 928, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 837, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 253, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 614, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 624, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 209, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 676, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 516, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1097, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 228, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 836, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 802, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 739, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 175, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 269, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 567, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 688, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 683, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 298, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 659, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 668, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 884, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 422, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 739, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 862, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 697, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 166, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 651, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 766, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 650, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 810, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 719, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 654, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 457, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 790, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 471, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 884, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 949, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 376, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 801, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 304, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 619, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 158, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 595, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 267, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 504, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 426, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 199, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 683, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 693, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "hybrid_rrf_weighted_trigram": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 388, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 584, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1038, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 528, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1003, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 693, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 474, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 993, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 368, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 522, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1144, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 272, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 590, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "hit_rank": 9, + "k3_calls": 3, + "k3_tokens": 664, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 703, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 681, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 154, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 255, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 506, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 410, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 222, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 722, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 305, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 340, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 925, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 284, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 638, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 510, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1006, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 299, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 468, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 987, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 842, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 617, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 496, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 909, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 441, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 848, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 640, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 649, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 574, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 372, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1054, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 635, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 832, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 688, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 882, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 867, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 787, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 187, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 631, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 848, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 799, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 749, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 426, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 850, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 548, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 810, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 636, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 672, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 701, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 666, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 288, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 325, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 287, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 593, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 841, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 411, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 888, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 415, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 878, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 833, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 656, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 627, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 240, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 536, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 290, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 640, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 355, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 582, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1050, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 332, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 600, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 999, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 643, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 278, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 434, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 862, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 392, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 876, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 789, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 383, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 272, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 473, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 966, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 326, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 639, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 468, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 878, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 527, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1127, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 278, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 472, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 860, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 668, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 302, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 465, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 816, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 661, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 748, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 283, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 617, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 610, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 690, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 383, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 831, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 696, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 776, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 253, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 561, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 176, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 625, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 613, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 332, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 424, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 906, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 392, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 851, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 817, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 253, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 655, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 598, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 564, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 589, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 289, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 564, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 553, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1037, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 439, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 903, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 762, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 754, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 440, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 685, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 255, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 688, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 201, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 639, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 198, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 288, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 618, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 674, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 435, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 923, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 644, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 736, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 708, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 413, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 860, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 166, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 666, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 586, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 585, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 734, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 691, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 732, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 783, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 440, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 802, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 874, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 376, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 864, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 309, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 603, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 158, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 596, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 492, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 220, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 683, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "lexical_rerank_weighted_top20": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 388, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 598, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 955, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 312, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1035, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 368, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 653, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1052, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 167, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 544, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 782, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 473, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 730, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 708, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 156, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 524, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 372, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 304, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 340, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 352, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 432, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 769, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 573, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1028, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 468, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1121, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 450, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 287, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 445, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 798, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 685, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 496, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 829, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 895, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 834, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 644, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 536, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 711, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 792, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 279, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 856, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 904, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 787, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 838, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 723, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 863, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 318, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 950, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 579, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 730, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 656, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 690, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 394, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 884, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 589, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 240, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 514, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 272, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 386, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 596, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1084, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 377, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 600, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1003, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 607, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 783, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 309, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 604, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 792, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 912, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 406, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 697, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 169, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 585, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 434, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 928, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 830, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 390, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 519, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 869, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 501, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 964, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 357, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 689, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 334, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 464, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 831, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 280, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 433, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 394, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 259, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 529, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 546, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 602, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 336, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 677, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 897, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 927, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 817, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 263, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 253, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 566, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 554, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 306, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 599, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 471, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1091, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 835, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 784, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 306, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 881, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 422, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 761, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 731, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 904, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 166, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 625, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 266, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 543, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 629, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 433, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 736, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 450, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 747, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 750, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 666, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 158, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 555, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 549, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 462, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 688, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 270, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 670, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 382, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 615, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 593, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "name_exact_substr": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "hit_rank": 29, + "k3_calls": 10, + "k3_tokens": 2782, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 6, + "k5_tokens": 2698, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 3, + "k10_tokens": 2635, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "hit_rank": 146, + "k3_calls": 49, + "k3_tokens": 12183, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 30, + "k5_tokens": 11995, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11695, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "hit_rank": 121, + "k3_calls": 41, + "k3_tokens": 10292, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 25, + "k5_tokens": 10122, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 13, + "k10_tokens": 10175, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "hit_rank": 37, + "k3_calls": 13, + "k3_tokens": 3710, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 8, + "k5_tokens": 3752, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3672, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "hit_rank": 36, + "k3_calls": 12, + "k3_tokens": 3495, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 8, + "k5_tokens": 3776, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3684, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "hit_rank": 98, + "k3_calls": 33, + "k3_tokens": 8598, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 20, + "k5_tokens": 8376, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 10, + "k10_tokens": 8156, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "hit_rank": 76, + "k3_calls": 26, + "k3_tokens": 7282, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 16, + "k5_tokens": 7224, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 8, + "k10_tokens": 7048, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "hit_rank": 82, + "k3_calls": 28, + "k3_tokens": 7711, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 17, + "k5_tokens": 7530, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7616, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "hit_rank": 89, + "k3_calls": 30, + "k3_tokens": 8108, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7832, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7625, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "hit_rank": 119, + "k3_calls": 40, + "k3_tokens": 10063, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 24, + "k5_tokens": 9727, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9475, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "hit_rank": 103, + "k3_calls": 35, + "k3_tokens": 8982, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 21, + "k5_tokens": 8674, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 11, + "k10_tokens": 8746, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q012", + "query": "Show git status", + "hit_rank": 104, + "k3_calls": 35, + "k3_tokens": 8877, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 21, + "k5_tokens": 8611, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 11, + "k10_tokens": 8713, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "hit_rank": 108, + "k3_calls": 36, + "k3_tokens": 9075, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 22, + "k5_tokens": 8922, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 11, + "k10_tokens": 8713, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "hit_rank": 136, + "k3_calls": 46, + "k3_tokens": 11528, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 28, + "k5_tokens": 11279, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 14, + "k10_tokens": 10999, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "hit_rank": 136, + "k3_calls": 46, + "k3_tokens": 11712, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 28, + "k5_tokens": 11391, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 14, + "k10_tokens": 11055, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "hit_rank": 34, + "k3_calls": 12, + "k3_tokens": 3507, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 7, + "k5_tokens": 3237, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3688, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "hit_rank": 41, + "k3_calls": 14, + "k3_tokens": 4072, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 9, + "k5_tokens": 4373, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 5, + "k10_tokens": 4766, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "hit_rank": 62, + "k3_calls": 21, + "k3_tokens": 6123, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 13, + "k5_tokens": 6050, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 7, + "k10_tokens": 6293, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "hit_rank": 51, + "k3_calls": 17, + "k3_tokens": 5097, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 11, + "k5_tokens": 5225, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 6, + "k10_tokens": 5482, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "hit_rank": 228, + "k3_calls": 76, + "k3_tokens": 18006, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 46, + "k5_tokens": 17548, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 23, + "k10_tokens": 17088, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "hit_rank": 200, + "k3_calls": 67, + "k3_tokens": 15974, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 40, + "k5_tokens": 15384, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 20, + "k10_tokens": 14984, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "hit_rank": 211, + "k3_calls": 71, + "k3_tokens": 16820, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 43, + "k5_tokens": 16424, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 22, + "k10_tokens": 16307, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "hit_rank": 244, + "k3_calls": 82, + "k3_tokens": 19588, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 49, + "k5_tokens": 18841, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 25, + "k10_tokens": 18655, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "hit_rank": 245, + "k3_calls": 82, + "k3_tokens": 19506, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 49, + "k5_tokens": 18792, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 25, + "k10_tokens": 18630, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "hit_rank": 253, + "k3_calls": 85, + "k3_tokens": 20366, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 51, + "k5_tokens": 19618, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 26, + "k10_tokens": 19523, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "hit_rank": 258, + "k3_calls": 86, + "k3_tokens": 20511, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 52, + "k5_tokens": 19991, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 26, + "k10_tokens": 19471, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "hit_rank": 191, + "k3_calls": 64, + "k3_tokens": 15221, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 39, + "k5_tokens": 14999, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 20, + "k10_tokens": 14964, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "hit_rank": 193, + "k3_calls": 65, + "k3_tokens": 15818, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 39, + "k5_tokens": 15194, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 20, + "k10_tokens": 15064, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "hit_rank": 185, + "k3_calls": 62, + "k3_tokens": 14888, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 37, + "k5_tokens": 14342, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 19, + "k10_tokens": 14250, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "hit_rank": 184, + "k3_calls": 62, + "k3_tokens": 14888, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 37, + "k5_tokens": 14342, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 19, + "k10_tokens": 14250, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "hit_rank": 166, + "k3_calls": 56, + "k3_tokens": 13816, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 34, + "k5_tokens": 13458, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 17, + "k10_tokens": 13084, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "hit_rank": 171, + "k3_calls": 57, + "k3_tokens": 14038, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 35, + "k5_tokens": 13740, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 18, + "k10_tokens": 13672, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "hit_rank": 265, + "k3_calls": 89, + "k3_tokens": 21452, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 53, + "k5_tokens": 20508, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 27, + "k10_tokens": 20309, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "hit_rank": 156, + "k3_calls": 52, + "k3_tokens": 12874, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 32, + "k5_tokens": 12736, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 16, + "k10_tokens": 12416, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "hit_rank": 158, + "k3_calls": 53, + "k3_tokens": 13156, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 32, + "k5_tokens": 12768, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 16, + "k10_tokens": 12432, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "hit_rank": 162, + "k3_calls": 54, + "k3_tokens": 13386, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 33, + "k5_tokens": 13124, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 17, + "k10_tokens": 13067, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "hit_rank": 357, + "k3_calls": 119, + "k3_tokens": 28019, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 72, + "k5_tokens": 27234, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 36, + "k10_tokens": 26478, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "hit_rank": 352, + "k3_calls": 118, + "k3_tokens": 27762, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 71, + "k5_tokens": 26866, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 36, + "k10_tokens": 26478, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "hit_rank": 226, + "k3_calls": 76, + "k3_tokens": 18006, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 46, + "k5_tokens": 17548, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 23, + "k10_tokens": 17088, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "hit_rank": 220, + "k3_calls": 74, + "k3_tokens": 17580, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 44, + "k5_tokens": 16791, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 22, + "k10_tokens": 16329, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "hit_rank": 216, + "k3_calls": 72, + "k3_tokens": 16998, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 44, + "k5_tokens": 16703, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 22, + "k10_tokens": 16285, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "hit_rank": 240, + "k3_calls": 80, + "k3_tokens": 19130, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 48, + "k5_tokens": 18458, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 24, + "k10_tokens": 17954, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "hit_rank": 243, + "k3_calls": 81, + "k3_tokens": 19511, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 49, + "k5_tokens": 18939, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 25, + "k10_tokens": 18705, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "hit_rank": 233, + "k3_calls": 78, + "k3_tokens": 18569, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 47, + "k5_tokens": 17977, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 24, + "k10_tokens": 17954, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "hit_rank": 237, + "k3_calls": 79, + "k3_tokens": 18863, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 48, + "k5_tokens": 18458, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 24, + "k10_tokens": 17954, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "hit_rank": 293, + "k3_calls": 98, + "k3_tokens": 23231, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 59, + "k5_tokens": 22522, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 30, + "k10_tokens": 22294, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "hit_rank": 296, + "k3_calls": 99, + "k3_tokens": 23677, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 60, + "k5_tokens": 23014, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 30, + "k10_tokens": 22354, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "hit_rank": 301, + "k3_calls": 101, + "k3_tokens": 23981, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 61, + "k5_tokens": 23259, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 31, + "k10_tokens": 22962, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "hit_rank": 300, + "k3_calls": 100, + "k3_tokens": 23894, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 60, + "k5_tokens": 23014, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 30, + "k10_tokens": 22354, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "hit_rank": 303, + "k3_calls": 101, + "k3_tokens": 23981, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 61, + "k5_tokens": 23259, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 31, + "k10_tokens": 22962, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "hit_rank": 308, + "k3_calls": 103, + "k3_tokens": 24500, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 62, + "k5_tokens": 23675, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 31, + "k10_tokens": 22993, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "hit_rank": 315, + "k3_calls": 105, + "k3_tokens": 24863, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 63, + "k5_tokens": 23981, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 32, + "k10_tokens": 23662, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "hit_rank": 316, + "k3_calls": 106, + "k3_tokens": 25217, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 64, + "k5_tokens": 24398, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 32, + "k10_tokens": 23694, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "hit_rank": 321, + "k3_calls": 107, + "k3_tokens": 25289, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 65, + "k5_tokens": 24664, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 33, + "k10_tokens": 24298, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "hit_rank": 323, + "k3_calls": 108, + "k3_tokens": 25409, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 65, + "k5_tokens": 24599, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 33, + "k10_tokens": 24265, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "hit_rank": 326, + "k3_calls": 109, + "k3_tokens": 25591, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 66, + "k5_tokens": 24925, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 33, + "k10_tokens": 24265, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "hit_rank": 331, + "k3_calls": 111, + "k3_tokens": 26111, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 67, + "k5_tokens": 25340, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 34, + "k10_tokens": 25017, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "hit_rank": 332, + "k3_calls": 111, + "k3_tokens": 25889, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 67, + "k5_tokens": 25206, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 34, + "k10_tokens": 24949, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "hit_rank": 335, + "k3_calls": 112, + "k3_tokens": 26352, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 67, + "k5_tokens": 25340, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 34, + "k10_tokens": 25017, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "hit_rank": 340, + "k3_calls": 114, + "k3_tokens": 26830, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 68, + "k5_tokens": 25731, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 34, + "k10_tokens": 25017, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "hit_rank": 344, + "k3_calls": 115, + "k3_tokens": 27066, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 69, + "k5_tokens": 26100, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 35, + "k10_tokens": 25770, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "hit_rank": 349, + "k3_calls": 117, + "k3_tokens": 27546, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 70, + "k5_tokens": 26505, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 35, + "k10_tokens": 25770, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "hit_rank": 362, + "k3_calls": 121, + "k3_tokens": 28315, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 73, + "k5_tokens": 27492, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 37, + "k10_tokens": 27133, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "hit_rank": 290, + "k3_calls": 97, + "k3_tokens": 22834, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 58, + "k5_tokens": 22047, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 29, + "k10_tokens": 21525, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "hit_rank": 286, + "k3_calls": 96, + "k3_tokens": 22810, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 58, + "k5_tokens": 22163, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 29, + "k10_tokens": 21583, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "hit_rank": 281, + "k3_calls": 94, + "k3_tokens": 22397, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 57, + "k5_tokens": 21868, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 29, + "k10_tokens": 21583, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "hit_rank": 277, + "k3_calls": 93, + "k3_tokens": 22087, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 56, + "k5_tokens": 21431, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 28, + "k10_tokens": 20899, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "hit_rank": 100, + "k3_calls": 34, + "k3_tokens": 8708, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 20, + "k5_tokens": 8336, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 10, + "k10_tokens": 8136, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "hit_rank": 29, + "k3_calls": 10, + "k3_tokens": 2762, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 6, + "k5_tokens": 2686, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 3, + "k10_tokens": 2629, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "hit_rank": 29, + "k3_calls": 10, + "k3_tokens": 2752, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 6, + "k5_tokens": 2680, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 3, + "k10_tokens": 2626, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "hit_rank": 82, + "k3_calls": 28, + "k3_tokens": 7599, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 17, + "k5_tokens": 7462, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7580, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "hit_rank": 116, + "k3_calls": 39, + "k3_tokens": 9799, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 24, + "k5_tokens": 9727, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9475, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "hit_rank": 117, + "k3_calls": 39, + "k3_tokens": 9838, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 24, + "k5_tokens": 9751, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9487, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "hit_rank": 118, + "k3_calls": 40, + "k3_tokens": 10063, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 24, + "k5_tokens": 9727, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9475, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "hit_rank": 25, + "k3_calls": 9, + "k3_tokens": 2387, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 5, + "k5_tokens": 2154, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 3, + "k10_tokens": 2635, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "hit_rank": 20, + "k3_calls": 7, + "k3_tokens": 1831, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 4, + "k5_tokens": 1674, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 2, + "k10_tokens": 1636, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "hit_rank": 15, + "k3_calls": 5, + "k3_tokens": 1242, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 3, + "k5_tokens": 1198, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 1.0, + "k10_calls": 2, + "k10_tokens": 1642, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "hit_rank": 66, + "k3_calls": 22, + "k3_tokens": 6260, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 14, + "k5_tokens": 6405, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 7, + "k10_tokens": 6272, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "hit_rank": 7, + "k3_calls": 3, + "k3_tokens": 725, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 802, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 782, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "hit_rank": 53, + "k3_calls": 18, + "k3_tokens": 5313, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 11, + "k5_tokens": 5225, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 6, + "k10_tokens": 5482, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "hit_rank": 45, + "k3_calls": 15, + "k3_tokens": 4523, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 9, + "k5_tokens": 4391, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 5, + "k10_tokens": 4776, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "hit_rank": 49, + "k3_calls": 17, + "k3_tokens": 5097, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 10, + "k5_tokens": 4866, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 5, + "k10_tokens": 4766, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "hit_rank": 17, + "k3_calls": 6, + "k3_tokens": 1593, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 4, + "k5_tokens": 1694, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 2, + "k10_tokens": 1646, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "hit_rank": 27, + "k3_calls": 9, + "k3_tokens": 2387, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 6, + "k5_tokens": 2698, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 3, + "k10_tokens": 2635, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "hit_rank": 122, + "k3_calls": 41, + "k3_tokens": 10333, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 25, + "k5_tokens": 10147, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 13, + "k10_tokens": 10188, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "hit_rank": 123, + "k3_calls": 41, + "k3_tokens": 10374, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 25, + "k5_tokens": 10172, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 13, + "k10_tokens": 10201, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "hit_rank": 134, + "k3_calls": 45, + "k3_tokens": 11132, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 27, + "k5_tokens": 10790, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 14, + "k10_tokens": 10985, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "hit_rank": 131, + "k3_calls": 44, + "k3_tokens": 10940, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 27, + "k5_tokens": 10817, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 14, + "k10_tokens": 10999, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "hit_rank": 151, + "k3_calls": 51, + "k3_tokens": 12585, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 31, + "k5_tokens": 12385, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 16, + "k10_tokens": 12416, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "hit_rank": 149, + "k3_calls": 50, + "k3_tokens": 12445, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 30, + "k5_tokens": 12025, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11710, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "hit_rank": 155, + "k3_calls": 52, + "k3_tokens": 12978, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 31, + "k5_tokens": 12447, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 16, + "k10_tokens": 12448, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "hit_rank": 142, + "k3_calls": 48, + "k3_tokens": 11860, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 29, + "k5_tokens": 11573, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11680, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "hit_rank": 141, + "k3_calls": 47, + "k3_tokens": 11773, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 29, + "k5_tokens": 11631, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11710, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q094", + "query": "Move a local file", + "hit_rank": 85, + "k3_calls": 29, + "k3_tokens": 7859, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 17, + "k5_tokens": 7496, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7598, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "hit_rank": 86, + "k3_calls": 29, + "k3_tokens": 7917, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7814, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7616, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "hit_rank": 80, + "k3_calls": 27, + "k3_tokens": 7629, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 16, + "k5_tokens": 7288, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 8, + "k10_tokens": 7080, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "hit_rank": 95, + "k3_calls": 32, + "k3_tokens": 8392, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 19, + "k5_tokens": 8062, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 10, + "k10_tokens": 8146, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "hit_rank": 96, + "k3_calls": 32, + "k3_tokens": 8328, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 20, + "k5_tokens": 8316, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 10, + "k10_tokens": 8126, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "hit_rank": 105, + "k3_calls": 35, + "k3_tokens": 8947, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 21, + "k5_tokens": 8653, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 11, + "k10_tokens": 8735, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "hit_rank": 109, + "k3_calls": 37, + "k3_tokens": 9343, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 22, + "k5_tokens": 8966, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 11, + "k10_tokens": 8735, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "hit_rank": 214, + "k3_calls": 72, + "k3_tokens": 17070, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 43, + "k5_tokens": 16424, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 22, + "k10_tokens": 16307, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "hit_rank": 210, + "k3_calls": 70, + "k3_tokens": 16604, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 42, + "k5_tokens": 16044, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 21, + "k10_tokens": 15624, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "hit_rank": 203, + "k3_calls": 68, + "k3_tokens": 16204, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 41, + "k5_tokens": 15712, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 21, + "k10_tokens": 15624, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "hit_rank": 241, + "k3_calls": 81, + "k3_tokens": 19511, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 49, + "k5_tokens": 18939, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 25, + "k10_tokens": 18705, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "hit_rank": 236, + "k3_calls": 79, + "k3_tokens": 18942, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 48, + "k5_tokens": 18506, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 24, + "k10_tokens": 17978, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "hit_rank": 231, + "k3_calls": 77, + "k3_tokens": 18333, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 47, + "k5_tokens": 17977, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 24, + "k10_tokens": 17954, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "hit_rank": 197, + "k3_calls": 66, + "k3_tokens": 15784, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 40, + "k5_tokens": 15384, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 20, + "k10_tokens": 14984, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "hit_rank": 194, + "k3_calls": 65, + "k3_tokens": 15558, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 39, + "k5_tokens": 15038, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 20, + "k10_tokens": 14984, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "hit_rank": 186, + "k3_calls": 62, + "k3_tokens": 14888, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 38, + "k5_tokens": 14630, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 19, + "k10_tokens": 14250, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "hit_rank": 189, + "k3_calls": 63, + "k3_tokens": 15066, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 38, + "k5_tokens": 14630, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 19, + "k10_tokens": 14250, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "hit_rank": 168, + "k3_calls": 56, + "k3_tokens": 13816, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 34, + "k5_tokens": 13458, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 17, + "k10_tokens": 13084, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "hit_rank": 169, + "k3_calls": 57, + "k3_tokens": 14038, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 34, + "k5_tokens": 13458, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 17, + "k10_tokens": 13084, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "hit_rank": 269, + "k3_calls": 90, + "k3_tokens": 21515, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 54, + "k5_tokens": 20795, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 27, + "k10_tokens": 20255, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "hit_rank": 272, + "k3_calls": 91, + "k3_tokens": 21988, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 55, + "k5_tokens": 21340, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 28, + "k10_tokens": 20983, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "hit_rank": 247, + "k3_calls": 83, + "k3_tokens": 19734, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 50, + "k5_tokens": 19130, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 25, + "k10_tokens": 18630, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "hit_rank": 256, + "k3_calls": 86, + "k3_tokens": 20597, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 52, + "k5_tokens": 20043, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 26, + "k10_tokens": 19497, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "hit_rank": 257, + "k3_calls": 86, + "k3_tokens": 20511, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 52, + "k5_tokens": 19991, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 26, + "k10_tokens": 19471, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "hit_rank": 260, + "k3_calls": 87, + "k3_tokens": 20729, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 52, + "k5_tokens": 19991, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 26, + "k10_tokens": 19471, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "hit_rank": 302, + "k3_calls": 101, + "k3_tokens": 23981, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 61, + "k5_tokens": 23259, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 31, + "k10_tokens": 22962, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "hit_rank": 307, + "k3_calls": 103, + "k3_tokens": 24397, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 62, + "k5_tokens": 23613, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 31, + "k10_tokens": 22962, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "hit_rank": 314, + "k3_calls": 105, + "k3_tokens": 24863, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 63, + "k5_tokens": 23981, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 32, + "k10_tokens": 23662, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q122", + "query": "List Okta users", + "hit_rank": 319, + "k3_calls": 107, + "k3_tokens": 25182, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 64, + "k5_tokens": 24270, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 32, + "k10_tokens": 23630, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "hit_rank": 324, + "k3_calls": 108, + "k3_tokens": 25409, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 65, + "k5_tokens": 24599, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 33, + "k10_tokens": 24265, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "hit_rank": 328, + "k3_calls": 110, + "k3_tokens": 25915, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 66, + "k5_tokens": 24991, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 33, + "k10_tokens": 24298, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "hit_rank": 333, + "k3_calls": 111, + "k3_tokens": 25778, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 67, + "k5_tokens": 25139, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 34, + "k10_tokens": 24915, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "hit_rank": 336, + "k3_calls": 112, + "k3_tokens": 26240, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 68, + "k5_tokens": 25663, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 34, + "k10_tokens": 24983, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "hit_rank": 343, + "k3_calls": 115, + "k3_tokens": 27181, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 69, + "k5_tokens": 26169, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 35, + "k10_tokens": 25805, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "hit_rank": 347, + "k3_calls": 116, + "k3_tokens": 27289, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 70, + "k5_tokens": 26505, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 35, + "k10_tokens": 25770, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "hit_rank": 351, + "k3_calls": 117, + "k3_tokens": 27312, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 71, + "k5_tokens": 26724, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 36, + "k10_tokens": 26406, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "hit_rank": 355, + "k3_calls": 119, + "k3_tokens": 28019, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 71, + "k5_tokens": 26866, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 36, + "k10_tokens": 26478, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "hit_rank": 359, + "k3_calls": 120, + "k3_tokens": 28122, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 72, + "k5_tokens": 27162, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 36, + "k10_tokens": 26442, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "hit_rank": 363, + "k3_calls": 121, + "k3_tokens": 28194, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 73, + "k5_tokens": 27419, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 37, + "k10_tokens": 27096, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "hit_rank": 287, + "k3_calls": 96, + "k3_tokens": 22810, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 58, + "k5_tokens": 22163, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 29, + "k10_tokens": 21583, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "hit_rank": 282, + "k3_calls": 94, + "k3_tokens": 22397, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 57, + "k5_tokens": 21868, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 29, + "k10_tokens": 21583, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "hit_rank": 279, + "k3_calls": 93, + "k3_tokens": 22273, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 56, + "k5_tokens": 21543, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 28, + "k10_tokens": 20955, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "hit_rank": 291, + "k3_calls": 97, + "k3_tokens": 23125, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 59, + "k5_tokens": 22581, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 30, + "k10_tokens": 22324, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "hit_rank": 295, + "k3_calls": 99, + "k3_tokens": 23578, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 59, + "k5_tokens": 22581, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 30, + "k10_tokens": 22324, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "hit_rank": 298, + "k3_calls": 100, + "k3_tokens": 23694, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 60, + "k5_tokens": 22894, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 30, + "k10_tokens": 22294, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "hit_rank": 205, + "k3_calls": 69, + "k3_tokens": 16389, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 41, + "k5_tokens": 15712, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 21, + "k10_tokens": 15624, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "hit_rank": 60, + "k3_calls": 20, + "k3_tokens": 5782, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 12, + "k5_tokens": 5614, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 6, + "k10_tokens": 5488, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "hit_rank": 63, + "k3_calls": 21, + "k3_tokens": 6123, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 13, + "k5_tokens": 6050, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 7, + "k10_tokens": 6293, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "hit_rank": 18, + "k3_calls": 6, + "k3_tokens": 1575, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 4, + "k5_tokens": 1682, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 2, + "k10_tokens": 1640, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "hit_rank": 113, + "k3_calls": 38, + "k3_tokens": 9533, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 23, + "k5_tokens": 9279, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9463, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q144", + "query": "Show git log", + "hit_rank": 111, + "k3_calls": 37, + "k3_tokens": 9269, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 23, + "k5_tokens": 9256, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9451, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "hit_rank": 101, + "k3_calls": 34, + "k3_tokens": 8674, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 21, + "k5_tokens": 8611, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 11, + "k10_tokens": 8713, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "hit_rank": 97, + "k3_calls": 33, + "k3_tokens": 8598, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 20, + "k5_tokens": 8376, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 10, + "k10_tokens": 8156, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "hit_rank": 87, + "k3_calls": 29, + "k3_tokens": 7917, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7814, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7616, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "hit_rank": 79, + "k3_calls": 27, + "k3_tokens": 7494, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 16, + "k5_tokens": 7208, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 8, + "k10_tokens": 7040, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "hit_rank": 32, + "k3_calls": 11, + "k3_tokens": 3079, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 7, + "k5_tokens": 3209, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3672, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 181, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 784, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "query_rewrite_weighted_bm25": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 388, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 598, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1021, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 477, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1035, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 368, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 605, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1069, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 167, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 590, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 782, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 473, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 730, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 708, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 154, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 261, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 524, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 372, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 304, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 340, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 352, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 432, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 769, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 573, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1067, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 523, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1121, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 287, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 445, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 798, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 496, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 791, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 895, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 732, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 644, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 265, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 536, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 711, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 833, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 279, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 896, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 832, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 787, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 838, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 490, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 792, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 769, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 877, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 318, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 950, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 579, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 656, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 690, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 394, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 884, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 834, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 589, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 156, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 252, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 514, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 272, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 386, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 596, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1019, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "hit_rank": 29, + "k3_calls": 10, + "k3_tokens": 2752, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 6, + "k5_tokens": 2680, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 3, + "k10_tokens": 2626, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 607, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 783, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 810, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 792, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 912, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 419, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 773, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 434, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 928, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 830, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 390, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 557, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 922, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 248, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 813, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 357, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 689, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 719, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 334, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 464, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 760, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 280, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 433, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 394, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 259, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 529, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 546, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 602, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 336, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 677, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 897, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 927, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 817, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 263, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 253, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 566, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 554, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 398, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 817, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 471, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1091, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 835, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 784, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 306, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 881, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 422, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 795, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 731, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 832, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 166, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 625, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 174, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 289, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 629, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 468, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 845, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 442, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 822, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 867, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 666, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 158, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 555, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 260, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 549, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 462, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 688, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 263, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "hit_rank": 9, + "k3_calls": 3, + "k3_tokens": 585, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 628, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "tfidf": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 349, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 527, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 947, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 466, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 951, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 460, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 883, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 364, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 611, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1040, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 553, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 780, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "hit_rank": 8, + "k3_calls": 3, + "k3_tokens": 712, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 745, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 723, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 154, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 544, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 710, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 372, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 574, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 958, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "hit_rank": 145, + "k3_calls": 49, + "k3_tokens": 12448, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 29, + "k5_tokens": 11769, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11810, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 307, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 444, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 732, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 495, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1121, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 791, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 420, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 236, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 410, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 711, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 652, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 523, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 415, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 750, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 962, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 299, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 871, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 596, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 678, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 939, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 283, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 672, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 279, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 875, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 899, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 359, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 387, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 857, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 419, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 772, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 824, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 170, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 650, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 905, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 173, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 618, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 203, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 683, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 679, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 635, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 413, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 903, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 770, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 657, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 579, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 147, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 245, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 543, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 315, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 472, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 900, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 314, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 433, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 864, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 654, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 222, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 201, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 780, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 248, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 201, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 421, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 911, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 419, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 434, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 879, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 545, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 957, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 307, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 462, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 948, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 346, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 438, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 802, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 633, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 207, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 816, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 602, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 285, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 839, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 137, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 283, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 591, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 798, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 716, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 735, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 260, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 561, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 620, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 621, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 220, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 610, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 426, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 916, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 948, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 745, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 374, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 654, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 255, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 609, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 456, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1091, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 844, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 441, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 175, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 269, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 815, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 306, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 582, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 679, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 410, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 900, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 708, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 241, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 422, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 695, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 677, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 899, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 639, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 595, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 619, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 621, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 685, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 456, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 812, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 859, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 406, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 837, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 266, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1001, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 619, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 158, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 604, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 260, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 531, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 769, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 263, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ] +} diff --git a/experiments/mcp-tool-search/results/decision_heuristic.csv b/experiments/mcp-tool-search/results/decision_heuristic.csv new file mode 100644 index 00000000..a5f1591f --- /dev/null +++ b/experiments/mcp-tool-search/results/decision_heuristic.csv @@ -0,0 +1,151 @@ +id,query,kind,hit1,recall@1,recall@3,recall@5,mrr@10,ndcg@10,hit_rank,k3_calls,k3_tokens,k3_success1,k5_calls,k5_tokens,k5_success1,k10_calls,k10_tokens,k10_success1,ranking +q001,Create a GitHub issue,verb_product,github.create_issue,1.0,1.0,1.0,1.0000,0.9502,1,1,388,1.0,1,598,1.0,1,955,1.0,github.create_issue github.issue_read github.issue_write github.add_issue_comment github.update_issue_comment git.git_checkout git.git_show git.git_branch git.git_create_branch github.assign_copilot_to_issue +q002,Create a Jira ticket,verb_product,jira.jira_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,312,1.0,1,412,1.0,1,763,1.0,jira.jira_create_issue jira.jira_create_sprint jira.jira_assign_issue jira.jira_list_projects jira.jira_update_issue jira.jira_get_issue jira.jira_search_issues jira.jira_add_comment jira.jira_add_worklog jira.jira_transition_issue +q003,Send a Slack message,verb_product,slack.slack_post_message,1.0,1.0,1.0,1.0000,1.0000,1,1,232,1.0,1,356,1.0,1,707,1.0,slack.slack_post_message slack.slack_add_reaction slack.slack_join_channel slack.slack_leave_channel slack.slack_create_channel slack.slack_search_messages slack.slack_upload_file slack.slack_list_users slack.slack_get_users slack.slack_list_channels +q004,List open pull requests,resource,github.list_pull_requests,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,429,1.0,1,1035,1.0,github.list_pull_requests github.search_pull_requests gitlab.gitlab_list_merge_requests github.merge_pull_request github.pull_request_read github.create_pull_request github.update_pull_request github.update_pull_request_branch github.pull_request_review_write bitbucket.bitbucket_list_prs +q005,Open a pull request on GitHub,verb_product,github.create_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,368,1.0,1,653,1.0,1,1052,1.0,github.create_pull_request github.merge_pull_request github.pull_request_read github.update_pull_request github.pull_request_review_write github.update_pull_request_branch github.request_copilot_review github.search_pull_requests github.list_pull_requests git.git_checkout +q006,Run a SQL query against postgres,verb_product,postgres.query,1.0,1.0,1.0,1.0000,1.0000,1,1,167,1.0,1,280,1.0,1,590,1.0,postgres.query postgres.explain_query postgres.list_schemas postgres.list_tables postgres.describe_table mysql.mysql_query sqlite.sqlite_query salesforce.salesforce_query prometheus.prom_query docker.docker_run +q007,Read a local file from disk,paraphrase,filesystem.read_file,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,385,1.0,1,782,1.0,filesystem.read_file filesystem.read_text_file filesystem.read_media_file filesystem.write_file filesystem.edit_file filesystem.get_file_info filesystem.move_file brave_search.brave_local_search github.delete_file gitlab.gitlab_get_file +q008,What files are in this folder,paraphrase,filesystem.search_files,0.0,0.0,0.0,0.1667,0.4348,6,2,473,0.0,2,730,0.0,1,708,1.0,filesystem.search_files google_drive.gdrive_list_folder slack.list_files github.push_files filesystem.move_file filesystem.list_directory filesystem.list_directory_with_sizes filesystem.directory_tree filesystem.read_file filesystem.read_text_file +q009,Remember this fact in the knowledge graph,paraphrase,memory.read_graph,0.0,1.0,1.0,0.5000,0.5339,2,1,154,1.0,1,261,1.0,1,524,1.0,memory.read_graph memory.create_entities memory.delete_relations memory.create_relations memory.add_observations memory.open_nodes memory.search_nodes memory.delete_entities memory.delete_observations time.get_current_time +q010,Search the web with Brave,verb_product,brave_search.brave_web_search,1.0,1.0,1.0,1.0000,1.0000,1,1,245,1.0,1,412,1.0,1,726,1.0,brave_search.brave_web_search brave_search.brave_local_search filesystem.list_directory_with_sizes jira.jira_search_issues github.update_pull_request_branch memory.search_nodes puppeteer.puppeteer_screenshot puppeteer.puppeteer_navigate puppeteer.puppeteer_click puppeteer.puppeteer_fill +q011,Fetch the contents of a URL,verb_product,fetch.fetch,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,372,1.0,1,707,1.0,fetch.fetch github.get_file_contents playwright.browser_navigate filesystem.read_file puppeteer.puppeteer_navigate git.git_add slack.slack_set_channel_topic github.get_repository_tree git.git_status playwright.browser_snapshot +q012,Show git status,identifier,git.git_status,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,304,1.0,1,600,1.0,git.git_status git.git_show git.git_diff git.git_checkout git.git_branch git.git_commit git.git_add git.git_log git.git_diff_unstaged git.git_diff_staged +q013,Commit staged changes,paraphrase,git.git_commit,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,328,1.0,1,663,1.0,git.git_commit github.get_commit git.git_diff_staged git.git_reset git.git_log git.git_show github.push_files +q014,Create a Linear issue,verb_product,linear.linear_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,340,1.0,1,463,1.0,1,765,1.0,linear.linear_create_issue linear.linear_update_issue linear.linear_assign_issue linear.linear_get_issue linear.linear_create_comment linear.linear_create_project linear.linear_list_teams linear.linear_list_projects linear.linear_list_issues linear.linear_search_issues +q015,Make a task in my project tracker Linear,paraphrase,linear.linear_create_project,0.0,1.0,1.0,0.5000,0.6309,2,1,352,1.0,1,432,1.0,1,769,1.0,linear.linear_create_project linear.linear_create_issue linear.linear_update_issue linear.linear_list_teams linear.linear_list_projects linear.linear_list_issues linear.linear_search_issues linear.linear_create_comment linear.linear_assign_issue linear.linear_get_issue +q016,Post a comment on a GitHub issue,verb_product,github.update_issue_comment,0.0,1.0,1.0,0.5000,0.6309,2,1,333,1.0,1,601,1.0,1,938,1.0,github.update_issue_comment github.add_issue_comment github.issue_read github.issue_write github.create_issue github.assign_copilot_to_issue git.git_checkout git.git_show git.git_branch git.git_diff +q017,Merge the pull request,ambiguous,github.merge_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,283,1.0,1,523,1.0,1,1121,1.0,github.merge_pull_request gitlab.gitlab_merge_merge_request gitlab.gitlab_get_merge_request github.pull_request_read github.create_pull_request github.update_pull_request_branch github.update_pull_request gitlab.gitlab_create_merge_request github.pull_request_review_write github.request_copilot_review +q018,Trigger a GitHub Actions workflow,verb_product,github.actions_run_trigger,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,450,1.0,1,737,1.0,github.actions_run_trigger github.actions_get github.actions_list github.actions_get_job_logs git.git_checkout git.git_show git.git_branch git.git_diff git.git_commit git.git_status +q019,Get Dependabot alerts,resource,github.list_dependabot_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,287,1.0,1,445,1.0,1,798,1.0,github.list_dependabot_alerts github.get_dependabot_alert github.list_code_scanning_alerts github.list_secret_scanning_alerts kubernetes.k8s_get_pod redis.redis_get github.get_discussion github.projects_get github.get_label aws_s3.s3_get_object +q020,List Sentry issues,verb_product,sentry.sentry_list_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,396,1.0,1,796,1.0,sentry.sentry_list_issues sentry.sentry_list_projects sentry.sentry_list_events sentry.sentry_get_issue sentry.sentry_update_issue github.list_issues github.search_issues gitlab.gitlab_list_issues linear.linear_list_issues linear.linear_search_issues +q021,Create a Stripe customer,verb_product,stripe.stripe_get_customer,0.0,1.0,1.0,0.5000,0.6309,2,1,186,1.0,1,300,1.0,1,669,1.0,stripe.stripe_get_customer stripe.stripe_create_customer stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_list_subscriptions stripe.stripe_create_product stripe.stripe_create_price stripe.stripe_create_refund stripe.stripe_create_payment_intent stripe.stripe_list_customers +q022,Refund a Stripe payment,paraphrase,stripe.stripe_create_refund,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,352,1.0,1,662,1.0,stripe.stripe_create_refund stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_list_customers stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_create_product stripe.stripe_create_price +q023,Send an email with Gmail,verb_product,gmail.gmail_send_email,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,389,1.0,1,703,1.0,gmail.gmail_send_email gmail.gmail_get_email gmail.gmail_trash_email gmail.gmail_modify_email gmail.gmail_search_emails gmail.gmail_create_draft gmail.gmail_list_labels filesystem.list_directory_with_sizes twilio.twilio_send_sms puppeteer.puppeteer_click +q024,Search my Gmail inbox,verb_product,gmail.gmail_search_emails,1.0,1.0,1.0,1.0000,1.0000,1,1,205,1.0,1,352,1.0,1,714,1.0,gmail.gmail_search_emails gmail.gmail_create_draft gmail.gmail_list_labels gmail.gmail_send_email gmail.gmail_get_email gmail.gmail_trash_email gmail.gmail_modify_email brave_search.brave_web_search brave_search.brave_local_search mysql.mysql_list_tables +q025,Upload a file to Google Drive,paraphrase,google_drive.gdrive_update_file,0.0,0.0,1.0,0.2500,0.4307,4,2,496,0.0,1,418,1.0,1,791,1.0,google_drive.gdrive_update_file google_drive.gdrive_share_file google_drive.gdrive_get_file google_drive.gdrive_create_file google_drive.gdrive_search google_drive.gdrive_list_folder slack.slack_upload_file gitlab.gitlab_get_file slack.slack_reply_to_thread figma.figma_get_file +q026,Create a calendar event,resource,google_calendar.gcal_delete_event,0.0,1.0,1.0,0.5000,0.6309,2,1,300,1.0,1,427,1.0,1,895,1.0,google_calendar.gcal_delete_event google_calendar.gcal_create_event google_calendar.gcal_update_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars github.create_gist github.create_milestone github.create_hook github.create_environment github.create_package +q027,List Kubernetes pods,verb_product,kubernetes.k8s_list_pods,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,314,1.0,1,718,1.0,kubernetes.k8s_list_pods kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_get_pod kubernetes.k8s_apply kubernetes.k8s_delete kubernetes.k8s_scale_deployment kubernetes.k8s_pod_logs clickup.clickup_list_tasks +q028,Get logs from a k8s pod,verb_product,kubernetes.k8s_pod_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,262,1.0,1,455,1.0,1,834,1.0,kubernetes.k8s_pod_logs kubernetes.k8s_get_pod docker.docker_logs datadog.datadog_list_logs github.actions_get_job_logs git.git_log aws_s3.s3_get_object elasticsearch.es_get_doc gitlab.gitlab_get_file github.get_tag +q029,Run a docker container,verb_product,docker.docker_run,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,345,1.0,1,644,1.0,docker.docker_run docker.docker_stop docker.docker_rm docker.docker_exec docker.docker_logs docker.docker_images docker.docker_build docker.docker_ps github.create_check_run github.get_check_run +q030,Show docker container logs,verb_product,docker.docker_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,345,1.0,1,686,1.0,docker.docker_logs docker.docker_stop docker.docker_rm docker.docker_run docker.docker_exec docker.docker_images docker.docker_build docker.docker_ps kubernetes.k8s_pod_logs datadog.datadog_list_logs +q031,Navigate the browser to a URL,ambiguous,playwright.browser_navigate,1.0,1.0,1.0,1.0000,0.9502,1,1,180,1.0,1,265,1.0,1,536,1.0,playwright.browser_navigate playwright.browser_tab_new puppeteer.puppeteer_navigate playwright.browser_tab_close playwright.browser_tab_list playwright.browser_click playwright.browser_evaluate playwright.browser_snapshot playwright.browser_press_key playwright.browser_wait_for +q032,Take a screenshot of the page,ambiguous,playwright.browser_take_screenshot,1.0,1.0,1.0,1.0000,1.0000,1,1,226,1.0,1,352,1.0,1,711,1.0,playwright.browser_take_screenshot puppeteer.puppeteer_screenshot notion.notion_get_page notion.notion_update_page confluence.confluence_get_page confluence.confluence_create_page confluence.confluence_update_page notion.notion_create_page playwright.browser_snapshot playwright.browser_press_key +q033,Create a GitLab merge request,verb_product,gitlab.gitlab_merge_merge_request,0.0,1.0,1.0,0.3333,0.5000,3,1,297,1.0,1,475,1.0,1,792,1.0,gitlab.gitlab_merge_merge_request gitlab.gitlab_get_merge_request gitlab.gitlab_create_merge_request github.merge_pull_request gitlab.gitlab_list_merge_requests git.git_checkout git.git_show git.git_branch git.git_create_branch git.git_diff +q034,Search Notion pages,verb_product,notion.notion_search,1.0,1.0,1.0,1.0000,1.0000,1,1,195,1.0,1,366,1.0,1,686,1.0,notion.notion_search notion.notion_get_page notion.notion_get_block notion.notion_query_database notion.notion_create_database notion.notion_update_page notion.notion_list_users notion.notion_get_comments notion.notion_append_block notion.notion_create_page +q035,Create a Notion page,verb_product,notion.notion_get_page,0.0,1.0,1.0,0.3333,0.5000,3,1,230,1.0,1,371,1.0,1,687,1.0,notion.notion_get_page notion.notion_update_page notion.notion_create_page notion.notion_get_comments notion.notion_create_database notion.notion_search notion.notion_get_block notion.notion_query_database notion.notion_list_users notion.notion_append_block +q036,Query a Notion database,verb_product,notion.notion_query_database,1.0,1.0,1.0,1.0000,1.0000,1,1,279,1.0,1,401,1.0,1,687,1.0,notion.notion_query_database notion.notion_create_database notion.notion_create_page notion.notion_search notion.notion_get_page notion.notion_get_block notion.notion_update_page notion.notion_list_users notion.notion_get_comments notion.notion_append_block +q037,Create a Zendesk ticket,verb_product,zendesk.zendesk_create_ticket,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,368,1.0,1,856,1.0,zendesk.zendesk_create_ticket zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_search zendesk.zendesk_list_tickets jira.jira_create_issue github.create_gist github.create_milestone github.create_hook github.create_environment +q038,Create a HubSpot contact,verb_product,hubspot.hubspot_create_contact,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,904,1.0,hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_create_deal hubspot.hubspot_list_contacts hubspot.hubspot_search_contacts github.create_gist github.create_repository github.create_branch github.create_issue github.create_pull_request +q039,Create a Salesforce lead,verb_product,salesforce.salesforce_convert_lead,0.0,1.0,1.0,0.5000,0.6309,2,1,252,1.0,1,402,1.0,1,787,1.0,salesforce.salesforce_convert_lead salesforce.salesforce_create_lead salesforce.salesforce_create_record salesforce.salesforce_query salesforce.salesforce_update_record salesforce.salesforce_delete_record salesforce.salesforce_list_opportunities salesforce.salesforce_get_record github.create_gist github.create_milestone +q040,Run a SOQL query,identifier,salesforce.salesforce_query,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,291,1.0,1,724,1.0,salesforce.salesforce_query postgres.query sqlite.sqlite_query mysql.mysql_query prometheus.prom_query docker.docker_run postgres.explain_query prometheus.prom_query_range datadog.datadog_query_metrics notion.notion_query_database +q041,List Shopify orders,verb_product,shopify.shopify_list_orders,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,354,1.0,1,682,1.0,shopify.shopify_list_orders shopify.shopify_list_products shopify.shopify_list_customers shopify.shopify_get_product shopify.shopify_create_product shopify.shopify_update_product shopify.shopify_get_order shopify.shopify_create_order clickup.clickup_list_tasks trello.trello_list_cards +q042,Create a PagerDuty incident,verb_product,pagerduty.pagerduty_create_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,403,1.0,1,838,1.0,pagerduty.pagerduty_create_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls github.create_gist github.create_milestone github.create_hook github.create_environment +q043,Who is on call in PagerDuty,paraphrase,pagerduty.pagerduty_list_oncalls,1.0,1.0,1.0,1.0000,1.0000,1,1,227,1.0,1,401,1.0,1,721,1.0,pagerduty.pagerduty_list_oncalls pagerduty.pagerduty_list_incidents pagerduty.pagerduty_get_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident twilio.twilio_make_call filesystem.list_allowed_directories trello.trello_add_comment notion.notion_get_comments +q044,Query Datadog metrics,verb_product,datadog.datadog_query_metrics,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,458,1.0,1,728,1.0,datadog.datadog_query_metrics datadog.datadog_list_monitors datadog.datadog_create_monitor datadog.datadog_list_logs datadog.datadog_get_monitor prometheus.prom_list_metrics prometheus.prom_query postgres.query postgres.explain_query salesforce.salesforce_query +q045,Search Datadog logs,verb_product,datadog.datadog_list_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,458,1.0,1,877,1.0,datadog.datadog_list_logs datadog.datadog_list_monitors datadog.datadog_get_monitor datadog.datadog_create_monitor datadog.datadog_query_metrics docker.docker_logs kubernetes.k8s_pod_logs github.actions_get_job_logs brave_search.brave_web_search brave_search.brave_local_search +q046,List S3 buckets,verb_product,aws_s3.s3_list_buckets,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,318,1.0,1,667,1.0,aws_s3.s3_list_buckets aws_s3.s3_list_objects clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments kubernetes.k8s_list_services github.projects_list github.list_milestones github.list_hooks github.list_environments +q047,Upload an object to S3,paraphrase,aws_s3.s3_put_object,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,437,1.0,1,950,1.0,aws_s3.s3_put_object aws_s3.s3_delete_object aws_s3.s3_get_object slack.slack_upload_file github.assign_copilot_to_issue slack.slack_reply_to_thread github.add_comment_to_pending_review slack.slack_add_reaction twilio.twilio_send_sms gmail.gmail_send_email +q048,Purge Cloudflare cache,verb_product,cloudflare.cf_purge_cache,1.0,1.0,1.0,1.0000,1.0000,1,1,231,1.0,1,340,1.0,1,746,1.0,cloudflare.cf_purge_cache cloudflare.cf_list_zones cloudflare.cf_create_dns cloudflare.cf_list_dns +q049,Create a Cloudflare DNS record,verb_product,cloudflare.cf_create_dns,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,356,1.0,1,738,1.0,cloudflare.cf_create_dns cloudflare.cf_list_dns cloudflare.cf_list_zones cloudflare.cf_purge_cache salesforce.salesforce_create_record salesforce.salesforce_update_record salesforce.salesforce_delete_record salesforce.salesforce_get_record git.git_commit github.create_gist +q050,List Vercel deployments,verb_product,vercel.vercel_list_deployments,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,291,1.0,1,579,1.0,vercel.vercel_list_deployments vercel.vercel_list_projects vercel.vercel_get_deployment vercel.vercel_create_project kubernetes.k8s_list_deployments clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_services aws_s3.s3_list_buckets github.projects_list +q051,Insert a row into Supabase,verb_product,supabase.supabase_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,355,1.0,1,665,1.0,supabase.supabase_insert supabase.supabase_select supabase.supabase_update supabase.supabase_invoke_fn supabase.supabase_list_tables mongodb.mongo_insert playwright.browser_type +q052,Export images from Figma,paraphrase,figma.figma_get_images,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,368,1.0,1,658,1.0,figma.figma_get_images figma.figma_get_file figma.figma_get_file_nodes figma.figma_get_comments figma.figma_post_comment docker.docker_images elasticsearch.es_get_doc aws_s3.s3_get_object memory.delete_observations docker.docker_logs +q053,Send an SMS with Twilio,verb_product,twilio.twilio_send_sms,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,401,1.0,1,740,1.0,twilio.twilio_send_sms twilio.twilio_list_messages twilio.twilio_make_call filesystem.list_directory_with_sizes gmail.gmail_send_email puppeteer.puppeteer_click puppeteer.puppeteer_fill discord.discord_send_message puppeteer.puppeteer_screenshot jira.jira_search_issues +q054,Create an Okta user,verb_product,okta.okta_get_user,0.0,1.0,1.0,0.3333,0.5000,3,1,183,1.0,1,329,1.0,1,656,1.0,okta.okta_get_user okta.okta_deactivate_user okta.okta_create_user okta.okta_list_users linear.create_user linear.get_user linear.update_user linear.delete_user slack.slack_get_user_profile asana.asana_create_task +q055,Instant query Prometheus,verb_product,prometheus.prom_query,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,329,1.0,1,680,1.0,prometheus.prom_query prometheus.prom_query_range prometheus.prom_list_metrics postgres.query postgres.explain_query salesforce.salesforce_query sqlite.sqlite_query mysql.mysql_query datadog.datadog_query_metrics notion.notion_query_database +q056,Search Grafana dashboards,verb_product,grafana.grafana_search_dashboards,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,382,1.0,1,690,1.0,grafana.grafana_search_dashboards grafana.grafana_get_dashboard grafana.grafana_create_annotation brave_search.brave_web_search brave_search.brave_local_search confluence.confluence_search github.search_issues github.search_users github.search_orgs elasticsearch.es_search +q057,Trigger a CircleCI pipeline,verb_product,circleci.circleci_trigger_pipeline,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,390,1.0,1,755,1.0,circleci.circleci_trigger_pipeline circleci.circleci_get_pipeline circleci.circleci_list_pipelines gitlab.gitlab_create_pipeline github.actions_run_trigger gitlab.create_pipeline gitlab.get_pipeline gitlab.update_pipeline gitlab.delete_pipeline hubspot.hubspot_create_deal +q058,Run terraform plan,identifier,terraform.tf_plan,1.0,1.0,1.0,1.0000,1.0000,1,1,172,1.0,1,320,1.0,1,717,1.0,terraform.tf_plan terraform.tf_apply terraform.tf_state_list postgres.explain_query docker.docker_run github.create_check_run github.get_check_run github.update_check_run github.delete_check_run github.actions_run_trigger +q059,Create an Asana task,verb_product,asana.asana_get_task,0.0,1.0,1.0,0.5000,0.6309,2,1,267,1.0,1,405,1.0,1,825,1.0,asana.asana_get_task asana.asana_create_task asana.asana_update_task asana.asana_create_project asana.asana_list_tasks clickup.clickup_get_task clickup.clickup_create_task clickup.clickup_update_task okta.okta_create_user github.create_issue +q060,Create a Trello card,verb_product,trello.trello_create_card,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,394,1.0,1,884,1.0,trello.trello_create_card trello.trello_add_comment trello.trello_create_board trello.trello_list_cards github.create_gist github.create_milestone github.create_hook github.create_environment github.create_package github.create_artifact +q061,Create a ClickUp task,verb_product,clickup.clickup_get_task,0.0,1.0,1.0,0.5000,0.6309,2,1,257,1.0,1,364,1.0,1,751,1.0,clickup.clickup_get_task clickup.clickup_create_task clickup.clickup_update_task clickup.clickup_list_tasks puppeteer.puppeteer_click playwright.browser_click asana.asana_get_task asana.asana_create_task asana.asana_update_task github.create_gist +q062,Create a Confluence page,verb_product,confluence.confluence_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,367,1.0,1,703,1.0,confluence.confluence_create_page confluence.confluence_get_page confluence.confluence_update_page confluence.confluence_search notion.notion_get_page notion.notion_create_page notion.notion_update_page playwright.browser_take_screenshot notion.notion_get_comments playwright.browser_press_key +q063,Send a Discord message,verb_product,discord.discord_send_message,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,322,1.0,1,682,1.0,discord.discord_send_message discord.discord_add_reaction discord.discord_get_channel discord.discord_list_channels discord.discord_list_members slack.slack_post_message gmail.gmail_send_email twilio.twilio_send_sms gmail.gmail_get_email gmail.gmail_trash_email +q064,Search Elasticsearch,verb_product,elasticsearch.es_search,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,404,1.0,1,699,1.0,elasticsearch.es_search elasticsearch.es_index_doc elasticsearch.es_get_doc brave_search.brave_web_search brave_search.brave_local_search confluence.confluence_search github.search_issues github.search_users github.search_orgs github.search_commits +q065,Get a Redis key,verb_product,redis.redis_get,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,280,1.0,1,589,1.0,redis.redis_get redis.redis_set redis.redis_delete redis.redis_keys playwright.browser_press_key jira.jira_get_issue aws_s3.s3_get_object figma.figma_get_file figma.figma_get_comments figma.figma_get_file_nodes +q066,Find MongoDB documents,verb_product,mongodb.mongo_find,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,381,1.0,1,738,1.0,mongodb.mongo_find mongodb.mongo_insert mongodb.mongo_update mongodb.mongo_delete mongodb.mongo_list_collections +q067,List SQLite tables,verb_product,sqlite.sqlite_list_tables,1.0,1.0,1.0,1.0000,1.0000,1,1,151,1.0,1,240,1.0,1,514,1.0,sqlite.sqlite_list_tables postgres.list_tables mysql.mysql_list_tables supabase.supabase_list_tables sqlite.sqlite_describe_table sqlite.sqlite_query postgres.list_schemas clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments +q068,Describe a postgres table,verb_product,postgres.describe_table,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,272,1.0,1,597,1.0,postgres.describe_table sqlite.sqlite_describe_table postgres.query postgres.list_tables postgres.list_schemas postgres.explain_query supabase.supabase_select supabase.supabase_insert supabase.supabase_update +q069,github_create_issue,identifier,github.create_issue,1.0,1.0,1.0,1.0000,0.9502,1,1,386,1.0,1,596,1.0,1,1019,1.0,github.create_issue github.issue_read github.issue_write github.add_issue_comment github.update_issue_comment github.assign_copilot_to_issue bitbucket.bitbucket_create_issue sentry.sentry_get_issue gitlab.gitlab_create_issue jira.jira_assign_issue +q070,createIssue,identifier,github.create_issue,1.0,1.0,1.0,1.0000,0.7349,1,1,340,1.0,1,546,1.0,1,961,1.0,github.create_issue github.issue_read bitbucket.bitbucket_create_issue github.issue_write sentry.sentry_get_issue gitlab.gitlab_create_issue jira.jira_assign_issue jira.jira_get_issue linear.linear_create_issue gitlab.gitlab_get_issue +q071,list-directory,identifier,filesystem.create_directory,0.0,1.0,1.0,0.5000,0.6309,2,1,182,1.0,1,296,1.0,1,607,1.0,filesystem.create_directory filesystem.list_directory filesystem.directory_tree filesystem.list_directory_with_sizes filesystem.get_file_info filesystem.list_allowed_directories filesystem.read_file filesystem.read_text_file filesystem.read_media_file filesystem.write_file +q072,sequential thinking next thought,paraphrase,sequential_thinking.sequentialthinking,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,382,1.0,1,797,1.0,sequential_thinking.sequentialthinking +q073,What time is it in Tokyo,paraphrase,time.convert_time,1.0,1.0,1.0,1.0000,0.8597,1,1,217,1.0,1,329,1.0,1,783,1.0,time.convert_time time.get_current_time playwright.browser_wait_for filesystem.list_allowed_directories slack.slack_create_channel google_calendar.gcal_list_events grafana.grafana_create_annotation jira.jira_add_worklog elasticsearch.es_index_doc github.create_milestone +q074,Convert time between timezones,verb_product,time.convert_time,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,335,1.0,1,704,1.0,time.convert_time time.get_current_time salesforce.salesforce_convert_lead playwright.browser_wait_for git.git_diff memory.create_relations google_calendar.gcal_list_events grafana.grafana_create_annotation jira.jira_add_worklog +q075,Star a GitHub repo,verb_product,github.star_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,195,1.0,1,309,1.0,1,604,1.0,github.star_repository git.git_branch git.git_show git.git_checkout git.git_status git.git_diff git.git_commit git.git_add git.git_reset git.git_diff_unstaged +q076,Fork a repository,resource,github.fork_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,235,1.0,1,386,1.0,1,792,1.0,github.fork_repository github.delete_repository github.star_repository github.unstar_repository github.create_repository github.list_repository_collaborators github.get_repository_tree github.list_dependabot_alerts github.list_branches github.list_releases +q077,Get the latest GitHub release,verb_product,github.get_latest_release,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,366,1.0,1,647,1.0,github.get_latest_release github.get_release_by_tag git.git_commit git.git_status git.git_add git.git_checkout git.git_show git.git_branch git.git_diff git.git_reset +q078,Create a gist,resource,github.create_gist,1.0,1.0,1.0,1.0000,1.0000,1,1,229,1.0,1,429,1.0,1,912,1.0,github.create_gist github.update_gist github.get_gist github.create_milestone github.create_hook github.create_environment github.create_package github.create_artifact github.create_runner gitlab.create_pipeline +q079,Search GitHub code,verb_product,github.search_code,1.0,1.0,1.0,1.0000,1.0000,1,1,283,1.0,1,406,1.0,1,697,1.0,github.search_code github.get_code_scanning_alert github.list_code_scanning_alerts git.git_checkout git.git_show git.git_branch git.git_diff git.git_commit git.git_status git.git_add +q080,List GitHub notifications,verb_product,github.list_notifications,1.0,1.0,1.0,1.0000,1.0000,1,1,169,1.0,1,292,1.0,1,585,1.0,github.list_notifications github.mark_all_notifications_read git.git_branch git.git_checkout git.git_show git.git_diff git.git_commit git.git_status git.git_add git.git_reset +q081,Assign Copilot to an issue,verb_product,github.assign_copilot_to_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,434,1.0,1,928,1.0,github.assign_copilot_to_issue jira.jira_assign_issue linear.linear_assign_issue github.request_copilot_review jira.jira_update_issue jira.jira_transition_issue github.create_issue github.add_issue_comment slack.slack_reply_to_thread github.issue_read +q082,Get secret scanning alerts,resource,github.list_secret_scanning_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,295,1.0,1,470,1.0,1,830,1.0,github.list_secret_scanning_alerts github.get_secret_scanning_alert github.list_code_scanning_alerts github.list_dependabot_alerts github.get_code_scanning_alert kubernetes.k8s_get_pod redis.redis_get github.get_discussion github.projects_get github.get_label +q083,Create or update a file on GitHub,verb_product,github.create_or_update_file,1.0,1.0,1.0,1.0000,0.7602,1,1,347,1.0,1,519,1.0,1,869,1.0,github.create_or_update_file github.delete_file git.git_diff git.git_add github.get_file_contents git.git_checkout git.git_show git.git_branch github.get_file_blame git.git_create_branch +q084,List collaborators on a repo,resource,github.list_repository_collaborators,1.0,1.0,1.0,1.0000,1.0000,1,1,278,1.0,1,433,1.0,1,816,1.0,github.list_repository_collaborators github.list_commits trello.trello_list_cards pagerduty.pagerduty_list_oncalls github.get_discussion_comments github.update_issue_comment trello.trello_add_comment github.request_copilot_review notion.notion_get_comments playwright.browser_press_key +q085,Reply in a Slack thread,verb_product,slack.slack_reply_to_thread,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,358,1.0,1,717,1.0,slack.slack_reply_to_thread slack.slack_get_thread_replies slack.slack_list_channels slack.slack_join_channel slack.slack_leave_channel slack.slack_create_channel slack.slack_search_messages slack.slack_upload_file slack.slack_list_users slack.slack_post_message +q086,Add an emoji reaction on Slack,verb_product,slack.slack_add_reaction,1.0,1.0,1.0,1.0000,0.7602,1,1,216,1.0,1,357,1.0,1,709,1.0,slack.slack_add_reaction slack.slack_join_channel slack.slack_leave_channel slack.slack_create_channel slack.slack_search_messages slack.slack_upload_file slack.slack_list_users slack.slack_post_message slack.slack_get_users slack.slack_list_channels +q087,Search Slack messages,verb_product,slack.slack_search_messages,1.0,1.0,1.0,1.0000,1.0000,1,1,213,1.0,1,337,1.0,1,689,1.0,slack.slack_search_messages slack.slack_get_channel_history slack.slack_join_channel slack.slack_leave_channel slack.slack_create_channel slack.slack_upload_file slack.slack_list_users slack.slack_post_message slack.slack_get_users slack.slack_list_channels +q088,Create a Slack channel,verb_product,slack.slack_join_channel,0.0,1.0,1.0,0.3333,0.5000,3,1,196,1.0,1,342,1.0,1,719,1.0,slack.slack_join_channel slack.slack_leave_channel slack.slack_create_channel slack.slack_get_channel_history slack.slack_set_channel_topic slack.slack_post_message slack.slack_upload_file slack.slack_add_reaction slack.slack_search_messages slack.slack_list_users +q089,Transition a Jira issue,verb_product,jira.jira_transition_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,389,1.0,1,763,1.0,jira.jira_transition_issue jira.jira_assign_issue jira.jira_get_issue jira.jira_update_issue jira.jira_create_issue jira.jira_add_comment jira.jira_add_worklog jira.jira_list_projects jira.jira_create_sprint jira.jira_search_issues +q090,Search Jira with JQL,identifier,jira.jira_search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,194,1.0,1,370,1.0,1,764,1.0,jira.jira_search_issues jira.jira_assign_issue jira.jira_list_projects jira.jira_create_sprint jira.jira_update_issue jira.jira_get_issue jira.jira_add_comment jira.jira_add_worklog jira.jira_transition_issue jira.jira_create_issue +q091,Add a worklog to Jira,verb_product,jira.jira_add_worklog,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,350,1.0,1,765,1.0,jira.jira_add_worklog jira.jira_add_comment jira.jira_transition_issue jira.jira_assign_issue jira.jira_list_projects jira.jira_create_sprint jira.jira_update_issue jira.jira_get_issue jira.jira_search_issues jira.jira_create_issue +q092,List Linear teams,verb_product,linear.linear_list_teams,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,380,1.0,1,764,1.0,linear.linear_list_teams linear.linear_list_projects linear.linear_list_issues linear.linear_update_issue linear.linear_create_project linear.linear_create_issue linear.linear_search_issues linear.linear_create_comment linear.linear_assign_issue linear.linear_get_issue +q093,Comment on a Linear issue,verb_product,linear.linear_create_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,334,1.0,1,464,1.0,1,760,1.0,linear.linear_create_comment linear.linear_update_issue linear.linear_create_issue linear.linear_assign_issue linear.linear_get_issue linear.create_comment linear.get_comment linear.update_comment linear.delete_comment linear.linear_list_teams +q094,Move a local file,paraphrase,filesystem.move_file,1.0,1.0,1.0,1.0000,1.0000,1,1,214,1.0,1,354,1.0,1,699,1.0,filesystem.move_file brave_search.brave_local_search filesystem.read_file filesystem.read_text_file figma.figma_get_file filesystem.read_media_file filesystem.write_file slack.create_file slack.get_file slack.update_file +q095,Recursively search files by pattern,paraphrase,filesystem.search_files,1.0,1.0,1.0,1.0000,1.0000,1,1,280,1.0,1,433,1.0,1,705,1.0,filesystem.search_files slack.list_files github.push_files github.get_release_by_tag filesystem.move_file filesystem.list_directory filesystem.list_directory_with_sizes filesystem.directory_tree redis.redis_keys filesystem.read_file +q096,Edit a file in place with old and new string,paraphrase,filesystem.edit_file,1.0,1.0,1.0,1.0000,1.0000,1,1,227,1.0,1,394,1.0,1,682,1.0,filesystem.edit_file playwright.browser_tab_new filesystem.list_directory_with_sizes filesystem.write_file slack.create_file slack.get_file slack.update_file slack.delete_file figma.figma_get_file filesystem.move_file +q097,Read the whole knowledge graph,paraphrase,memory.read_graph,1.0,1.0,1.0,1.0000,1.0000,1,1,152,1.0,1,259,1.0,1,529,1.0,memory.read_graph memory.delete_relations memory.create_entities memory.create_relations memory.add_observations memory.delete_entities memory.open_nodes memory.search_nodes memory.delete_observations filesystem.read_file +q098,Search memory nodes,verb_product,memory.search_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,202,1.0,1,301,1.0,1,546,1.0,memory.search_nodes memory.open_nodes figma.figma_get_file_nodes memory.create_entities memory.create_relations memory.add_observations memory.delete_entities memory.delete_observations memory.delete_relations memory.read_graph +q099,Show unstaged git diff,verb_product,git.git_diff_unstaged,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,303,1.0,1,602,1.0,git.git_diff_unstaged git.git_diff git.git_diff_staged git.git_show git.git_status git.git_checkout git.git_branch git.git_commit git.git_add git.git_log +q100,Stage files with git add,identifier,filesystem.search_files,0.0,1.0,1.0,0.3333,0.5000,3,1,213,1.0,1,336,1.0,1,677,1.0,filesystem.search_files filesystem.list_directory_with_sizes git.git_add git.git_checkout git.git_show git.git_branch git.git_diff git.git_commit git.git_status github.push_files +q101,Create a Shopify product,verb_product,shopify.shopify_create_product,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,380,1.0,1,702,1.0,shopify.shopify_create_product shopify.shopify_get_product shopify.shopify_update_product shopify.shopify_create_order shopify.shopify_list_products shopify.shopify_list_orders shopify.shopify_get_order shopify.shopify_list_customers stripe.stripe_create_product stripe.stripe_create_price +q102,Cancel a Stripe subscription,verb_product,stripe.stripe_cancel_subscription,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,301,1.0,1,632,1.0,stripe.stripe_cancel_subscription stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_list_customers stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_create_product stripe.stripe_create_price stripe.stripe_list_subscriptions stripe.stripe_create_refund +q103,Create a payment intent,resource,stripe.stripe_create_payment_intent,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,417,1.0,1,897,1.0,stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent stripe.stripe_create_refund github.create_gist github.create_milestone github.create_hook github.create_environment github.create_package github.create_artifact github.create_runner +q104,Acknowledge a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,261,1.0,1,405,1.0,1,717,1.0,pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls +q105,Create a Datadog monitor,verb_product,datadog.datadog_get_monitor,0.0,1.0,1.0,0.5000,0.6309,2,1,259,1.0,1,459,1.0,1,927,1.0,datadog.datadog_get_monitor datadog.datadog_create_monitor datadog.datadog_list_monitors datadog.datadog_list_logs datadog.datadog_query_metrics github.create_gist github.create_milestone github.create_hook github.create_environment github.create_package +q106,Resolve a Sentry issue,paraphrase,sentry.sentry_get_issue,0.0,1.0,1.0,0.5000,0.6309,2,1,251,1.0,1,397,1.0,1,817,1.0,sentry.sentry_get_issue sentry.sentry_update_issue sentry.sentry_list_events sentry.sentry_list_projects sentry.sentry_list_issues pagerduty.pagerduty_resolve_incident github.issue_read jira.jira_assign_issue github.issue_write jira.jira_get_issue +q107,Scale a Kubernetes deployment,verb_product,kubernetes.k8s_scale_deployment,1.0,1.0,1.0,1.0000,1.0000,1,1,263,1.0,1,407,1.0,1,726,1.0,kubernetes.k8s_scale_deployment kubernetes.k8s_get_pod kubernetes.k8s_apply kubernetes.k8s_delete kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_list_pods kubernetes.k8s_pod_logs vercel.vercel_get_deployment +q108,Apply a Kubernetes manifest,verb_product,kubernetes.k8s_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,253,1.0,1,407,1.0,1,727,1.0,kubernetes.k8s_apply kubernetes.k8s_get_pod kubernetes.k8s_delete kubernetes.k8s_list_deployments kubernetes.k8s_scale_deployment kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_list_pods kubernetes.k8s_pod_logs terraform.tf_apply +q109,Stop a docker container,verb_product,docker.docker_stop,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,345,1.0,1,641,1.0,docker.docker_stop docker.docker_rm docker.docker_logs docker.docker_run docker.docker_exec docker.docker_images docker.docker_build docker.docker_ps kubernetes.k8s_pod_logs +q110,Build a docker image,verb_product,docker.docker_build,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,317,1.0,1,601,1.0,docker.docker_build docker.docker_run docker.docker_stop docker.docker_rm docker.docker_images docker.docker_ps docker.docker_logs docker.docker_exec filesystem.read_media_file +q111,Click an element in Playwright,verb_product,playwright.browser_click,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,329,1.0,1,566,1.0,playwright.browser_click puppeteer.puppeteer_click playwright.browser_type playwright.browser_take_screenshot playwright.browser_snapshot playwright.browser_navigate playwright.browser_fill_form playwright.browser_evaluate playwright.browser_tab_list playwright.browser_tab_new +q112,Type text into a browser input,paraphrase,playwright.browser_type,1.0,1.0,1.0,1.0000,0.7602,1,1,217,1.0,1,302,1.0,1,554,1.0,playwright.browser_type playwright.browser_navigate playwright.browser_wait_for playwright.browser_tab_close playwright.browser_tab_list playwright.browser_tab_new playwright.browser_snapshot playwright.browser_click playwright.browser_evaluate playwright.browser_fill_form +q113,List GitLab projects,verb_product,gitlab.gitlab_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,179,1.0,1,304,1.0,1,599,1.0,gitlab.gitlab_list_projects git.git_branch git.git_checkout git.git_show git.git_diff git.git_commit git.git_status git.git_add github.projects_list git.git_reset +q114,Create a Bitbucket pull request,verb_product,bitbucket.bitbucket_create_pr,1.0,1.0,1.0,1.0000,1.0000,1,1,313,1.0,1,471,1.0,1,1091,1.0,bitbucket.bitbucket_create_pr bitbucket.bitbucket_list_prs bitbucket.bitbucket_create_issue bitbucket.bitbucket_list_repos github.merge_pull_request github.pull_request_read github.create_pull_request github.update_pull_request github.update_pull_request_branch github.pull_request_review_write +q115,Create a Gmail draft,verb_product,gmail.gmail_create_draft,1.0,1.0,1.0,1.0000,1.0000,1,1,205,1.0,1,352,1.0,1,835,1.0,gmail.gmail_create_draft gmail.gmail_search_emails gmail.gmail_list_labels gmail.gmail_send_email gmail.gmail_get_email gmail.gmail_trash_email gmail.gmail_modify_email github.create_pull_request github.create_gist github.create_milestone +q116,Share a Google Drive file,verb_product,google_drive.gdrive_share_file,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,417,1.0,1,726,1.0,google_drive.gdrive_share_file google_drive.gdrive_update_file google_drive.gdrive_get_file google_drive.gdrive_create_file google_drive.gdrive_search google_drive.gdrive_list_folder gitlab.gitlab_get_file figma.figma_get_file google_calendar.gcal_list_calendars filesystem.write_file +q117,List Google Calendar events,verb_product,google_calendar.gcal_list_events,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,427,1.0,1,784,1.0,google_calendar.gcal_list_events google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event google_calendar.gcal_list_calendars kubernetes.k8s_get_events sentry.sentry_list_events google_drive.gdrive_list_folder google_drive.gdrive_update_file google_drive.gdrive_share_file +q118,Delete a calendar event,resource,google_calendar.gcal_delete_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,427,1.0,1,751,1.0,google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars kubernetes.k8s_delete redis.redis_delete aws_s3.s3_delete_object github.delete_repository github.delete_milestone +q119,List Vercel projects,verb_product,vercel.vercel_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,301,1.0,1,550,1.0,vercel.vercel_list_projects vercel.vercel_list_deployments vercel.vercel_get_deployment vercel.vercel_create_project github.projects_list linear.linear_list_projects jira.jira_list_projects sentry.sentry_list_projects gitlab.gitlab_list_projects github.projects_get +q120,Select rows from Supabase,verb_product,supabase.supabase_select,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,354,1.0,1,709,1.0,supabase.supabase_select supabase.supabase_insert supabase.supabase_update supabase.supabase_invoke_fn supabase.supabase_list_tables elasticsearch.es_get_doc aws_s3.s3_get_object memory.delete_observations docker.docker_logs kubernetes.k8s_pod_logs +q121,Post a Figma comment,verb_product,figma.figma_post_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,368,1.0,1,727,1.0,figma.figma_post_comment figma.figma_get_file figma.figma_get_comments figma.figma_get_images figma.figma_get_file_nodes linear.create_comment linear.get_comment linear.update_comment linear.delete_comment notion.create_comment +q122,List Okta users,verb_product,okta.okta_list_users,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,306,1.0,1,606,1.0,okta.okta_list_users okta.okta_get_user okta.okta_create_user okta.okta_deactivate_user linear.list_users github.search_users slack.slack_list_users notion.notion_list_users notion.list_users_tokens slack.slack_get_users +q123,Range query Prometheus,verb_product,prometheus.prom_query_range,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,329,1.0,1,680,1.0,prometheus.prom_query_range prometheus.prom_query prometheus.prom_list_metrics postgres.query postgres.explain_query salesforce.salesforce_query sqlite.sqlite_query mysql.mysql_query datadog.datadog_query_metrics notion.notion_query_database +q124,Create a Grafana annotation,verb_product,grafana.grafana_create_annotation,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,401,1.0,1,881,1.0,grafana.grafana_create_annotation grafana.grafana_search_dashboards grafana.grafana_get_dashboard github.create_gist github.create_milestone github.create_hook github.create_environment github.create_package github.create_artifact github.create_runner +q125,Apply terraform,identifier,terraform.tf_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,171,1.0,1,292,1.0,1,698,1.0,terraform.tf_apply terraform.tf_plan terraform.tf_state_list kubernetes.k8s_apply +q126,List Asana tasks,verb_product,asana.asana_list_tasks,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,404,1.0,1,682,1.0,asana.asana_list_tasks asana.asana_create_task asana.asana_get_task asana.asana_update_task asana.asana_create_project clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments kubernetes.k8s_list_services aws_s3.s3_list_buckets +q127,Comment on a Trello card,verb_product,trello.trello_add_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,422,1.0,1,757,1.0,trello.trello_add_comment trello.trello_create_card trello.trello_list_cards trello.trello_create_board github.update_issue_comment figma.figma_post_comment linear.create_comment linear.get_comment linear.update_comment linear.delete_comment +q128,Update a ClickUp task,verb_product,clickup.clickup_update_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,364,1.0,1,761,1.0,clickup.clickup_update_task clickup.clickup_get_task clickup.clickup_create_task clickup.clickup_list_tasks puppeteer.puppeteer_click playwright.browser_click asana.asana_update_task asana.asana_get_task asana.asana_create_task mongodb.mongo_update +q129,Search Confluence,verb_product,confluence.confluence_search,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,401,1.0,1,731,1.0,confluence.confluence_search confluence.confluence_get_page confluence.confluence_create_page confluence.confluence_update_page brave_search.brave_web_search brave_search.brave_local_search github.search_issues github.search_users github.search_orgs elasticsearch.es_search +q130,Create a HubSpot deal,verb_product,hubspot.hubspot_create_deal,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,904,1.0,hubspot.hubspot_create_deal hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_list_contacts hubspot.hubspot_search_contacts github.create_gist github.create_repository github.create_branch github.create_issue github.create_pull_request +q131,List Zendesk tickets,verb_product,zendesk.zendesk_list_tickets,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,367,1.0,1,645,1.0,zendesk.zendesk_list_tickets zendesk.zendesk_search zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_create_ticket clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments kubernetes.k8s_list_services aws_s3.s3_list_buckets +q132,List Discord channels,verb_product,discord.discord_list_channels,1.0,1.0,1.0,1.0000,1.0000,1,1,166,1.0,1,321,1.0,1,625,1.0,discord.discord_list_channels discord.discord_list_members discord.discord_get_channel discord.discord_send_message discord.discord_add_reaction slack.slack_list_channels clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments kubernetes.k8s_list_services +q133,Set a Redis key,verb_product,redis.redis_set,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,280,1.0,1,600,1.0,redis.redis_set redis.redis_get redis.redis_delete redis.redis_keys playwright.browser_press_key jira.jira_get_issue slack.slack_set_channel_topic aws_s3.s3_get_object aws_s3.s3_delete_object figma.figma_get_file +q134,Insert MongoDB documents,verb_product,mongodb.mongo_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,256,1.0,1,381,1.0,1,698,1.0,mongodb.mongo_insert mongodb.mongo_update mongodb.mongo_delete mongodb.mongo_find mongodb.mongo_list_collections supabase.supabase_insert +q135,Run a MySQL query,verb_product,postgres.query,0.0,1.0,1.0,0.3333,0.5000,3,1,182,1.0,1,268,1.0,1,543,1.0,postgres.query postgres.explain_query mysql.mysql_query postgres.list_schemas postgres.list_tables postgres.describe_table mysql.mysql_list_tables sqlite.sqlite_query salesforce.salesforce_query prometheus.prom_query +q136,Index a document in Elasticsearch,verb_product,elasticsearch.es_index_doc,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,396,1.0,1,753,1.0,elasticsearch.es_index_doc elasticsearch.es_search elasticsearch.es_get_doc playwright.browser_tab_close github.create_milestone github.get_milestone github.list_milestones github.update_milestone github.delete_milestone github.create_hook +q137,Get an S3 object,verb_product,aws_s3.s3_get_object,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,355,1.0,1,706,1.0,aws_s3.s3_get_object aws_s3.s3_delete_object aws_s3.s3_put_object okta.okta_get_user asana.asana_get_task aws_s3.s3_list_objects elasticsearch.es_search okta.okta_create_user okta.okta_deactivate_user asana.asana_create_task +q138,List Cloudflare zones,verb_product,cloudflare.cf_list_zones,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,338,1.0,1,629,1.0,cloudflare.cf_list_zones cloudflare.cf_purge_cache cloudflare.cf_list_dns cloudflare.cf_create_dns clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments kubernetes.k8s_list_services aws_s3.s3_list_buckets github.projects_list +q139,Create a Stripe invoice,verb_product,stripe.stripe_create_invoice,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,353,1.0,1,632,1.0,stripe.stripe_create_invoice stripe.stripe_create_customer stripe.stripe_create_product stripe.stripe_create_price stripe.stripe_create_refund stripe.stripe_get_customer stripe.stripe_list_customers stripe.stripe_list_invoices stripe.stripe_list_subscriptions stripe.stripe_cancel_subscription +q140,List GitHub Actions workflows,verb_product,github.actions_list,1.0,1.0,1.0,1.0000,1.0000,1,1,293,1.0,1,433,1.0,1,736,1.0,github.actions_list github.actions_get github.actions_run_trigger github.actions_get_job_logs git.git_branch git.git_checkout git.git_show git.git_diff git.git_commit git.git_status +q141,Get GitHub Actions job logs,verb_product,github.actions_get_job_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,450,1.0,1,747,1.0,github.actions_get_job_logs github.actions_get github.actions_list github.actions_run_trigger git.git_log git.git_checkout git.git_show git.git_branch git.git_diff git.git_commit +q142,Create a GitHub repository,verb_product,github.create_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,388,1.0,1,750,1.0,github.create_repository github.delete_repository github.fork_repository github.star_repository github.unstar_repository github.list_repository_collaborators github.get_repository_tree git.git_commit git.git_status git.git_checkout +q143,Checkout a git branch,verb_product,git.git_checkout,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,373,1.0,1,666,1.0,git.git_checkout git.git_create_branch git.git_branch git.git_show github.create_branch git.git_diff git.git_commit git.git_status git.git_add git.git_reset +q144,Show git log,identifier,git.git_log,1.0,1.0,1.0,1.0000,1.0000,1,1,206,1.0,1,320,1.0,1,600,1.0,git.git_log git.git_show git.git_diff git.git_status git.git_checkout git.git_branch git.git_commit git.git_add git.git_diff_unstaged git.git_diff_staged +q145,List postgres schemas,verb_product,postgres.list_schemas,1.0,1.0,1.0,1.0000,1.0000,1,1,158,1.0,1,277,1.0,1,555,1.0,postgres.list_schemas postgres.list_tables postgres.query postgres.describe_table postgres.explain_query clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments kubernetes.k8s_list_services aws_s3.s3_list_buckets +q146,Open nodes in the knowledge graph,verb_product,memory.open_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,163,1.0,1,260,1.0,1,549,1.0,memory.open_nodes memory.read_graph memory.search_nodes memory.create_entities memory.delete_relations memory.create_relations memory.add_observations memory.delete_entities memory.delete_observations figma.figma_get_file_nodes +q147,Get file metadata from the filesystem,paraphrase,filesystem.read_file,0.0,0.0,1.0,0.2500,0.4307,4,2,462,0.0,1,348,1.0,1,688,1.0,filesystem.read_file filesystem.read_text_file filesystem.read_media_file filesystem.get_file_info filesystem.write_file filesystem.edit_file filesystem.move_file filesystem.list_allowed_directories google_drive.gdrive_get_file gitlab.gitlab_get_file +q148,Write a new local file,paraphrase,playwright.browser_tab_new,0.0,1.0,1.0,0.5000,0.6309,2,1,217,1.0,1,366,1.0,1,714,1.0,playwright.browser_tab_new filesystem.write_file brave_search.brave_local_search filesystem.edit_file filesystem.read_file filesystem.read_text_file figma.figma_get_file filesystem.read_media_file slack.create_file slack.get_file +q149,Search GitHub issues,verb_product,github.search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,270,1.0,1,377,1.0,1,670,1.0,github.search_issues github.list_issues git.git_checkout git.git_show git.git_branch git.git_diff git.git_commit git.git_status git.git_add git.git_reset +q150,Get the authenticated GitHub user,paraphrase,git.git_commit,0.0,0.0,0.0,0.0000,0.0000,11,4,771,0.0,3,1050,0.0,2,1402,0.0,git.git_commit git.git_status git.git_add git.git_checkout git.git_show git.git_branch git.git_diff github.get_teams git.git_reset git.git_log diff --git a/experiments/mcp-tool-search/results/decision_heuristic.json b/experiments/mcp-tool-search/results/decision_heuristic.json new file mode 100644 index 00000000..50177ce7 --- /dev/null +++ b/experiments/mcp-tool-search/results/decision_heuristic.json @@ -0,0 +1,6969 @@ +{ + "summary": { + "n": 150, + "index_s": 0.00289805900013107, + "rss_mb": 0.09765625, + "p50_ms": 0.8133175000466508, + "p95_ms": 1.0684915500291936, + "recall@1": 0.8533333333333334, + "recall@1_lo": 0.7933333333333333, + "recall@1_hi": 0.9066666666666666, + "recall@3": 0.9733333333333334, + "recall@3_lo": 0.9466666666666667, + "recall@3_hi": 0.9933333333333333, + "recall@5": 0.9866666666666667, + "recall@5_lo": 0.9666666666666667, + "recall@5_hi": 1.0, + "mrr@10": 0.9111111111111111, + "mrr@10_lo": 0.8744444444444444, + "mrr@10_hi": 0.946111111111111, + "ndcg@10": 0.9233077080171845, + "ndcg@10_lo": 0.894350781004159, + "ndcg@10_hi": 0.9506941957337339 + }, + "meta": { + "name": "decision_heuristic", + "needs_model": false, + "needs_api": false, + "notes": "rule-based router (identifier \u2192 name match, else weighted BM25); not a trained decision model" + }, + "rows": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "github.issue_read", + "github.issue_write", + "github.add_issue_comment", + "github.update_issue_comment", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "git.git_create_branch", + "github.assign_copilot_to_issue" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 388, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 598, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 955, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "kind": "verb_product", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_create_issue", + "jira.jira_create_sprint", + "jira.jira_assign_issue", + "jira.jira_list_projects", + "jira.jira_update_issue", + "jira.jira_get_issue", + "jira.jira_search_issues", + "jira.jira_add_comment", + "jira.jira_add_worklog", + "jira.jira_transition_issue" + ], + "hit1": "jira.jira_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 312, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "kind": "verb_product", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "ranking": [ + "slack.slack_post_message", + "slack.slack_add_reaction", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_create_channel", + "slack.slack_search_messages", + "slack.slack_upload_file", + "slack.slack_list_users", + "slack.slack_get_users", + "slack.slack_list_channels" + ], + "hit1": "slack.slack_post_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "kind": "resource", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "ranking": [ + "github.list_pull_requests", + "github.search_pull_requests", + "gitlab.gitlab_list_merge_requests", + "github.merge_pull_request", + "github.pull_request_read", + "github.create_pull_request", + "github.update_pull_request", + "github.update_pull_request_branch", + "github.pull_request_review_write", + "bitbucket.bitbucket_list_prs" + ], + "hit1": "github.list_pull_requests", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1035, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "ranking": [ + "github.create_pull_request", + "github.merge_pull_request", + "github.pull_request_read", + "github.update_pull_request", + "github.pull_request_review_write", + "github.update_pull_request_branch", + "github.request_copilot_review", + "github.search_pull_requests", + "github.list_pull_requests", + "git.git_checkout" + ], + "hit1": "github.create_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 368, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 653, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1052, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "kind": "verb_product", + "gold": [ + "postgres.query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "postgres.explain_query", + "postgres.list_schemas", + "postgres.list_tables", + "postgres.describe_table", + "mysql.mysql_query", + "sqlite.sqlite_query", + "salesforce.salesforce_query", + "prometheus.prom_query", + "docker.docker_run" + ], + "hit1": "postgres.query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 167, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 590, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "kind": "paraphrase", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "ranking": [ + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "filesystem.write_file", + "filesystem.edit_file", + "filesystem.get_file_info", + "filesystem.move_file", + "brave_search.brave_local_search", + "github.delete_file", + "gitlab.gitlab_get_file" + ], + "hit1": "filesystem.read_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 782, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "kind": "paraphrase", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "ranking": [ + "filesystem.search_files", + "google_drive.gdrive_list_folder", + "slack.list_files", + "github.push_files", + "filesystem.move_file", + "filesystem.list_directory", + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree", + "filesystem.read_file", + "filesystem.read_text_file" + ], + "hit1": "filesystem.search_files", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.16666666666666666, + "ndcg@10": 0.43476305489842704, + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 473, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 730, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 708, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "ranking": [ + "memory.read_graph", + "memory.create_entities", + "memory.delete_relations", + "memory.create_relations", + "memory.add_observations", + "memory.open_nodes", + "memory.search_nodes", + "memory.delete_entities", + "memory.delete_observations", + "time.get_current_time" + ], + "hit1": "memory.read_graph", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.5338931479009518, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 154, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 261, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 524, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "kind": "verb_product", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "ranking": [ + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "filesystem.list_directory_with_sizes", + "jira.jira_search_issues", + "github.update_pull_request_branch", + "memory.search_nodes", + "puppeteer.puppeteer_screenshot", + "puppeteer.puppeteer_navigate", + "puppeteer.puppeteer_click", + "puppeteer.puppeteer_fill" + ], + "hit1": "brave_search.brave_web_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "kind": "verb_product", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "ranking": [ + "fetch.fetch", + "github.get_file_contents", + "playwright.browser_navigate", + "filesystem.read_file", + "puppeteer.puppeteer_navigate", + "git.git_add", + "slack.slack_set_channel_topic", + "github.get_repository_tree", + "git.git_status", + "playwright.browser_snapshot" + ], + "hit1": "fetch.fetch", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 372, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "kind": "identifier", + "gold": [ + "git.git_status" + ], + "alts": [], + "ranking": [ + "git.git_status", + "git.git_show", + "git.git_diff", + "git.git_checkout", + "git.git_branch", + "git.git_commit", + "git.git_add", + "git.git_log", + "git.git_diff_unstaged", + "git.git_diff_staged" + ], + "hit1": "git.git_status", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 304, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "kind": "paraphrase", + "gold": [ + "git.git_commit" + ], + "alts": [], + "ranking": [ + "git.git_commit", + "github.get_commit", + "git.git_diff_staged", + "git.git_reset", + "git.git_log", + "git.git_show", + "github.push_files" + ], + "hit1": "git.git_commit", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_issue", + "linear.linear_update_issue", + "linear.linear_assign_issue", + "linear.linear_get_issue", + "linear.linear_create_comment", + "linear.linear_create_project", + "linear.linear_list_teams", + "linear.linear_list_projects", + "linear.linear_list_issues", + "linear.linear_search_issues" + ], + "hit1": "linear.linear_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 340, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "kind": "paraphrase", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_project", + "linear.linear_create_issue", + "linear.linear_update_issue", + "linear.linear_list_teams", + "linear.linear_list_projects", + "linear.linear_list_issues", + "linear.linear_search_issues", + "linear.linear_create_comment", + "linear.linear_assign_issue", + "linear.linear_get_issue" + ], + "hit1": "linear.linear_create_project", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 352, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 432, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 769, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "ranking": [ + "github.update_issue_comment", + "github.add_issue_comment", + "github.issue_read", + "github.issue_write", + "github.create_issue", + "github.assign_copilot_to_issue", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "git.git_diff" + ], + "hit1": "github.update_issue_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 601, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 938, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "kind": "ambiguous", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "ranking": [ + "github.merge_pull_request", + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_get_merge_request", + "github.pull_request_read", + "github.create_pull_request", + "github.update_pull_request_branch", + "github.update_pull_request", + "gitlab.gitlab_create_merge_request", + "github.pull_request_review_write", + "github.request_copilot_review" + ], + "hit1": "github.merge_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 523, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1121, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "kind": "verb_product", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "ranking": [ + "github.actions_run_trigger", + "github.actions_get", + "github.actions_list", + "github.actions_get_job_logs", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "git.git_diff", + "git.git_commit", + "git.git_status" + ], + "hit1": "github.actions_run_trigger", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 450, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "kind": "resource", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "ranking": [ + "github.list_dependabot_alerts", + "github.get_dependabot_alert", + "github.list_code_scanning_alerts", + "github.list_secret_scanning_alerts", + "kubernetes.k8s_get_pod", + "redis.redis_get", + "github.get_discussion", + "github.projects_get", + "github.get_label", + "aws_s3.s3_get_object" + ], + "hit1": "github.list_dependabot_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 287, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 445, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 798, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "kind": "verb_product", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_issues", + "sentry.sentry_list_projects", + "sentry.sentry_list_events", + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "github.list_issues", + "github.search_issues", + "gitlab.gitlab_list_issues", + "linear.linear_list_issues", + "linear.linear_search_issues" + ], + "hit1": "sentry.sentry_list_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "ranking": [ + "stripe.stripe_get_customer", + "stripe.stripe_create_customer", + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_list_subscriptions", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "stripe.stripe_create_refund", + "stripe.stripe_create_payment_intent", + "stripe.stripe_list_customers" + ], + "hit1": "stripe.stripe_get_customer", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "kind": "paraphrase", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_refund", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_list_customers", + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_create_product", + "stripe.stripe_create_price" + ], + "hit1": "stripe.stripe_create_refund", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "kind": "verb_product", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "ranking": [ + "gmail.gmail_send_email", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "gmail.gmail_modify_email", + "gmail.gmail_search_emails", + "gmail.gmail_create_draft", + "gmail.gmail_list_labels", + "filesystem.list_directory_with_sizes", + "twilio.twilio_send_sms", + "puppeteer.puppeteer_click" + ], + "hit1": "gmail.gmail_send_email", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "kind": "verb_product", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "ranking": [ + "gmail.gmail_search_emails", + "gmail.gmail_create_draft", + "gmail.gmail_list_labels", + "gmail.gmail_send_email", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "gmail.gmail_modify_email", + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "mysql.mysql_list_tables" + ], + "hit1": "gmail.gmail_search_emails", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "kind": "paraphrase", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_update_file", + "google_drive.gdrive_share_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "slack.slack_upload_file", + "gitlab.gitlab_get_file", + "slack.slack_reply_to_thread", + "figma.figma_get_file" + ], + "hit1": "google_drive.gdrive_update_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 496, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 791, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment", + "github.create_package" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 895, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_list_pods", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_apply", + "kubernetes.k8s_delete", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_pod_logs", + "clickup.clickup_list_tasks" + ], + "hit1": "kubernetes.k8s_list_pods", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_get_pod", + "docker.docker_logs", + "datadog.datadog_list_logs", + "github.actions_get_job_logs", + "git.git_log", + "aws_s3.s3_get_object", + "elasticsearch.es_get_doc", + "gitlab.gitlab_get_file", + "github.get_tag" + ], + "hit1": "kubernetes.k8s_pod_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 834, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "ranking": [ + "docker.docker_run", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_exec", + "docker.docker_logs", + "docker.docker_images", + "docker.docker_build", + "docker.docker_ps", + "github.create_check_run", + "github.get_check_run" + ], + "hit1": "docker.docker_run", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 644, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "kind": "verb_product", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "ranking": [ + "docker.docker_logs", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_run", + "docker.docker_exec", + "docker.docker_images", + "docker.docker_build", + "docker.docker_ps", + "kubernetes.k8s_pod_logs", + "datadog.datadog_list_logs" + ], + "hit1": "docker.docker_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "kind": "ambiguous", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "ranking": [ + "playwright.browser_navigate", + "playwright.browser_tab_new", + "puppeteer.puppeteer_navigate", + "playwright.browser_tab_close", + "playwright.browser_tab_list", + "playwright.browser_click", + "playwright.browser_evaluate", + "playwright.browser_snapshot", + "playwright.browser_press_key", + "playwright.browser_wait_for" + ], + "hit1": "playwright.browser_navigate", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 265, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 536, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "kind": "ambiguous", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "ranking": [ + "playwright.browser_take_screenshot", + "puppeteer.puppeteer_screenshot", + "notion.notion_get_page", + "notion.notion_update_page", + "confluence.confluence_get_page", + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "notion.notion_create_page", + "playwright.browser_snapshot", + "playwright.browser_press_key" + ], + "hit1": "playwright.browser_take_screenshot", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 711, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_get_merge_request", + "gitlab.gitlab_create_merge_request", + "github.merge_pull_request", + "gitlab.gitlab_list_merge_requests", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "git.git_create_branch", + "git.git_diff" + ], + "hit1": "gitlab.gitlab_merge_merge_request", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 792, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "kind": "verb_product", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "ranking": [ + "notion.notion_search", + "notion.notion_get_page", + "notion.notion_get_block", + "notion.notion_query_database", + "notion.notion_create_database", + "notion.notion_update_page", + "notion.notion_list_users", + "notion.notion_get_comments", + "notion.notion_append_block", + "notion.notion_create_page" + ], + "hit1": "notion.notion_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "kind": "verb_product", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "ranking": [ + "notion.notion_get_page", + "notion.notion_update_page", + "notion.notion_create_page", + "notion.notion_get_comments", + "notion.notion_create_database", + "notion.notion_search", + "notion.notion_get_block", + "notion.notion_query_database", + "notion.notion_list_users", + "notion.notion_append_block" + ], + "hit1": "notion.notion_get_page", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "kind": "verb_product", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "ranking": [ + "notion.notion_query_database", + "notion.notion_create_database", + "notion.notion_create_page", + "notion.notion_search", + "notion.notion_get_page", + "notion.notion_get_block", + "notion.notion_update_page", + "notion.notion_list_users", + "notion.notion_get_comments", + "notion.notion_append_block" + ], + "hit1": "notion.notion_query_database", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 279, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_search", + "zendesk.zendesk_list_tickets", + "jira.jira_create_issue", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment" + ], + "hit1": "zendesk.zendesk_create_ticket", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 856, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_create_deal", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_search_contacts", + "github.create_gist", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.create_pull_request" + ], + "hit1": "hubspot.hubspot_create_contact", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 904, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "kind": "verb_product", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_convert_lead", + "salesforce.salesforce_create_lead", + "salesforce.salesforce_create_record", + "salesforce.salesforce_query", + "salesforce.salesforce_update_record", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_list_opportunities", + "salesforce.salesforce_get_record", + "github.create_gist", + "github.create_milestone" + ], + "hit1": "salesforce.salesforce_convert_lead", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 787, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "kind": "identifier", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_query", + "postgres.query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "prometheus.prom_query", + "docker.docker_run", + "postgres.explain_query", + "prometheus.prom_query_range", + "datadog.datadog_query_metrics", + "notion.notion_query_database" + ], + "hit1": "salesforce.salesforce_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "kind": "verb_product", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "ranking": [ + "shopify.shopify_list_orders", + "shopify.shopify_list_products", + "shopify.shopify_list_customers", + "shopify.shopify_get_product", + "shopify.shopify_create_product", + "shopify.shopify_update_product", + "shopify.shopify_get_order", + "shopify.shopify_create_order", + "clickup.clickup_list_tasks", + "trello.trello_list_cards" + ], + "hit1": "shopify.shopify_list_orders", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment" + ], + "hit1": "pagerduty.pagerduty_create_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 838, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "kind": "paraphrase", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_list_oncalls", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "twilio.twilio_make_call", + "filesystem.list_allowed_directories", + "trello.trello_add_comment", + "notion.notion_get_comments" + ], + "hit1": "pagerduty.pagerduty_list_oncalls", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "kind": "verb_product", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "ranking": [ + "datadog.datadog_query_metrics", + "datadog.datadog_list_monitors", + "datadog.datadog_create_monitor", + "datadog.datadog_list_logs", + "datadog.datadog_get_monitor", + "prometheus.prom_list_metrics", + "prometheus.prom_query", + "postgres.query", + "postgres.explain_query", + "salesforce.salesforce_query" + ], + "hit1": "datadog.datadog_query_metrics", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "kind": "verb_product", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "ranking": [ + "datadog.datadog_list_logs", + "datadog.datadog_list_monitors", + "datadog.datadog_get_monitor", + "datadog.datadog_create_monitor", + "datadog.datadog_query_metrics", + "docker.docker_logs", + "kubernetes.k8s_pod_logs", + "github.actions_get_job_logs", + "brave_search.brave_web_search", + "brave_search.brave_local_search" + ], + "hit1": "datadog.datadog_list_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 877, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_list_buckets", + "aws_s3.s3_list_objects", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "github.projects_list", + "github.list_milestones", + "github.list_hooks", + "github.list_environments" + ], + "hit1": "aws_s3.s3_list_buckets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 318, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "kind": "paraphrase", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_put_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_get_object", + "slack.slack_upload_file", + "github.assign_copilot_to_issue", + "slack.slack_reply_to_thread", + "github.add_comment_to_pending_review", + "slack.slack_add_reaction", + "twilio.twilio_send_sms", + "gmail.gmail_send_email" + ], + "hit1": "aws_s3.s3_put_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 950, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns" + ], + "hit1": "cloudflare.cf_purge_cache", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "cloudflare.cf_list_zones", + "cloudflare.cf_purge_cache", + "salesforce.salesforce_create_record", + "salesforce.salesforce_update_record", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_get_record", + "git.git_commit", + "github.create_gist" + ], + "hit1": "cloudflare.cf_create_dns", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_deployments", + "vercel.vercel_list_projects", + "vercel.vercel_get_deployment", + "vercel.vercel_create_project", + "kubernetes.k8s_list_deployments", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_services", + "aws_s3.s3_list_buckets", + "github.projects_list" + ], + "hit1": "vercel.vercel_list_deployments", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 579, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "ranking": [ + "supabase.supabase_insert", + "supabase.supabase_select", + "supabase.supabase_update", + "supabase.supabase_invoke_fn", + "supabase.supabase_list_tables", + "mongodb.mongo_insert", + "playwright.browser_type" + ], + "hit1": "supabase.supabase_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "kind": "paraphrase", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "ranking": [ + "figma.figma_get_images", + "figma.figma_get_file", + "figma.figma_get_file_nodes", + "figma.figma_get_comments", + "figma.figma_post_comment", + "docker.docker_images", + "elasticsearch.es_get_doc", + "aws_s3.s3_get_object", + "memory.delete_observations", + "docker.docker_logs" + ], + "hit1": "figma.figma_get_images", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "kind": "verb_product", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "ranking": [ + "twilio.twilio_send_sms", + "twilio.twilio_list_messages", + "twilio.twilio_make_call", + "filesystem.list_directory_with_sizes", + "gmail.gmail_send_email", + "puppeteer.puppeteer_click", + "puppeteer.puppeteer_fill", + "discord.discord_send_message", + "puppeteer.puppeteer_screenshot", + "jira.jira_search_issues" + ], + "hit1": "twilio.twilio_send_sms", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "kind": "verb_product", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "ranking": [ + "okta.okta_get_user", + "okta.okta_deactivate_user", + "okta.okta_create_user", + "okta.okta_list_users", + "linear.create_user", + "linear.get_user", + "linear.update_user", + "linear.delete_user", + "slack.slack_get_user_profile", + "asana.asana_create_task" + ], + "hit1": "okta.okta_get_user", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 656, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query", + "prometheus.prom_query_range", + "prometheus.prom_list_metrics", + "postgres.query", + "postgres.explain_query", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "datadog.datadog_query_metrics", + "notion.notion_query_database" + ], + "hit1": "prometheus.prom_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "kind": "verb_product", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "ranking": [ + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "grafana.grafana_create_annotation", + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "confluence.confluence_search", + "github.search_issues", + "github.search_users", + "github.search_orgs", + "elasticsearch.es_search" + ], + "hit1": "grafana.grafana_search_dashboards", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 690, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "kind": "verb_product", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "ranking": [ + "circleci.circleci_trigger_pipeline", + "circleci.circleci_get_pipeline", + "circleci.circleci_list_pipelines", + "gitlab.gitlab_create_pipeline", + "github.actions_run_trigger", + "gitlab.create_pipeline", + "gitlab.get_pipeline", + "gitlab.update_pipeline", + "gitlab.delete_pipeline", + "hubspot.hubspot_create_deal" + ], + "hit1": "circleci.circleci_trigger_pipeline", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "kind": "identifier", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "ranking": [ + "terraform.tf_plan", + "terraform.tf_apply", + "terraform.tf_state_list", + "postgres.explain_query", + "docker.docker_run", + "github.create_check_run", + "github.get_check_run", + "github.update_check_run", + "github.delete_check_run", + "github.actions_run_trigger" + ], + "hit1": "terraform.tf_plan", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "kind": "verb_product", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "ranking": [ + "asana.asana_get_task", + "asana.asana_create_task", + "asana.asana_update_task", + "asana.asana_create_project", + "asana.asana_list_tasks", + "clickup.clickup_get_task", + "clickup.clickup_create_task", + "clickup.clickup_update_task", + "okta.okta_create_user", + "github.create_issue" + ], + "hit1": "asana.asana_get_task", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "ranking": [ + "trello.trello_create_card", + "trello.trello_add_comment", + "trello.trello_create_board", + "trello.trello_list_cards", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment", + "github.create_package", + "github.create_artifact" + ], + "hit1": "trello.trello_create_card", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 394, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 884, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_get_task", + "clickup.clickup_create_task", + "clickup.clickup_update_task", + "clickup.clickup_list_tasks", + "puppeteer.puppeteer_click", + "playwright.browser_click", + "asana.asana_get_task", + "asana.asana_create_task", + "asana.asana_update_task", + "github.create_gist" + ], + "hit1": "clickup.clickup_get_task", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "kind": "verb_product", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "ranking": [ + "confluence.confluence_create_page", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "confluence.confluence_search", + "notion.notion_get_page", + "notion.notion_create_page", + "notion.notion_update_page", + "playwright.browser_take_screenshot", + "notion.notion_get_comments", + "playwright.browser_press_key" + ], + "hit1": "confluence.confluence_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "kind": "verb_product", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "discord.discord_add_reaction", + "discord.discord_get_channel", + "discord.discord_list_channels", + "discord.discord_list_members", + "slack.slack_post_message", + "gmail.gmail_send_email", + "twilio.twilio_send_sms", + "gmail.gmail_get_email", + "gmail.gmail_trash_email" + ], + "hit1": "discord.discord_send_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_search", + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "confluence.confluence_search", + "github.search_issues", + "github.search_users", + "github.search_orgs", + "github.search_commits" + ], + "hit1": "elasticsearch.es_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "redis.redis_set", + "redis.redis_delete", + "redis.redis_keys", + "playwright.browser_press_key", + "jira.jira_get_issue", + "aws_s3.s3_get_object", + "figma.figma_get_file", + "figma.figma_get_comments", + "figma.figma_get_file_nodes" + ], + "hit1": "redis.redis_get", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 589, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_find", + "mongodb.mongo_insert", + "mongodb.mongo_update", + "mongodb.mongo_delete", + "mongodb.mongo_list_collections" + ], + "hit1": "mongodb.mongo_find", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "kind": "verb_product", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "ranking": [ + "sqlite.sqlite_list_tables", + "postgres.list_tables", + "mysql.mysql_list_tables", + "supabase.supabase_list_tables", + "sqlite.sqlite_describe_table", + "sqlite.sqlite_query", + "postgres.list_schemas", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments" + ], + "hit1": "sqlite.sqlite_list_tables", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 240, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 514, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "kind": "verb_product", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "ranking": [ + "postgres.describe_table", + "sqlite.sqlite_describe_table", + "postgres.query", + "postgres.list_tables", + "postgres.list_schemas", + "postgres.explain_query", + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update" + ], + "hit1": "postgres.describe_table", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 272, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "github.issue_read", + "github.issue_write", + "github.add_issue_comment", + "github.update_issue_comment", + "github.assign_copilot_to_issue", + "bitbucket.bitbucket_create_issue", + "sentry.sentry_get_issue", + "gitlab.gitlab_create_issue", + "jira.jira_assign_issue" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 386, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 596, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1019, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "ranking": [ + "github.create_issue", + "github.issue_read", + "bitbucket.bitbucket_create_issue", + "github.issue_write", + "sentry.sentry_get_issue", + "gitlab.gitlab_create_issue", + "jira.jira_assign_issue", + "jira.jira_get_issue", + "linear.linear_create_issue", + "gitlab.gitlab_get_issue" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7349350438281765, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 340, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 546, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 961, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "kind": "identifier", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "ranking": [ + "filesystem.create_directory", + "filesystem.list_directory", + "filesystem.directory_tree", + "filesystem.list_directory_with_sizes", + "filesystem.get_file_info", + "filesystem.list_allowed_directories", + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "filesystem.write_file" + ], + "hit1": "filesystem.create_directory", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 607, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "kind": "paraphrase", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "ranking": [ + "sequential_thinking.sequentialthinking" + ], + "hit1": "sequential_thinking.sequentialthinking", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "kind": "paraphrase", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "ranking": [ + "time.convert_time", + "time.get_current_time", + "playwright.browser_wait_for", + "filesystem.list_allowed_directories", + "slack.slack_create_channel", + "google_calendar.gcal_list_events", + "grafana.grafana_create_annotation", + "jira.jira_add_worklog", + "elasticsearch.es_index_doc", + "github.create_milestone" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8597186998521972, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 783, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "kind": "verb_product", + "gold": [ + "time.convert_time" + ], + "alts": [], + "ranking": [ + "time.convert_time", + "time.get_current_time", + "salesforce.salesforce_convert_lead", + "playwright.browser_wait_for", + "git.git_diff", + "memory.create_relations", + "google_calendar.gcal_list_events", + "grafana.grafana_create_annotation", + "jira.jira_add_worklog" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "kind": "verb_product", + "gold": [ + "github.star_repository" + ], + "alts": [], + "ranking": [ + "github.star_repository", + "git.git_branch", + "git.git_show", + "git.git_checkout", + "git.git_status", + "git.git_diff", + "git.git_commit", + "git.git_add", + "git.git_reset", + "git.git_diff_unstaged" + ], + "hit1": "github.star_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 309, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 604, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "kind": "resource", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "ranking": [ + "github.fork_repository", + "github.delete_repository", + "github.star_repository", + "github.unstar_repository", + "github.create_repository", + "github.list_repository_collaborators", + "github.get_repository_tree", + "github.list_dependabot_alerts", + "github.list_branches", + "github.list_releases" + ], + "hit1": "github.fork_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 792, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "kind": "verb_product", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "ranking": [ + "github.get_latest_release", + "github.get_release_by_tag", + "git.git_commit", + "git.git_status", + "git.git_add", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "git.git_diff", + "git.git_reset" + ], + "hit1": "github.get_latest_release", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "kind": "resource", + "gold": [ + "github.create_gist" + ], + "alts": [], + "ranking": [ + "github.create_gist", + "github.update_gist", + "github.get_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment", + "github.create_package", + "github.create_artifact", + "github.create_runner", + "gitlab.create_pipeline" + ], + "hit1": "github.create_gist", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 912, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "kind": "verb_product", + "gold": [ + "github.search_code" + ], + "alts": [], + "ranking": [ + "github.search_code", + "github.get_code_scanning_alert", + "github.list_code_scanning_alerts", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "git.git_diff", + "git.git_commit", + "git.git_status", + "git.git_add" + ], + "hit1": "github.search_code", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 406, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 697, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "kind": "verb_product", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "ranking": [ + "github.list_notifications", + "github.mark_all_notifications_read", + "git.git_branch", + "git.git_checkout", + "git.git_show", + "git.git_diff", + "git.git_commit", + "git.git_status", + "git.git_add", + "git.git_reset" + ], + "hit1": "github.list_notifications", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 169, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 585, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "kind": "verb_product", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "ranking": [ + "github.assign_copilot_to_issue", + "jira.jira_assign_issue", + "linear.linear_assign_issue", + "github.request_copilot_review", + "jira.jira_update_issue", + "jira.jira_transition_issue", + "github.create_issue", + "github.add_issue_comment", + "slack.slack_reply_to_thread", + "github.issue_read" + ], + "hit1": "github.assign_copilot_to_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 434, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 928, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "kind": "resource", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "ranking": [ + "github.list_secret_scanning_alerts", + "github.get_secret_scanning_alert", + "github.list_code_scanning_alerts", + "github.list_dependabot_alerts", + "github.get_code_scanning_alert", + "kubernetes.k8s_get_pod", + "redis.redis_get", + "github.get_discussion", + "github.projects_get", + "github.get_label" + ], + "hit1": "github.list_secret_scanning_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 830, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "ranking": [ + "github.create_or_update_file", + "github.delete_file", + "git.git_diff", + "git.git_add", + "github.get_file_contents", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "github.get_file_blame", + "git.git_create_branch" + ], + "hit1": "github.create_or_update_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 347, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 519, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 869, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "kind": "resource", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "ranking": [ + "github.list_repository_collaborators", + "github.list_commits", + "trello.trello_list_cards", + "pagerduty.pagerduty_list_oncalls", + "github.get_discussion_comments", + "github.update_issue_comment", + "trello.trello_add_comment", + "github.request_copilot_review", + "notion.notion_get_comments", + "playwright.browser_press_key" + ], + "hit1": "github.list_repository_collaborators", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 278, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 433, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 816, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "kind": "verb_product", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "ranking": [ + "slack.slack_reply_to_thread", + "slack.slack_get_thread_replies", + "slack.slack_list_channels", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_create_channel", + "slack.slack_search_messages", + "slack.slack_upload_file", + "slack.slack_list_users", + "slack.slack_post_message" + ], + "hit1": "slack.slack_reply_to_thread", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "kind": "verb_product", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "ranking": [ + "slack.slack_add_reaction", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_create_channel", + "slack.slack_search_messages", + "slack.slack_upload_file", + "slack.slack_list_users", + "slack.slack_post_message", + "slack.slack_get_users", + "slack.slack_list_channels" + ], + "hit1": "slack.slack_add_reaction", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 357, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "kind": "verb_product", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "ranking": [ + "slack.slack_search_messages", + "slack.slack_get_channel_history", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_create_channel", + "slack.slack_upload_file", + "slack.slack_list_users", + "slack.slack_post_message", + "slack.slack_get_users", + "slack.slack_list_channels" + ], + "hit1": "slack.slack_search_messages", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 689, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "kind": "verb_product", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "ranking": [ + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_create_channel", + "slack.slack_get_channel_history", + "slack.slack_set_channel_topic", + "slack.slack_post_message", + "slack.slack_upload_file", + "slack.slack_add_reaction", + "slack.slack_search_messages", + "slack.slack_list_users" + ], + "hit1": "slack.slack_join_channel", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 719, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "kind": "verb_product", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_transition_issue", + "jira.jira_assign_issue", + "jira.jira_get_issue", + "jira.jira_update_issue", + "jira.jira_create_issue", + "jira.jira_add_comment", + "jira.jira_add_worklog", + "jira.jira_list_projects", + "jira.jira_create_sprint", + "jira.jira_search_issues" + ], + "hit1": "jira.jira_transition_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "kind": "identifier", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "ranking": [ + "jira.jira_search_issues", + "jira.jira_assign_issue", + "jira.jira_list_projects", + "jira.jira_create_sprint", + "jira.jira_update_issue", + "jira.jira_get_issue", + "jira.jira_add_comment", + "jira.jira_add_worklog", + "jira.jira_transition_issue", + "jira.jira_create_issue" + ], + "hit1": "jira.jira_search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "kind": "verb_product", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "ranking": [ + "jira.jira_add_worklog", + "jira.jira_add_comment", + "jira.jira_transition_issue", + "jira.jira_assign_issue", + "jira.jira_list_projects", + "jira.jira_create_sprint", + "jira.jira_update_issue", + "jira.jira_get_issue", + "jira.jira_search_issues", + "jira.jira_create_issue" + ], + "hit1": "jira.jira_add_worklog", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "kind": "verb_product", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "ranking": [ + "linear.linear_list_teams", + "linear.linear_list_projects", + "linear.linear_list_issues", + "linear.linear_update_issue", + "linear.linear_create_project", + "linear.linear_create_issue", + "linear.linear_search_issues", + "linear.linear_create_comment", + "linear.linear_assign_issue", + "linear.linear_get_issue" + ], + "hit1": "linear.linear_list_teams", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "ranking": [ + "linear.linear_create_comment", + "linear.linear_update_issue", + "linear.linear_create_issue", + "linear.linear_assign_issue", + "linear.linear_get_issue", + "linear.create_comment", + "linear.get_comment", + "linear.update_comment", + "linear.delete_comment", + "linear.linear_list_teams" + ], + "hit1": "linear.linear_create_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 334, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 464, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 760, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "kind": "paraphrase", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "ranking": [ + "filesystem.move_file", + "brave_search.brave_local_search", + "filesystem.read_file", + "filesystem.read_text_file", + "figma.figma_get_file", + "filesystem.read_media_file", + "filesystem.write_file", + "slack.create_file", + "slack.get_file", + "slack.update_file" + ], + "hit1": "filesystem.move_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "kind": "paraphrase", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "ranking": [ + "filesystem.search_files", + "slack.list_files", + "github.push_files", + "github.get_release_by_tag", + "filesystem.move_file", + "filesystem.list_directory", + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree", + "redis.redis_keys", + "filesystem.read_file" + ], + "hit1": "filesystem.search_files", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 280, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 433, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "kind": "paraphrase", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "ranking": [ + "filesystem.edit_file", + "playwright.browser_tab_new", + "filesystem.list_directory_with_sizes", + "filesystem.write_file", + "slack.create_file", + "slack.get_file", + "slack.update_file", + "slack.delete_file", + "figma.figma_get_file", + "filesystem.move_file" + ], + "hit1": "filesystem.edit_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 394, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "ranking": [ + "memory.read_graph", + "memory.delete_relations", + "memory.create_entities", + "memory.create_relations", + "memory.add_observations", + "memory.delete_entities", + "memory.open_nodes", + "memory.search_nodes", + "memory.delete_observations", + "filesystem.read_file" + ], + "hit1": "memory.read_graph", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 259, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 529, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "kind": "verb_product", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "memory.open_nodes", + "figma.figma_get_file_nodes", + "memory.create_entities", + "memory.create_relations", + "memory.add_observations", + "memory.delete_entities", + "memory.delete_observations", + "memory.delete_relations", + "memory.read_graph" + ], + "hit1": "memory.search_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 546, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "kind": "verb_product", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "ranking": [ + "git.git_diff_unstaged", + "git.git_diff", + "git.git_diff_staged", + "git.git_show", + "git.git_status", + "git.git_checkout", + "git.git_branch", + "git.git_commit", + "git.git_add", + "git.git_log" + ], + "hit1": "git.git_diff_unstaged", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 602, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "kind": "identifier", + "gold": [ + "git.git_add" + ], + "alts": [], + "ranking": [ + "filesystem.search_files", + "filesystem.list_directory_with_sizes", + "git.git_add", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "git.git_diff", + "git.git_commit", + "git.git_status", + "github.push_files" + ], + "hit1": "filesystem.search_files", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 336, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 677, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "kind": "verb_product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "ranking": [ + "shopify.shopify_create_product", + "shopify.shopify_get_product", + "shopify.shopify_update_product", + "shopify.shopify_create_order", + "shopify.shopify_list_products", + "shopify.shopify_list_orders", + "shopify.shopify_get_order", + "shopify.shopify_list_customers", + "stripe.stripe_create_product", + "stripe.stripe_create_price" + ], + "hit1": "shopify.shopify_create_product", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "kind": "verb_product", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "ranking": [ + "stripe.stripe_cancel_subscription", + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_list_customers", + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "stripe.stripe_list_subscriptions", + "stripe.stripe_create_refund" + ], + "hit1": "stripe.stripe_cancel_subscription", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "kind": "resource", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_refund", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment", + "github.create_package", + "github.create_artifact", + "github.create_runner" + ], + "hit1": "stripe.stripe_create_payment_intent", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 897, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "kind": "verb_product", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "ranking": [ + "datadog.datadog_get_monitor", + "datadog.datadog_create_monitor", + "datadog.datadog_list_monitors", + "datadog.datadog_list_logs", + "datadog.datadog_query_metrics", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment", + "github.create_package" + ], + "hit1": "datadog.datadog_get_monitor", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 927, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "kind": "paraphrase", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "ranking": [ + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "sentry.sentry_list_events", + "sentry.sentry_list_projects", + "sentry.sentry_list_issues", + "pagerduty.pagerduty_resolve_incident", + "github.issue_read", + "jira.jira_assign_issue", + "github.issue_write", + "jira.jira_get_issue" + ], + "hit1": "sentry.sentry_get_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 817, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_apply", + "kubernetes.k8s_delete", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_pod_logs", + "vercel.vercel_get_deployment" + ], + "hit1": "kubernetes.k8s_scale_deployment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 263, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_apply", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_delete", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_pod_logs", + "terraform.tf_apply" + ], + "hit1": "kubernetes.k8s_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 253, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "ranking": [ + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_run", + "docker.docker_exec", + "docker.docker_images", + "docker.docker_build", + "docker.docker_ps", + "kubernetes.k8s_pod_logs" + ], + "hit1": "docker.docker_stop", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "kind": "verb_product", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "ranking": [ + "docker.docker_build", + "docker.docker_run", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_images", + "docker.docker_ps", + "docker.docker_logs", + "docker.docker_exec", + "filesystem.read_media_file" + ], + "hit1": "docker.docker_build", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "kind": "verb_product", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "ranking": [ + "playwright.browser_click", + "puppeteer.puppeteer_click", + "playwright.browser_type", + "playwright.browser_take_screenshot", + "playwright.browser_snapshot", + "playwright.browser_navigate", + "playwright.browser_fill_form", + "playwright.browser_evaluate", + "playwright.browser_tab_list", + "playwright.browser_tab_new" + ], + "hit1": "playwright.browser_click", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 566, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "kind": "paraphrase", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "ranking": [ + "playwright.browser_type", + "playwright.browser_navigate", + "playwright.browser_wait_for", + "playwright.browser_tab_close", + "playwright.browser_tab_list", + "playwright.browser_tab_new", + "playwright.browser_snapshot", + "playwright.browser_click", + "playwright.browser_evaluate", + "playwright.browser_fill_form" + ], + "hit1": "playwright.browser_type", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 554, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_list_projects", + "git.git_branch", + "git.git_checkout", + "git.git_show", + "git.git_diff", + "git.git_commit", + "git.git_status", + "git.git_add", + "github.projects_list", + "git.git_reset" + ], + "hit1": "gitlab.gitlab_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 304, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 599, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "kind": "verb_product", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "ranking": [ + "bitbucket.bitbucket_create_pr", + "bitbucket.bitbucket_list_prs", + "bitbucket.bitbucket_create_issue", + "bitbucket.bitbucket_list_repos", + "github.merge_pull_request", + "github.pull_request_read", + "github.create_pull_request", + "github.update_pull_request", + "github.update_pull_request_branch", + "github.pull_request_review_write" + ], + "hit1": "bitbucket.bitbucket_create_pr", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 471, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1091, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "kind": "verb_product", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "ranking": [ + "gmail.gmail_create_draft", + "gmail.gmail_search_emails", + "gmail.gmail_list_labels", + "gmail.gmail_send_email", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "gmail.gmail_modify_email", + "github.create_pull_request", + "github.create_gist", + "github.create_milestone" + ], + "hit1": "gmail.gmail_create_draft", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 835, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "kind": "verb_product", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_share_file", + "google_drive.gdrive_update_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "gitlab.gitlab_get_file", + "figma.figma_get_file", + "google_calendar.gcal_list_calendars", + "filesystem.write_file" + ], + "hit1": "google_drive.gdrive_share_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "kind": "verb_product", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_list_events", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_list_calendars", + "kubernetes.k8s_get_events", + "sentry.sentry_list_events", + "google_drive.gdrive_list_folder", + "google_drive.gdrive_update_file", + "google_drive.gdrive_share_file" + ], + "hit1": "google_calendar.gcal_list_events", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 784, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "kubernetes.k8s_delete", + "redis.redis_delete", + "aws_s3.s3_delete_object", + "github.delete_repository", + "github.delete_milestone" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_projects", + "vercel.vercel_list_deployments", + "vercel.vercel_get_deployment", + "vercel.vercel_create_project", + "github.projects_list", + "linear.linear_list_projects", + "jira.jira_list_projects", + "sentry.sentry_list_projects", + "gitlab.gitlab_list_projects", + "github.projects_get" + ], + "hit1": "vercel.vercel_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "ranking": [ + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update", + "supabase.supabase_invoke_fn", + "supabase.supabase_list_tables", + "elasticsearch.es_get_doc", + "aws_s3.s3_get_object", + "memory.delete_observations", + "docker.docker_logs", + "kubernetes.k8s_pod_logs" + ], + "hit1": "supabase.supabase_select", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "kind": "verb_product", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "figma.figma_get_file", + "figma.figma_get_comments", + "figma.figma_get_images", + "figma.figma_get_file_nodes", + "linear.create_comment", + "linear.get_comment", + "linear.update_comment", + "linear.delete_comment", + "notion.create_comment" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "kind": "verb_product", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "ranking": [ + "okta.okta_list_users", + "okta.okta_get_user", + "okta.okta_create_user", + "okta.okta_deactivate_user", + "linear.list_users", + "github.search_users", + "slack.slack_list_users", + "notion.notion_list_users", + "notion.list_users_tokens", + "slack.slack_get_users" + ], + "hit1": "okta.okta_list_users", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 306, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query_range", + "prometheus.prom_query", + "prometheus.prom_list_metrics", + "postgres.query", + "postgres.explain_query", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "datadog.datadog_query_metrics", + "notion.notion_query_database" + ], + "hit1": "prometheus.prom_query_range", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "kind": "verb_product", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "ranking": [ + "grafana.grafana_create_annotation", + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment", + "github.create_package", + "github.create_artifact", + "github.create_runner" + ], + "hit1": "grafana.grafana_create_annotation", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 881, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "kind": "identifier", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "ranking": [ + "terraform.tf_apply", + "terraform.tf_plan", + "terraform.tf_state_list", + "kubernetes.k8s_apply" + ], + "hit1": "terraform.tf_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "kind": "verb_product", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "ranking": [ + "asana.asana_list_tasks", + "asana.asana_create_task", + "asana.asana_get_task", + "asana.asana_update_task", + "asana.asana_create_project", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "aws_s3.s3_list_buckets" + ], + "hit1": "asana.asana_list_tasks", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "ranking": [ + "trello.trello_add_comment", + "trello.trello_create_card", + "trello.trello_list_cards", + "trello.trello_create_board", + "github.update_issue_comment", + "figma.figma_post_comment", + "linear.create_comment", + "linear.get_comment", + "linear.update_comment", + "linear.delete_comment" + ], + "hit1": "trello.trello_add_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 422, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_update_task", + "clickup.clickup_get_task", + "clickup.clickup_create_task", + "clickup.clickup_list_tasks", + "puppeteer.puppeteer_click", + "playwright.browser_click", + "asana.asana_update_task", + "asana.asana_get_task", + "asana.asana_create_task", + "mongodb.mongo_update" + ], + "hit1": "clickup.clickup_update_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 761, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "kind": "verb_product", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "ranking": [ + "confluence.confluence_search", + "confluence.confluence_get_page", + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "github.search_issues", + "github.search_users", + "github.search_orgs", + "elasticsearch.es_search" + ], + "hit1": "confluence.confluence_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 731, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_deal", + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_search_contacts", + "github.create_gist", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.create_pull_request" + ], + "hit1": "hubspot.hubspot_create_deal", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 904, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_search", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_create_ticket", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "aws_s3.s3_list_buckets" + ], + "hit1": "zendesk.zendesk_list_tickets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "kind": "verb_product", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "ranking": [ + "discord.discord_list_channels", + "discord.discord_list_members", + "discord.discord_get_channel", + "discord.discord_send_message", + "discord.discord_add_reaction", + "slack.slack_list_channels", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services" + ], + "hit1": "discord.discord_list_channels", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 166, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 625, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "ranking": [ + "redis.redis_set", + "redis.redis_get", + "redis.redis_delete", + "redis.redis_keys", + "playwright.browser_press_key", + "jira.jira_get_issue", + "slack.slack_set_channel_topic", + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "figma.figma_get_file" + ], + "hit1": "redis.redis_set", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert", + "mongodb.mongo_update", + "mongodb.mongo_delete", + "mongodb.mongo_find", + "mongodb.mongo_list_collections", + "supabase.supabase_insert" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "kind": "verb_product", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "postgres.explain_query", + "mysql.mysql_query", + "postgres.list_schemas", + "postgres.list_tables", + "postgres.describe_table", + "mysql.mysql_list_tables", + "sqlite.sqlite_query", + "salesforce.salesforce_query", + "prometheus.prom_query" + ], + "hit1": "postgres.query", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 268, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 543, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_index_doc", + "elasticsearch.es_search", + "elasticsearch.es_get_doc", + "playwright.browser_tab_close", + "github.create_milestone", + "github.get_milestone", + "github.list_milestones", + "github.update_milestone", + "github.delete_milestone", + "github.create_hook" + ], + "hit1": "elasticsearch.es_index_doc", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_put_object", + "okta.okta_get_user", + "asana.asana_get_task", + "aws_s3.s3_list_objects", + "elasticsearch.es_search", + "okta.okta_create_user", + "okta.okta_deactivate_user", + "asana.asana_create_task" + ], + "hit1": "aws_s3.s3_get_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones", + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_dns", + "cloudflare.cf_create_dns", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "aws_s3.s3_list_buckets", + "github.projects_list" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 629, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_create_customer", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "stripe.stripe_create_refund", + "stripe.stripe_get_customer", + "stripe.stripe_list_customers", + "stripe.stripe_list_invoices", + "stripe.stripe_list_subscriptions", + "stripe.stripe_cancel_subscription" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "kind": "verb_product", + "gold": [ + "github.actions_list" + ], + "alts": [], + "ranking": [ + "github.actions_list", + "github.actions_get", + "github.actions_run_trigger", + "github.actions_get_job_logs", + "git.git_branch", + "git.git_checkout", + "git.git_show", + "git.git_diff", + "git.git_commit", + "git.git_status" + ], + "hit1": "github.actions_list", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 433, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 736, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "kind": "verb_product", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "ranking": [ + "github.actions_get_job_logs", + "github.actions_get", + "github.actions_list", + "github.actions_run_trigger", + "git.git_log", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "git.git_diff", + "git.git_commit" + ], + "hit1": "github.actions_get_job_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 450, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 747, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "kind": "verb_product", + "gold": [ + "github.create_repository" + ], + "alts": [], + "ranking": [ + "github.create_repository", + "github.delete_repository", + "github.fork_repository", + "github.star_repository", + "github.unstar_repository", + "github.list_repository_collaborators", + "github.get_repository_tree", + "git.git_commit", + "git.git_status", + "git.git_checkout" + ], + "hit1": "github.create_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 750, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "kind": "verb_product", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "ranking": [ + "git.git_checkout", + "git.git_create_branch", + "git.git_branch", + "git.git_show", + "github.create_branch", + "git.git_diff", + "git.git_commit", + "git.git_status", + "git.git_add", + "git.git_reset" + ], + "hit1": "git.git_checkout", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 666, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "kind": "identifier", + "gold": [ + "git.git_log" + ], + "alts": [], + "ranking": [ + "git.git_log", + "git.git_show", + "git.git_diff", + "git.git_status", + "git.git_checkout", + "git.git_branch", + "git.git_commit", + "git.git_add", + "git.git_diff_unstaged", + "git.git_diff_staged" + ], + "hit1": "git.git_log", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "kind": "verb_product", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "ranking": [ + "postgres.list_schemas", + "postgres.list_tables", + "postgres.query", + "postgres.describe_table", + "postgres.explain_query", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "aws_s3.s3_list_buckets" + ], + "hit1": "postgres.list_schemas", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 158, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 555, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "kind": "verb_product", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "ranking": [ + "memory.open_nodes", + "memory.read_graph", + "memory.search_nodes", + "memory.create_entities", + "memory.delete_relations", + "memory.create_relations", + "memory.add_observations", + "memory.delete_entities", + "memory.delete_observations", + "figma.figma_get_file_nodes" + ], + "hit1": "memory.open_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 260, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 549, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "kind": "paraphrase", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "ranking": [ + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "filesystem.get_file_info", + "filesystem.write_file", + "filesystem.edit_file", + "filesystem.move_file", + "filesystem.list_allowed_directories", + "google_drive.gdrive_get_file", + "gitlab.gitlab_get_file" + ], + "hit1": "filesystem.read_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 462, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 688, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "kind": "paraphrase", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "ranking": [ + "playwright.browser_tab_new", + "filesystem.write_file", + "brave_search.brave_local_search", + "filesystem.edit_file", + "filesystem.read_file", + "filesystem.read_text_file", + "figma.figma_get_file", + "filesystem.read_media_file", + "slack.create_file", + "slack.get_file" + ], + "hit1": "playwright.browser_tab_new", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "kind": "verb_product", + "gold": [ + "github.search_issues" + ], + "alts": [], + "ranking": [ + "github.search_issues", + "github.list_issues", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "git.git_diff", + "git.git_commit", + "git.git_status", + "git.git_add", + "git.git_reset" + ], + "hit1": "github.search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 270, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 670, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "kind": "paraphrase", + "gold": [ + "github.get_me" + ], + "alts": [], + "ranking": [ + "git.git_commit", + "git.git_status", + "git.git_add", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "git.git_diff", + "github.get_teams", + "git.git_reset", + "git.git_log" + ], + "hit1": "git.git_commit", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 11, + "k3_calls": 4, + "k3_tokens": 771, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 3, + "k5_tokens": 1050, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 1.0, + "k10_calls": 2, + "k10_tokens": 1402, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "cost": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.04, + "p95_calls": 1.0, + "success_within_1": 0.9733333333333334, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 242.9, + "p95_tokens": 349.7499999999999, + "total_tokens": 36435, + "tokens_per_successful_query": 242.9, + "total_tokens_successful": 36435 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.02, + "p95_calls": 1.0, + "success_within_1": 0.9866666666666667, + "success_within_2": 0.9933333333333333, + "success_within_3": 1.0, + "mean_tokens": 382.8466666666667, + "p95_tokens": 521.1999999999999, + "total_tokens": 57427, + "tokens_per_successful_query": 382.8466666666667, + "total_tokens_successful": 57427 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.0066666666666666, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 733.46, + "p95_tokens": 952.75, + "total_tokens": 110019, + "tokens_per_successful_query": 733.46, + "total_tokens_successful": 110019 + } + } +} diff --git a/experiments/mcp-tool-search/results/embed_bge_small.csv b/experiments/mcp-tool-search/results/embed_bge_small.csv new file mode 100644 index 00000000..0001345d --- /dev/null +++ b/experiments/mcp-tool-search/results/embed_bge_small.csv @@ -0,0 +1,151 @@ +id,query,kind,hit1,recall@1,recall@3,recall@5,mrr@10,ndcg@10,hit_rank,k3_calls,k3_tokens,k3_success1,k5_calls,k5_tokens,k5_success1,k10_calls,k10_tokens,k10_success1,ranking +q001,Create a GitHub issue,verb_product,github.issue_write,1.0,1.0,1.0,1.0000,0.8597,1,1,375,1.0,1,560,1.0,1,1034,1.0,github.issue_write github.create_issue github.search_issues github.create_repository github.add_issue_comment gitlab.gitlab_create_issue github.assign_copilot_to_issue github.create_gist github.issue_read github.list_issues +q002,Create a Jira ticket,verb_product,jira.jira_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,345,1.0,1,493,1.0,1,902,1.0,jira.jira_create_issue jira.jira_create_sprint jira.create_sprint jira.get_sprint jira.create_board jira.create_version jira.get_board jira.create_webhook jira.create_filter jira.update_sprint +q003,Send a Slack message,verb_product,slack.slack_post_message,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,407,1.0,1,795,1.0,slack.slack_post_message slack.slack_add_reaction slack.slack_reply_to_thread slack.get_reminder slack.slack_upload_file slack.update_reminder slack.slack_create_channel slack.slack_get_thread_replies slack.create_reminder slack.update_pin +q004,List open pull requests,resource,github.list_pull_requests,1.0,1.0,1.0,1.0000,0.9502,1,1,237,1.0,1,477,1.0,1,828,1.0,github.list_pull_requests bitbucket.bitbucket_list_prs github.search_pull_requests github.create_pull_request github.pull_request_read gitlab.list_snippets bitbucket.bitbucket_list_repos github.list_hooks github.list_notifications github.list_secret_scanning_alerts +q005,Open a pull request on GitHub,verb_product,github.create_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,314,1.0,1,547,1.0,1,1094,1.0,github.create_pull_request github.search_pull_requests github.list_pull_requests github.pull_request_review_write github.pull_request_read github.merge_pull_request github.update_pull_request github.update_pull_request_branch github.add_comment_to_pending_review github.get_hook +q006,Run a SQL query against postgres,verb_product,postgres.query,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,295,1.0,1,596,1.0,postgres.query mysql.mysql_query sqlite.sqlite_query salesforce.salesforce_query postgres.explain_query prometheus.prom_query postgres.list_schemas postgres.describe_table prometheus.prom_query_range postgres.list_tables +q007,Read a local file from disk,paraphrase,filesystem.read_text_file,1.0,1.0,1.0,1.0000,0.8597,1,1,223,1.0,1,385,1.0,1,677,1.0,filesystem.read_text_file filesystem.read_file filesystem.read_media_file filesystem.get_file_info github.get_file_contents filesystem.list_directory figma.figma_get_file slack.get_file filesystem.directory_tree figma.figma_get_file_nodes +q008,What files are in this folder,paraphrase,filesystem.list_directory,1.0,1.0,1.0,1.0000,0.9541,1,1,190,1.0,1,303,1.0,1,656,1.0,filesystem.list_directory filesystem.list_directory_with_sizes filesystem.get_file_info google_drive.gdrive_list_folder filesystem.read_file filesystem.directory_tree filesystem.read_media_file github.get_file_contents git.git_add filesystem.search_files +q009,Remember this fact in the knowledge graph,paraphrase,memory.read_graph,0.0,1.0,1.0,0.3333,0.5000,3,1,164,1.0,1,266,1.0,1,585,1.0,memory.read_graph memory.create_relations memory.add_observations memory.open_nodes memory.delete_entities memory.search_nodes memory.delete_relations memory.create_entities sequential_thinking.sequentialthinking memory.delete_observations +q010,Search the web with Brave,verb_product,brave_search.brave_web_search,1.0,1.0,1.0,1.0000,1.0000,1,1,245,1.0,1,356,1.0,1,642,1.0,brave_search.brave_web_search brave_search.brave_local_search grafana.grafana_search_dashboards hubspot.hubspot_search_contacts confluence.confluence_search github.search_pull_requests slack.slack_search_messages github.search_orgs linear.linear_search_issues zendesk.zendesk_search +q011,Fetch the contents of a URL,verb_product,fetch.fetch,1.0,1.0,1.0,1.0000,1.0000,1,1,205,1.0,1,369,1.0,1,643,1.0,fetch.fetch playwright.browser_navigate confluence.confluence_search github.get_file_contents puppeteer.puppeteer_navigate gitlab.get_snippet github.search_pull_requests filesystem.read_file slack.get_file filesystem.get_file_info +q012,Show git status,identifier,git.git_status,1.0,1.0,1.0,1.0000,1.0000,1,1,194,1.0,1,312,1.0,1,696,1.0,git.git_status git.git_show git.git_log git.git_diff_unstaged git.git_diff github.list_code_scanning_alerts github.get_notification_details github.get_milestone github.get_code_scanning_alert github.list_milestones +q013,Commit staged changes,paraphrase,git.git_commit,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,314,1.0,1,640,1.0,git.git_commit git.git_reset git.git_diff_staged github.update_milestone filesystem.move_file git.git_show slack.update_reminder gitlab.update_milestone jira.jira_transition_issue salesforce.salesforce_list_opportunities +q014,Create a Linear issue,verb_product,linear.linear_get_issue,0.0,1.0,1.0,0.5000,0.6309,2,1,275,1.0,1,404,1.0,1,875,1.0,linear.linear_get_issue linear.linear_create_issue linear.linear_create_comment linear.linear_search_issues linear.linear_assign_issue linear.linear_update_issue linear.linear_list_issues linear.linear_create_project linear.create_label linear.create_attachment +q015,Make a task in my project tracker Linear,paraphrase,linear.linear_create_project,0.0,1.0,1.0,0.3333,0.5000,3,1,284,1.0,1,483,1.0,1,870,1.0,linear.linear_create_project linear.linear_list_projects linear.linear_create_issue linear.linear_update_issue linear.linear_list_issues linear.create_attachment linear.create_label linear.linear_get_issue linear.linear_create_comment linear.linear_assign_issue +q016,Post a comment on a GitHub issue,verb_product,github.add_issue_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,388,1.0,1,617,1.0,1,1118,1.0,github.add_issue_comment github.update_issue_comment github.add_comment_to_pending_review github.issue_write github.get_discussion_comments jira.jira_add_comment github.pull_request_review_write github.get_discussion github.assign_copilot_to_issue github.issue_read +q017,Merge the pull request,ambiguous,github.merge_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,283,1.0,1,528,1.0,1,1003,1.0,github.merge_pull_request gitlab.gitlab_merge_merge_request gitlab.gitlab_get_merge_request github.update_pull_request_branch github.create_pull_request gitlab.gitlab_list_merge_requests gitlab.gitlab_create_merge_request github.update_pull_request github.search_pull_requests github.pull_request_read +q018,Trigger a GitHub Actions workflow,verb_product,github.actions_run_trigger,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,457,1.0,1,780,1.0,github.actions_run_trigger github.actions_get github.actions_list github.actions_get_job_logs github.update_check_run github.dismiss_notification github.get_check_run github.create_check_run github.update_hook github.get_hook +q019,Get Dependabot alerts,resource,github.get_dependabot_alert,1.0,1.0,1.0,1.0000,0.8597,1,1,286,1.0,1,481,1.0,1,819,1.0,github.get_dependabot_alert github.list_dependabot_alerts github.get_code_scanning_alert github.get_secret_scanning_alert github.list_code_scanning_alerts github.list_secret_scanning_alerts pagerduty.pagerduty_list_incidents github.get_notification_details slack.get_reminder sentry.sentry_list_events +q020,List Sentry issues,verb_product,sentry.sentry_list_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,396,1.0,1,766,1.0,sentry.sentry_list_issues sentry.sentry_list_events sentry.sentry_list_projects sentry.sentry_get_issue sentry.sentry_update_issue linear.linear_list_issues bitbucket.bitbucket_list_prs pagerduty.pagerduty_list_incidents slack.list_pins hubspot.hubspot_list_contacts +q021,Create a Stripe customer,verb_product,stripe.stripe_create_invoice,0.0,1.0,1.0,0.5000,0.6309,2,1,223,1.0,1,322,1.0,1,669,1.0,stripe.stripe_create_invoice stripe.stripe_create_customer stripe.stripe_create_payment_intent stripe.stripe_list_invoices stripe.stripe_get_customer stripe.stripe_create_product stripe.stripe_create_price stripe.stripe_create_refund stripe.stripe_list_subscriptions stripe.stripe_list_customers +q022,Refund a Stripe payment,paraphrase,stripe.stripe_create_refund,1.0,1.0,1.0,1.0000,1.0000,1,1,197,1.0,1,295,1.0,1,647,1.0,stripe.stripe_create_refund stripe.stripe_cancel_subscription stripe.stripe_get_payment_intent stripe.stripe_get_customer stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_create_payment_intent stripe.stripe_create_price stripe.stripe_create_customer stripe.stripe_create_product +q023,Send an email with Gmail,verb_product,gmail.gmail_send_email,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,399,1.0,1,743,1.0,gmail.gmail_send_email gmail.gmail_get_email gmail.gmail_trash_email gmail.gmail_create_draft gmail.gmail_modify_email google_drive.gdrive_share_file gmail.gmail_search_emails gmail.gmail_list_labels hubspot.hubspot_create_contact shopify.shopify_create_order +q024,Search my Gmail inbox,verb_product,gmail.gmail_search_emails,1.0,1.0,1.0,1.0000,1.0000,1,1,178,1.0,1,291,1.0,1,678,1.0,gmail.gmail_search_emails gmail.gmail_get_email gmail.gmail_list_labels hubspot.hubspot_search_contacts gmail.gmail_trash_email gmail.gmail_modify_email gmail.gmail_create_draft slack.slack_search_messages gmail.gmail_send_email github.search_users +q025,Upload a file to Google Drive,paraphrase,google_drive.gdrive_update_file,0.0,1.0,1.0,0.5000,0.6309,2,1,307,1.0,1,418,1.0,1,806,1.0,google_drive.gdrive_update_file google_drive.gdrive_create_file google_drive.gdrive_share_file google_drive.gdrive_get_file google_drive.gdrive_search google_drive.gdrive_list_folder filesystem.write_file slack.slack_upload_file aws_s3.s3_put_object filesystem.edit_file +q026,Create a calendar event,resource,google_calendar.gcal_create_event,1.0,1.0,1.0,1.0000,1.0000,1,1,317,1.0,1,427,1.0,1,858,1.0,google_calendar.gcal_create_event google_calendar.gcal_update_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars google_calendar.gcal_delete_event sentry.sentry_list_events slack.create_reminder pagerduty.pagerduty_list_oncalls clickup.clickup_create_task pagerduty.pagerduty_create_incident +q027,List Kubernetes pods,verb_product,kubernetes.k8s_list_pods,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,334,1.0,1,661,1.0,kubernetes.k8s_list_pods kubernetes.k8s_list_services kubernetes.k8s_list_deployments kubernetes.k8s_get_events kubernetes.k8s_pod_logs kubernetes.k8s_get_pod docker.docker_images kubernetes.k8s_apply kubernetes.k8s_scale_deployment okta.okta_list_users +q028,Get logs from a k8s pod,verb_product,kubernetes.k8s_pod_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,252,1.0,1,371,1.0,1,738,1.0,kubernetes.k8s_pod_logs kubernetes.k8s_get_pod kubernetes.k8s_get_events docker.docker_logs kubernetes.k8s_list_pods kubernetes.k8s_list_services kubernetes.k8s_list_deployments github.actions_get_job_logs kubernetes.k8s_scale_deployment kubernetes.k8s_apply +q029,Run a docker container,verb_product,docker.docker_run,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,331,1.0,1,594,1.0,docker.docker_run docker.docker_exec docker.docker_ps docker.docker_stop docker.docker_logs docker.docker_build docker.docker_images docker.docker_rm terraform.tf_plan terraform.tf_apply +q030,Show docker container logs,verb_product,docker.docker_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,213,1.0,1,352,1.0,1,678,1.0,docker.docker_logs kubernetes.k8s_pod_logs docker.docker_images docker.docker_run docker.docker_ps git.git_log docker.docker_stop docker.docker_exec datadog.datadog_list_logs cloudflare.cf_list_dns +q031,Navigate the browser to a URL,ambiguous,playwright.browser_navigate,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,294,1.0,1,613,1.0,playwright.browser_navigate puppeteer.puppeteer_navigate playwright.browser_tab_new playwright.browser_click playwright.browser_press_key fetch.fetch playwright.browser_take_screenshot playwright.browser_type playwright.browser_tab_list confluence.confluence_search +q032,Take a screenshot of the page,ambiguous,playwright.browser_take_screenshot,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,346,1.0,1,633,1.0,playwright.browser_take_screenshot puppeteer.puppeteer_screenshot playwright.browser_snapshot pagerduty.pagerduty_get_incident pagerduty.pagerduty_acknowledge_incident playwright.browser_evaluate playwright.browser_click playwright.browser_press_key pagerduty.pagerduty_resolve_incident gitlab.get_snippet +q033,Create a GitLab merge request,verb_product,gitlab.gitlab_create_merge_request,1.0,1.0,1.0,1.0000,1.0000,1,1,297,1.0,1,475,1.0,1,943,1.0,gitlab.gitlab_create_merge_request gitlab.gitlab_merge_merge_request gitlab.gitlab_get_merge_request gitlab.gitlab_list_merge_requests github.merge_pull_request gitlab.create_release gitlab.create_pipeline gitlab.create_snippet gitlab.gitlab_create_pipeline gitlab.create_job +q034,Search Notion pages,verb_product,notion.notion_search,1.0,1.0,1.0,1.0000,1.0000,1,1,229,1.0,1,383,1.0,1,672,1.0,notion.notion_search notion.notion_get_page notion.notion_create_page notion.notion_query_database notion.notion_update_page notion.notion_list_users notion.notion_create_database notion.notion_get_comments notion.notion_get_block memory.search_nodes +q035,Create a Notion page,verb_product,notion.notion_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,247,1.0,1,385,1.0,1,781,1.0,notion.notion_create_page notion.notion_get_page notion.notion_create_database notion.notion_search notion.notion_update_page notion.create_data_source notion.create_users_token notion.create_comment notion.notion_list_users notion.notion_get_comments +q036,Query a Notion database,verb_product,notion.notion_query_database,1.0,1.0,1.0,1.0000,1.0000,1,1,261,1.0,1,370,1.0,1,675,1.0,notion.notion_query_database notion.notion_search notion.notion_create_database notion.get_data_source notion.notion_get_page notion.notion_list_users notion.notion_create_page notion.notion_get_block notion.get_users_token notion.update_data_source +q037,Create a Zendesk ticket,verb_product,zendesk.zendesk_get_ticket,0.0,1.0,1.0,0.5000,0.6309,2,1,221,1.0,1,368,1.0,1,855,1.0,zendesk.zendesk_get_ticket zendesk.zendesk_create_ticket zendesk.zendesk_list_tickets zendesk.zendesk_update_ticket zendesk.zendesk_search jira.jira_create_issue jira.jira_create_sprint google_calendar.gcal_create_event jira.get_board hubspot.hubspot_create_deal +q038,Create a HubSpot contact,verb_product,hubspot.hubspot_create_contact,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,364,1.0,1,786,1.0,hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_search_contacts hubspot.hubspot_list_contacts hubspot.hubspot_create_deal slack.create_usergroup slack.get_usergroup twilio.twilio_make_call pagerduty.pagerduty_create_incident slack.update_usergroup +q039,Create a Salesforce lead,verb_product,salesforce.salesforce_create_lead,1.0,1.0,1.0,1.0000,1.0000,1,1,252,1.0,1,401,1.0,1,915,1.0,salesforce.salesforce_create_lead salesforce.salesforce_convert_lead salesforce.salesforce_create_record salesforce.salesforce_list_opportunities slack.create_pin linear.create_user hubspot.hubspot_create_deal salesforce.salesforce_get_record slack.create_reminder github.create_pull_request +q040,Run a SOQL query,identifier,salesforce.salesforce_query,1.0,1.0,1.0,1.0000,1.0000,1,1,187,1.0,1,291,1.0,1,632,1.0,salesforce.salesforce_query mysql.mysql_query postgres.query sqlite.sqlite_query prometheus.prom_query postgres.explain_query prometheus.prom_query_range datadog.datadog_list_monitors supabase.supabase_select hubspot.hubspot_search_contacts +q041,List Shopify orders,verb_product,shopify.shopify_list_orders,1.0,1.0,1.0,1.0000,1.0000,1,1,210,1.0,1,333,1.0,1,667,1.0,shopify.shopify_list_orders shopify.shopify_get_order shopify.shopify_create_order shopify.shopify_list_products shopify.shopify_list_customers shopify.shopify_get_product shopify.shopify_update_product shopify.shopify_create_product stripe.stripe_list_subscriptions stripe.stripe_list_invoices +q042,Create a PagerDuty incident,verb_product,pagerduty.pagerduty_get_incident,0.0,1.0,1.0,0.5000,0.6309,2,1,259,1.0,1,403,1.0,1,823,1.0,pagerduty.pagerduty_get_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls sentry.sentry_list_events playwright.browser_press_key jira.jira_create_issue google_calendar.gcal_create_event +q043,Who is on call in PagerDuty,paraphrase,pagerduty.pagerduty_list_oncalls,1.0,1.0,1.0,1.0000,1.0000,1,1,253,1.0,1,401,1.0,1,719,1.0,pagerduty.pagerduty_list_oncalls pagerduty.pagerduty_get_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_resolve_incident twilio.twilio_make_call playwright.browser_press_key hubspot.hubspot_get_contact slack.get_bookmark +q044,Query Datadog metrics,verb_product,datadog.datadog_query_metrics,1.0,1.0,1.0,1.0000,1.0000,1,1,284,1.0,1,458,1.0,1,756,1.0,datadog.datadog_query_metrics datadog.datadog_list_monitors datadog.datadog_list_logs datadog.datadog_create_monitor datadog.datadog_get_monitor memory.search_nodes slack.slack_search_messages notion.notion_search github.search_pull_requests prometheus.prom_list_metrics +q045,Search Datadog logs,verb_product,datadog.datadog_list_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,298,1.0,1,458,1.0,1,773,1.0,datadog.datadog_list_logs datadog.datadog_list_monitors datadog.datadog_create_monitor datadog.datadog_get_monitor datadog.datadog_query_metrics github.search_pull_requests memory.search_nodes slack.slack_search_messages github.search_users github.search_issues +q046,List S3 buckets,verb_product,aws_s3.s3_list_buckets,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,327,1.0,1,624,1.0,aws_s3.s3_list_buckets aws_s3.s3_list_objects aws_s3.s3_get_object aws_s3.s3_delete_object bitbucket.bitbucket_list_repos bitbucket.bitbucket_list_prs aws_s3.s3_put_object jira.jira_list_projects salesforce.salesforce_list_opportunities terraform.tf_state_list +q047,Upload an object to S3,paraphrase,aws_s3.s3_put_object,1.0,1.0,1.0,1.0000,1.0000,1,1,252,1.0,1,405,1.0,1,734,1.0,aws_s3.s3_put_object aws_s3.s3_get_object aws_s3.s3_list_objects aws_s3.s3_delete_object slack.slack_upload_file twilio.twilio_send_sms aws_s3.s3_list_buckets jira.update_attachment linear.update_attachment gitlab.update_snippet +q048,Purge Cloudflare cache,verb_product,cloudflare.cf_purge_cache,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,296,1.0,1,618,1.0,cloudflare.cf_purge_cache slack.delete_bookmark cloudflare.cf_list_zones slack.delete_canvase cloudflare.cf_list_dns cloudflare.cf_create_dns memory.delete_observations redis.redis_delete aws_s3.s3_delete_object linear.delete_attachment +q049,Create a Cloudflare DNS record,verb_product,cloudflare.cf_create_dns,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,356,1.0,1,808,1.0,cloudflare.cf_create_dns cloudflare.cf_list_dns cloudflare.cf_purge_cache salesforce.salesforce_create_record cloudflare.cf_list_zones datadog.datadog_create_monitor slack.create_file salesforce.salesforce_get_record docker.docker_logs slack.create_bookmark +q050,List Vercel deployments,verb_product,vercel.vercel_list_deployments,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,291,1.0,1,601,1.0,vercel.vercel_list_deployments vercel.vercel_get_deployment vercel.vercel_list_projects kubernetes.k8s_list_deployments vercel.vercel_create_project github.list_packages gitlab.list_releases github.list_releases docker.docker_images kubernetes.k8s_list_services +q051,Insert a row into Supabase,verb_product,supabase.supabase_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,355,1.0,1,691,1.0,supabase.supabase_insert supabase.supabase_update supabase.supabase_select supabase.supabase_invoke_fn supabase.supabase_list_tables mongodb.mongo_insert linear.update_cycle linear.update_attachment linear.update_user linear.linear_create_comment +q052,Export images from Figma,paraphrase,figma.figma_get_images,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,368,1.0,1,670,1.0,figma.figma_get_images figma.figma_get_file figma.figma_get_file_nodes figma.figma_get_comments figma.figma_post_comment docker.docker_images linear.update_attachment linear.get_attachment filesystem.read_media_file linear.create_attachment +q053,Send an SMS with Twilio,verb_product,twilio.twilio_send_sms,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,428,1.0,1,820,1.0,twilio.twilio_send_sms twilio.twilio_list_messages twilio.twilio_make_call slack.slack_reply_to_thread slack.slack_add_reaction discord.discord_send_message slack.slack_post_message slack.slack_get_thread_replies gmail.gmail_send_email grafana.grafana_create_annotation +q054,Create an Okta user,verb_product,okta.okta_create_user,1.0,1.0,1.0,1.0000,1.0000,1,1,178,1.0,1,337,1.0,1,806,1.0,okta.okta_create_user okta.okta_get_user okta.okta_list_users okta.okta_deactivate_user notion.create_users_token linear.create_user asana.asana_create_task asana.asana_create_project slack.create_usergroup jira.create_attachment +q055,Instant query Prometheus,verb_product,prometheus.prom_query,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,379,1.0,1,687,1.0,prometheus.prom_query prometheus.prom_query_range prometheus.prom_list_metrics memory.search_nodes datadog.datadog_query_metrics github.search_pull_requests hubspot.hubspot_search_contacts elasticsearch.es_search datadog.datadog_list_monitors notion.notion_search +q056,Search Grafana dashboards,verb_product,grafana.grafana_search_dashboards,1.0,1.0,1.0,1.0000,1.0000,1,1,185,1.0,1,322,1.0,1,663,1.0,grafana.grafana_search_dashboards grafana.grafana_get_dashboard github.search_pull_requests grafana.grafana_create_annotation github.search_orgs github.search_users github.search_repositories github.search_issues zendesk.zendesk_search hubspot.hubspot_search_contacts +q057,Trigger a CircleCI pipeline,verb_product,circleci.circleci_trigger_pipeline,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,371,1.0,1,773,1.0,circleci.circleci_trigger_pipeline circleci.circleci_get_pipeline circleci.circleci_list_pipelines gitlab.list_pipelines hubspot.hubspot_create_deal gitlab.update_pipeline github.actions_run_trigger slack.slack_add_reaction gitlab.get_pipeline gitlab.gitlab_create_pipeline +q058,Run terraform plan,identifier,terraform.tf_plan,1.0,1.0,1.0,1.0000,1.0000,1,1,172,1.0,1,332,1.0,1,710,1.0,terraform.tf_plan terraform.tf_apply terraform.tf_state_list github.update_check_run hubspot.hubspot_create_deal github.get_check_run docker.docker_run github.update_runner github.create_check_run github.update_environment +q059,Create an Asana task,verb_product,asana.asana_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,264,1.0,1,405,1.0,1,856,1.0,asana.asana_create_task asana.asana_create_project asana.asana_list_tasks asana.asana_get_task asana.asana_update_task clickup.clickup_create_task slack.create_reminder okta.okta_create_user gitlab.create_snippet notion.create_data_source +q060,Create a Trello card,verb_product,trello.trello_create_card,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,364,1.0,1,752,1.0,trello.trello_create_card trello.trello_create_board trello.trello_add_comment trello.trello_list_cards stripe.stripe_create_invoice grafana.grafana_create_annotation stripe.stripe_create_customer stripe.stripe_create_product stripe.stripe_create_payment_intent stripe.stripe_create_price +q061,Create a ClickUp task,verb_product,clickup.clickup_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,364,1.0,1,782,1.0,clickup.clickup_create_task clickup.clickup_list_tasks clickup.clickup_get_task clickup.clickup_update_task puppeteer.puppeteer_click playwright.browser_click puppeteer.puppeteer_fill pagerduty.pagerduty_create_incident puppeteer.puppeteer_screenshot asana.asana_create_task +q062,Create a Confluence page,verb_product,confluence.confluence_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,401,1.0,1,866,1.0,confluence.confluence_create_page confluence.confluence_get_page confluence.confluence_update_page confluence.confluence_search notion.notion_create_page github.create_artifact notion.create_users_token shopify.shopify_create_product pagerduty.pagerduty_create_incident github.search_code +q063,Send a Discord message,verb_product,discord.discord_send_message,1.0,1.0,1.0,1.0000,1.0000,1,1,245,1.0,1,384,1.0,1,742,1.0,discord.discord_send_message discord.discord_add_reaction slack.slack_post_message discord.discord_get_channel slack.slack_add_reaction slack.slack_reply_to_thread gmail.gmail_trash_email discord.discord_list_channels twilio.twilio_send_sms slack.slack_get_thread_replies +q064,Search Elasticsearch,verb_product,elasticsearch.es_search,1.0,1.0,1.0,1.0000,1.0000,1,1,195,1.0,1,300,1.0,1,635,1.0,elasticsearch.es_search confluence.confluence_search hubspot.hubspot_search_contacts github.search_pull_requests github.search_orgs elasticsearch.es_get_doc github.search_users grafana.grafana_search_dashboards memory.search_nodes datadog.datadog_list_logs +q065,Get a Redis key,verb_product,redis.redis_get,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,280,1.0,1,550,1.0,redis.redis_get redis.redis_set redis.redis_delete redis.redis_keys playwright.browser_press_key slack.get_pin circleci.circleci_get_pipeline figma.figma_get_file clickup.clickup_get_task github.get_milestone +q066,Find MongoDB documents,verb_product,mongodb.mongo_find,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,381,1.0,1,729,1.0,mongodb.mongo_find mongodb.mongo_list_collections mongodb.mongo_delete mongodb.mongo_update mongodb.mongo_insert elasticsearch.es_get_doc filesystem.search_files confluence.confluence_search datadog.datadog_list_logs hubspot.hubspot_search_contacts +q067,List SQLite tables,verb_product,sqlite.sqlite_list_tables,1.0,1.0,1.0,1.0000,1.0000,1,1,142,1.0,1,240,1.0,1,526,1.0,sqlite.sqlite_list_tables mysql.mysql_list_tables supabase.supabase_list_tables postgres.list_tables sqlite.sqlite_describe_table okta.okta_list_users postgres.list_schemas datadog.datadog_list_monitors filesystem.list_directory gitlab.list_wikis +q068,Describe a postgres table,verb_product,postgres.describe_table,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,274,1.0,1,593,1.0,postgres.describe_table postgres.explain_query sqlite.sqlite_describe_table postgres.list_schemas postgres.list_tables mysql.mysql_list_tables postgres.query supabase.supabase_insert supabase.supabase_select notion.notion_query_database +q069,github_create_issue,identifier,github.create_repository,0.0,1.0,1.0,0.5000,0.6697,2,1,367,1.0,1,536,1.0,1,1012,1.0,github.create_repository github.create_issue github.issue_write github.create_gist gitlab.gitlab_create_issue github.create_environment github.create_package github.create_artifact bitbucket.bitbucket_create_issue github.create_runner +q070,createIssue,identifier,linear.create_user,0.0,0.0,0.0,0.0000,0.0000,39,13,3554,0.0,8,3600,0.0,4,3528,0.0,linear.create_user gmail.gmail_create_draft notion.create_data_source linear.create_cycle notion.create_comment linear.create_comment linear.create_attachment notion.create_users_token stripe.stripe_create_product okta.okta_create_user +q071,list-directory,identifier,filesystem.list_directory,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,271,1.0,1,489,1.0,filesystem.list_directory filesystem.list_directory_with_sizes okta.okta_list_users mongodb.mongo_list_collections filesystem.list_allowed_directories clickup.clickup_list_tasks terraform.tf_state_list sentry.sentry_list_projects docker.docker_images gmail.gmail_list_labels +q072,sequential thinking next thought,paraphrase,sequential_thinking.sequentialthinking,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,344,1.0,1,650,1.0,sequential_thinking.sequentialthinking memory.create_relations memory.add_observations memory.search_nodes memory.create_entities memory.delete_relations notion.update_comment notion.notion_search memory.open_nodes notion.notion_append_block +q073,What time is it in Tokyo,paraphrase,time.get_current_time,1.0,1.0,1.0,1.0000,0.9239,1,1,215,1.0,1,378,1.0,1,785,1.0,time.get_current_time zendesk.zendesk_list_tickets google_calendar.gcal_list_events time.convert_time grafana.grafana_create_annotation jira.jira_create_sprint playwright.browser_wait_for pagerduty.pagerduty_list_oncalls slack.slack_add_reaction pagerduty.pagerduty_list_incidents +q074,Convert time between timezones,verb_product,time.convert_time,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,402,1.0,1,699,1.0,time.convert_time time.get_current_time grafana.grafana_create_annotation playwright.browser_wait_for datadog.datadog_query_metrics cloudflare.cf_list_zones linear.update_cycle google_calendar.gcal_list_events linear.get_cycle cloudflare.cf_list_dns +q075,Star a GitHub repo,verb_product,github.star_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,214,1.0,1,437,1.0,1,950,1.0,github.star_repository github.unstar_repository github.list_starred_repositories github.create_or_update_file github.create_gist github.get_file_blame github.create_repository github.list_code_scanning_alerts github.list_pull_requests github.create_pull_request +q076,Fork a repository,resource,github.fork_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,277,1.0,1,468,1.0,1,950,1.0,github.fork_repository github.unstar_repository github.merge_pull_request github.star_repository bitbucket.bitbucket_create_pr github.delete_repository github.get_commit github.create_repository github.create_or_update_file github.get_repository_tree +q077,Get the latest GitHub release,verb_product,github.get_latest_release,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,366,1.0,1,673,1.0,github.get_latest_release gitlab.get_release github.get_release_by_tag github.get_milestone github.list_releases github.get_notification_details github.get_gist github.get_check_run github.get_code_scanning_alert github.get_package +q078,Create a gist,resource,github.create_gist,1.0,1.0,1.0,1.0000,1.0000,1,1,229,1.0,1,383,1.0,1,887,1.0,github.create_gist github.update_gist github.get_gist gitlab.create_snippet github.list_gists github.create_milestone notion.create_data_source github.create_hook gitlab.create_label gitlab.create_wiki +q079,Search GitHub code,verb_product,github.search_code,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,334,1.0,1,722,1.0,github.search_code github.search_orgs github.search_users github.search_pull_requests github.search_repositories github.search_commits github.get_code_scanning_alert github.get_check_run github.search_issues github.list_code_scanning_alerts +q080,List GitHub notifications,verb_product,github.list_notifications,1.0,1.0,1.0,1.0000,1.0000,1,1,176,1.0,1,354,1.0,1,713,1.0,github.list_notifications github.get_notification_details github.mark_all_notifications_read github.list_code_scanning_alerts github.list_secret_scanning_alerts github.list_dependabot_alerts github.dismiss_notification github.list_milestones github.list_gists github.get_code_scanning_alert +q081,Assign Copilot to an issue,verb_product,github.assign_copilot_to_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,301,1.0,1,434,1.0,1,826,1.0,github.assign_copilot_to_issue linear.linear_assign_issue github.request_copilot_review jira.jira_assign_issue pagerduty.pagerduty_resolve_incident jira.jira_get_issue linear.linear_create_comment linear.linear_list_issues github.create_issue linear.linear_get_issue +q082,Get secret scanning alerts,resource,github.get_secret_scanning_alert,1.0,1.0,1.0,1.0000,0.8597,1,1,294,1.0,1,483,1.0,1,781,1.0,github.get_secret_scanning_alert github.list_secret_scanning_alerts github.get_code_scanning_alert github.list_code_scanning_alerts github.get_dependabot_alert github.list_dependabot_alerts github.get_notification_details github.mark_all_notifications_read pagerduty.pagerduty_get_incident github.list_notifications +q083,Create or update a file on GitHub,verb_product,github.create_or_update_file,1.0,1.0,1.0,1.0000,0.7602,1,1,327,1.0,1,476,1.0,1,878,1.0,github.create_or_update_file github.update_gist github.create_gist github.projects_write github.update_environment github.update_package github.issue_write github.update_hook github.update_check_run slack.update_file +q084,List collaborators on a repo,resource,github.list_repository_collaborators,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,365,1.0,1,696,1.0,github.list_repository_collaborators github.list_labels github.list_starred_repositories github.list_tags github.list_releases github.list_discussions github.actions_list github.get_team_members bitbucket.bitbucket_list_repos github.list_branches +q085,Reply in a Slack thread,verb_product,slack.slack_reply_to_thread,1.0,1.0,1.0,1.0000,1.0000,1,1,277,1.0,1,402,1.0,1,777,1.0,slack.slack_reply_to_thread slack.slack_get_thread_replies slack.slack_add_reaction slack.slack_post_message slack.get_reminder slack.update_reminder slack.slack_search_messages slack.update_pin slack.create_reminder slack.update_file +q086,Add an emoji reaction on Slack,verb_product,slack.slack_add_reaction,1.0,1.0,1.0,1.0000,1.0000,1,1,289,1.0,1,440,1.0,1,795,1.0,slack.slack_add_reaction discord.discord_add_reaction slack.slack_reply_to_thread slack.slack_get_thread_replies slack.update_canvase slack.update_reminder slack.get_reminder slack.slack_post_message slack.get_canvase slack.create_reminder +q087,Search Slack messages,verb_product,slack.slack_search_messages,1.0,1.0,1.0,1.0000,1.0000,1,1,251,1.0,1,381,1.0,1,720,1.0,slack.slack_search_messages slack.slack_get_channel_history slack.slack_add_reaction github.search_pull_requests slack.slack_get_thread_replies slack.get_reminder slack.slack_reply_to_thread slack.get_canvase slack.slack_list_channels slack.slack_post_message +q088,Create a Slack channel,verb_product,slack.slack_create_channel,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,356,1.0,1,781,1.0,slack.slack_create_channel slack.slack_join_channel slack.slack_leave_channel slack.slack_post_message slack.slack_add_reaction slack.slack_reply_to_thread slack.slack_get_thread_replies slack.slack_set_channel_topic slack.create_usergroup slack.slack_upload_file +q089,Transition a Jira issue,verb_product,jira.jira_transition_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,405,1.0,1,745,1.0,jira.jira_transition_issue jira.jira_update_issue jira.jira_add_comment jira.jira_assign_issue jira.jira_create_issue jira.jira_get_issue jira.update_sprint jira.jira_search_issues jira.get_sprint jira.create_version +q090,Search Jira with JQL,identifier,jira.jira_search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,356,1.0,1,618,1.0,jira.jira_search_issues jira.get_filter jira.create_filter jira.jira_get_issue jira.update_filter jira.get_webhook jira.get_version jira.jira_list_projects jira.delete_filter jira.list_filters +q091,Add a worklog to Jira,verb_product,jira.jira_add_worklog,1.0,1.0,1.0,1.0000,1.0000,1,1,261,1.0,1,429,1.0,1,909,1.0,jira.jira_add_worklog jira.jira_list_projects jira.jira_create_issue jira.jira_add_comment jira.create_webhook jira.create_sprint jira.create_attachment jira.create_version jira.create_board jira.create_filter +q092,List Linear teams,verb_product,linear.linear_list_teams,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,398,1.0,1,718,1.0,linear.linear_list_teams linear.linear_list_projects linear.linear_list_issues linear.linear_create_issue linear.linear_create_project github.get_team_members linear.list_cycles linear.linear_get_issue linear.get_cycle linear.list_users +q093,Comment on a Linear issue,verb_product,linear.linear_create_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,324,1.0,1,771,1.0,linear.linear_create_comment linear.linear_get_issue linear.linear_assign_issue linear.linear_search_issues linear.get_comment linear.linear_update_issue linear.update_comment linear.linear_create_issue linear.delete_comment linear.linear_list_issues +q094,Move a local file,paraphrase,filesystem.move_file,1.0,1.0,1.0,1.0000,1.0000,1,1,225,1.0,1,352,1.0,1,726,1.0,filesystem.move_file filesystem.read_text_file filesystem.read_media_file filesystem.write_file filesystem.read_file filesystem.edit_file slack.update_file slack.get_file git.git_add slack.create_file +q095,Recursively search files by pattern,paraphrase,filesystem.search_files,1.0,1.0,1.0,1.0000,1.0000,1,1,202,1.0,1,304,1.0,1,609,1.0,filesystem.search_files filesystem.directory_tree memory.search_nodes google_drive.gdrive_search redis.redis_keys filesystem.get_file_info filesystem.list_directory_with_sizes filesystem.list_directory notion.notion_search github.search_code +q096,Edit a file in place with old and new string,paraphrase,filesystem.edit_file,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,381,1.0,1,785,1.0,filesystem.edit_file filesystem.move_file filesystem.write_file slack.update_file linear.update_attachment slack.create_file linear.delete_attachment linear.create_attachment slack.update_bookmark google_drive.gdrive_update_file +q097,Read the whole knowledge graph,paraphrase,memory.read_graph,1.0,1.0,1.0,1.0000,1.0000,1,1,162,1.0,1,270,1.0,1,582,1.0,memory.read_graph memory.open_nodes memory.search_nodes memory.add_observations memory.delete_entities memory.delete_relations memory.create_relations memory.create_entities datadog.datadog_list_monitors datadog.datadog_query_metrics +q098,Search memory nodes,verb_product,memory.search_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,194,1.0,1,285,1.0,1,632,1.0,memory.search_nodes memory.open_nodes filesystem.search_files hubspot.hubspot_search_contacts memory.read_graph github.search_orgs github.search_pull_requests datadog.datadog_list_logs memory.delete_entities github.search_repositories +q099,Show unstaged git diff,verb_product,git.git_diff_unstaged,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,299,1.0,1,620,1.0,git.git_diff_unstaged git.git_diff git.git_diff_staged git.git_show git.git_reset git.git_status git.git_log git.git_commit github.create_gist git.git_add +q100,Stage files with git add,identifier,git.git_add,1.0,1.0,1.0,1.0000,1.0000,1,1,201,1.0,1,329,1.0,1,723,1.0,git.git_add git.git_diff_staged git.git_commit github.update_gist git.git_reset github.create_or_update_file github.create_gist git.git_diff git.git_show git.git_branch +q101,Create a Shopify product,verb_product,shopify.shopify_create_product,1.0,1.0,1.0,1.0000,1.0000,1,1,235,1.0,1,365,1.0,1,670,1.0,shopify.shopify_create_product shopify.shopify_get_product shopify.shopify_create_order shopify.shopify_update_product shopify.shopify_get_order shopify.shopify_list_products stripe.stripe_create_product shopify.shopify_list_customers shopify.shopify_list_orders stripe.stripe_create_invoice +q102,Cancel a Stripe subscription,verb_product,stripe.stripe_cancel_subscription,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,301,1.0,1,629,1.0,stripe.stripe_cancel_subscription stripe.stripe_list_subscriptions stripe.stripe_create_refund stripe.stripe_list_invoices stripe.stripe_create_invoice stripe.stripe_get_payment_intent stripe.stripe_list_customers stripe.stripe_create_payment_intent stripe.stripe_get_customer stripe.stripe_create_customer +q103,Create a payment intent,resource,stripe.stripe_create_payment_intent,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,362,1.0,1,740,1.0,stripe.stripe_create_payment_intent stripe.stripe_create_refund stripe.stripe_get_payment_intent stripe.stripe_create_invoice stripe.stripe_create_price hubspot.hubspot_create_deal stripe.stripe_create_product stripe.stripe_create_customer slack.create_reminder stripe.stripe_list_invoices +q104,Acknowledge a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,405,1.0,1,702,1.0,pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls sentry.sentry_list_events slack.get_bookmark slack.get_reminder notion.notion_get_page +q105,Create a Datadog monitor,verb_product,datadog.datadog_create_monitor,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,459,1.0,1,799,1.0,datadog.datadog_create_monitor datadog.datadog_get_monitor datadog.datadog_list_monitors datadog.datadog_query_metrics datadog.datadog_list_logs memory.add_observations notion.list_data_sources github.list_runners memory.create_entities slack.list_bookmarks +q106,Resolve a Sentry issue,paraphrase,sentry.sentry_list_issues,0.0,1.0,1.0,0.3333,0.5000,3,1,274,1.0,1,397,1.0,1,800,1.0,sentry.sentry_list_issues sentry.sentry_get_issue sentry.sentry_update_issue sentry.sentry_list_events sentry.sentry_list_projects pagerduty.pagerduty_resolve_incident jira.jira_assign_issue jira.jira_get_issue github.assign_copilot_to_issue linear.linear_update_issue +q107,Scale a Kubernetes deployment,verb_product,kubernetes.k8s_scale_deployment,1.0,1.0,1.0,1.0000,1.0000,1,1,241,1.0,1,391,1.0,1,726,1.0,kubernetes.k8s_scale_deployment kubernetes.k8s_list_deployments kubernetes.k8s_apply kubernetes.k8s_list_services kubernetes.k8s_pod_logs kubernetes.k8s_get_pod kubernetes.k8s_list_pods kubernetes.k8s_delete kubernetes.k8s_get_events vercel.vercel_get_deployment +q108,Apply a Kubernetes manifest,verb_product,kubernetes.k8s_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,195,1.0,1,330,1.0,1,727,1.0,kubernetes.k8s_apply kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_get_pod kubernetes.k8s_list_pods kubernetes.k8s_scale_deployment kubernetes.k8s_get_events kubernetes.k8s_pod_logs terraform.tf_apply kubernetes.k8s_delete +q109,Stop a docker container,verb_product,docker.docker_stop,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,345,1.0,1,636,1.0,docker.docker_stop docker.docker_rm docker.docker_run docker.docker_logs docker.docker_exec docker.docker_ps docker.docker_images github.list_environments docker.docker_build cloudflare.cf_purge_cache +q110,Build a docker image,verb_product,docker.docker_build,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,319,1.0,1,757,1.0,docker.docker_build docker.docker_images docker.docker_run docker.docker_logs docker.docker_ps docker.docker_stop github.create_environment docker.docker_exec github.create_or_update_file figma.figma_get_images +q111,Click an element in Playwright,verb_product,playwright.browser_click,1.0,1.0,1.0,1.0000,1.0000,1,1,213,1.0,1,318,1.0,1,594,1.0,playwright.browser_click playwright.browser_press_key playwright.browser_type playwright.browser_navigate puppeteer.puppeteer_click playwright.browser_tab_list playwright.browser_tab_new playwright.browser_wait_for playwright.browser_tab_close playwright.browser_take_screenshot +q112,Type text into a browser input,paraphrase,playwright.browser_type,1.0,1.0,1.0,1.0000,1.0000,1,1,231,1.0,1,331,1.0,1,624,1.0,playwright.browser_type puppeteer.puppeteer_fill playwright.browser_click playwright.browser_press_key playwright.browser_fill_form playwright.browser_navigate playwright.browser_tab_new playwright.browser_evaluate playwright.browser_wait_for playwright.browser_take_screenshot +q113,List GitLab projects,verb_product,gitlab.gitlab_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,372,1.0,1,735,1.0,gitlab.gitlab_list_projects gitlab.gitlab_list_issues gitlab.gitlab_list_merge_requests gitlab.list_pipelines gitlab.list_jobs gitlab.list_snippets gitlab.list_wikis gitlab.list_milestones gitlab.list_releases vercel.vercel_list_projects +q114,Create a Bitbucket pull request,verb_product,bitbucket.bitbucket_create_pr,1.0,1.0,1.0,1.0000,1.0000,1,1,372,1.0,1,606,1.0,1,1091,1.0,bitbucket.bitbucket_create_pr bitbucket.bitbucket_list_prs github.create_pull_request bitbucket.bitbucket_create_issue github.pull_request_review_write github.search_pull_requests github.pull_request_read github.update_pull_request_branch github.merge_pull_request github.update_pull_request +q115,Create a Gmail draft,verb_product,gmail.gmail_create_draft,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,375,1.0,1,739,1.0,gmail.gmail_create_draft gmail.gmail_modify_email gmail.gmail_get_email gmail.gmail_list_labels gmail.gmail_send_email gmail.gmail_trash_email hubspot.hubspot_create_contact stripe.stripe_create_customer shopify.shopify_create_order google_drive.gdrive_share_file +q116,Share a Google Drive file,verb_product,google_drive.gdrive_share_file,1.0,1.0,1.0,1.0000,1.0000,1,1,271,1.0,1,417,1.0,1,717,1.0,google_drive.gdrive_share_file google_drive.gdrive_create_file google_drive.gdrive_get_file google_drive.gdrive_update_file google_drive.gdrive_search google_drive.gdrive_list_folder filesystem.move_file filesystem.read_media_file filesystem.get_file_info filesystem.write_file +q117,List Google Calendar events,verb_product,google_calendar.gcal_list_calendars,0.0,1.0,1.0,0.5000,0.6309,2,1,267,1.0,1,427,1.0,1,724,1.0,google_calendar.gcal_list_calendars google_calendar.gcal_list_events google_calendar.gcal_create_event google_calendar.gcal_update_event google_calendar.gcal_delete_event pagerduty.pagerduty_list_oncalls gmail.gmail_list_labels kubernetes.k8s_get_events sentry.sentry_list_events pagerduty.pagerduty_list_incidents +q118,Delete a calendar event,resource,google_calendar.gcal_delete_event,1.0,1.0,1.0,1.0000,1.0000,1,1,205,1.0,1,318,1.0,1,651,1.0,google_calendar.gcal_delete_event slack.delete_reminder slack.delete_canvase github.delete_milestone gitlab.delete_milestone linear.delete_cycle slack.delete_bookmark google_calendar.gcal_create_event linear.delete_label slack.delete_usergroup +q119,List Vercel projects,verb_product,vercel.vercel_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,175,1.0,1,260,1.0,1,524,1.0,vercel.vercel_list_projects vercel.vercel_list_deployments vercel.vercel_create_project jira.jira_list_projects sentry.sentry_list_projects gitlab.gitlab_list_projects vercel.vercel_get_deployment linear.linear_list_projects asana.asana_list_tasks supabase.supabase_list_tables +q120,Select rows from Supabase,verb_product,supabase.supabase_select,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,354,1.0,1,623,1.0,supabase.supabase_select supabase.supabase_insert supabase.supabase_update supabase.supabase_list_tables supabase.supabase_invoke_fn sqlite.sqlite_query salesforce.salesforce_query mysql.mysql_query postgres.query okta.okta_list_users +q121,Post a Figma comment,verb_product,figma.figma_post_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,350,1.0,1,699,1.0,figma.figma_post_comment figma.figma_get_comments figma.figma_get_file linear.get_comment linear.create_comment notion.get_comment linear.linear_create_comment linear.update_comment notion.create_comment notion.update_comment +q122,List Okta users,verb_product,okta.okta_list_users,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,271,1.0,1,613,1.0,okta.okta_list_users okta.okta_get_user okta.okta_create_user notion.notion_list_users okta.okta_deactivate_user notion.list_users_tokens github.list_repository_collaborators slack.list_usergroups github.list_starred_repositories discord.discord_list_members +q123,Range query Prometheus,verb_product,prometheus.prom_query_range,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,385,1.0,1,736,1.0,prometheus.prom_query_range prometheus.prom_query prometheus.prom_list_metrics datadog.datadog_query_metrics datadog.datadog_list_monitors gmail.gmail_search_emails memory.search_nodes github.search_pull_requests datadog.datadog_list_logs shopify.shopify_list_products +q124,Create a Grafana annotation,verb_product,grafana.grafana_create_annotation,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,429,1.0,1,911,1.0,grafana.grafana_create_annotation grafana.grafana_get_dashboard grafana.grafana_search_dashboards slack.create_canvase gitlab.create_snippet github.create_artifact linear.create_comment linear.create_attachment linear.create_label notion.create_comment +q125,Apply terraform,identifier,terraform.tf_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,171,1.0,1,303,1.0,1,653,1.0,terraform.tf_apply terraform.tf_plan terraform.tf_state_list kubernetes.k8s_apply github.update_environment cloudflare.cf_create_dns cloudflare.cf_purge_cache cloudflare.cf_list_zones playwright.browser_fill_form hubspot.hubspot_create_deal +q126,List Asana tasks,verb_product,asana.asana_list_tasks,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,404,1.0,1,705,1.0,asana.asana_list_tasks asana.asana_create_task asana.asana_get_task asana.asana_update_task asana.asana_create_project clickup.clickup_list_tasks okta.okta_list_users pagerduty.pagerduty_list_incidents github.actions_list salesforce.salesforce_list_opportunities +q127,Comment on a Trello card,verb_product,trello.trello_add_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,368,1.0,1,727,1.0,trello.trello_add_comment trello.trello_create_card trello.trello_list_cards linear.get_comment trello.trello_create_board linear.update_comment notion.get_comment figma.figma_post_comment linear.create_comment linear.list_comments +q128,Update a ClickUp task,verb_product,clickup.clickup_update_task,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,364,1.0,1,731,1.0,clickup.clickup_update_task clickup.clickup_list_tasks clickup.clickup_get_task clickup.clickup_create_task puppeteer.puppeteer_click gitlab.update_snippet slack.update_bookmark asana.asana_update_task shopify.shopify_update_product playwright.browser_click +q129,Search Confluence,verb_product,confluence.confluence_search,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,371,1.0,1,652,1.0,confluence.confluence_search confluence.confluence_get_page confluence.confluence_update_page confluence.confluence_create_page grafana.grafana_search_dashboards github.search_orgs github.search_pull_requests hubspot.hubspot_search_contacts github.search_users notion.notion_search +q130,Create a HubSpot deal,verb_product,hubspot.hubspot_create_deal,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,715,1.0,hubspot.hubspot_create_deal hubspot.hubspot_get_contact hubspot.hubspot_create_contact hubspot.hubspot_search_contacts hubspot.hubspot_list_contacts slack.create_usergroup slack.get_usergroup stripe.stripe_create_refund slack.update_usergroup slack.slack_join_channel +q131,List Zendesk tickets,verb_product,zendesk.zendesk_list_tickets,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,367,1.0,1,674,1.0,zendesk.zendesk_list_tickets zendesk.zendesk_get_ticket zendesk.zendesk_search zendesk.zendesk_create_ticket zendesk.zendesk_update_ticket google_calendar.gcal_list_calendars jira.list_boards salesforce.salesforce_list_opportunities pagerduty.pagerduty_list_oncalls jira.list_webhooks +q132,List Discord channels,verb_product,discord.discord_list_channels,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,300,1.0,1,642,1.0,discord.discord_list_channels discord.discord_get_channel slack.slack_list_channels discord.discord_list_members discord.discord_send_message slack.list_usergroups slack.slack_get_channel_history github.list_notifications twilio.twilio_list_messages slack.slack_create_channel +q133,Set a Redis key,verb_product,redis.redis_set,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,280,1.0,1,641,1.0,redis.redis_set redis.redis_get redis.redis_delete redis.redis_keys playwright.browser_press_key slack.update_pin clickup.clickup_update_task slack.update_reminder linear.update_user gitlab.update_pipeline +q134,Insert MongoDB documents,verb_product,mongodb.mongo_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,374,1.0,1,759,1.0,mongodb.mongo_insert mongodb.mongo_update supabase.supabase_insert mongodb.mongo_list_collections mongodb.mongo_find mongodb.mongo_delete aws_s3.s3_put_object salesforce.salesforce_create_record elasticsearch.es_index_doc elasticsearch.es_get_doc +q135,Run a MySQL query,verb_product,mysql.mysql_query,1.0,1.0,1.0,1.0000,1.0000,1,1,179,1.0,1,294,1.0,1,595,1.0,mysql.mysql_query sqlite.sqlite_query postgres.query salesforce.salesforce_query postgres.explain_query supabase.supabase_select mysql.mysql_list_tables fetch.fetch prometheus.prom_query zendesk.zendesk_search +q136,Index a document in Elasticsearch,verb_product,elasticsearch.es_index_doc,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,372,1.0,1,681,1.0,elasticsearch.es_index_doc elasticsearch.es_get_doc elasticsearch.es_search mongodb.mongo_find confluence.confluence_search google_drive.gdrive_search memory.search_nodes filesystem.search_files mongodb.mongo_insert linear.linear_search_issues +q137,Get an S3 object,verb_product,aws_s3.s3_get_object,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,362,1.0,1,641,1.0,aws_s3.s3_get_object aws_s3.s3_put_object aws_s3.s3_delete_object aws_s3.s3_list_objects aws_s3.s3_list_buckets gitlab.get_snippet linear.get_attachment salesforce.salesforce_get_record redis.redis_get jira.get_attachment +q138,List Cloudflare zones,verb_product,cloudflare.cf_list_zones,1.0,1.0,1.0,1.0000,1.0000,1,1,215,1.0,1,326,1.0,1,570,1.0,cloudflare.cf_list_zones cloudflare.cf_list_dns cloudflare.cf_create_dns cloudflare.cf_purge_cache aws_s3.s3_list_buckets kubernetes.k8s_list_services kubernetes.k8s_list_deployments circleci.circleci_list_pipelines terraform.tf_state_list docker.docker_images +q139,Create a Stripe invoice,verb_product,stripe.stripe_create_invoice,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,341,1.0,1,662,1.0,stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_create_payment_intent stripe.stripe_create_customer stripe.stripe_create_refund stripe.stripe_create_product stripe.stripe_create_price stripe.stripe_get_payment_intent stripe.stripe_get_customer stripe.stripe_list_subscriptions +q140,List GitHub Actions workflows,verb_product,github.actions_list,1.0,1.0,1.0,1.0000,1.0000,1,1,293,1.0,1,452,1.0,1,859,1.0,github.actions_list github.actions_get github.actions_run_trigger github.actions_get_job_logs github.list_notifications github.list_milestones github.list_hooks github.list_code_scanning_alerts github.list_artifacts github.list_check_runs +q141,Get GitHub Actions job logs,verb_product,github.actions_get_job_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,442,1.0,1,805,1.0,github.actions_get_job_logs github.actions_get github.actions_list github.get_check_run github.actions_run_trigger github.get_notification_details git.git_log github.search_pull_requests github.get_code_scanning_alert github.list_code_scanning_alerts +q142,Create a GitHub repository,verb_product,github.create_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,274,1.0,1,519,1.0,1,1091,1.0,github.create_repository github.create_gist github.create_environment github.create_branch github.create_or_update_file github.create_package github.create_pull_request github.create_artifact github.create_issue github.create_hook +q143,Checkout a git branch,verb_product,git.git_checkout,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,295,1.0,1,693,1.0,git.git_checkout github.get_check_run git.git_branch git.git_status git.git_show github.list_branches github.pull_request_read github.get_gist git.git_create_branch github.create_branch +q144,Show git log,identifier,git.git_log,1.0,1.0,1.0,1.0000,1.0000,1,1,194,1.0,1,352,1.0,1,668,1.0,git.git_log git.git_show git.git_status github.actions_get_job_logs git.git_diff git.git_diff_unstaged git.git_commit github.get_check_run git.git_branch github.list_code_scanning_alerts +q145,List postgres schemas,verb_product,postgres.list_schemas,1.0,1.0,1.0,1.0000,1.0000,1,1,152,1.0,1,254,1.0,1,493,1.0,postgres.list_schemas postgres.list_tables mysql.mysql_list_tables postgres.describe_table vercel.vercel_list_projects supabase.supabase_list_tables vercel.vercel_list_deployments mongodb.mongo_list_collections sentry.sentry_list_projects aws_s3.s3_list_buckets +q146,Open nodes in the knowledge graph,verb_product,memory.open_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,163,1.0,1,262,1.0,1,599,1.0,memory.open_nodes memory.search_nodes memory.read_graph memory.create_relations memory.create_entities memory.add_observations memory.delete_entities memory.delete_relations notion.notion_search sequential_thinking.sequentialthinking +q147,Get file metadata from the filesystem,paraphrase,filesystem.get_file_info,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,302,1.0,1,685,1.0,filesystem.get_file_info google_drive.gdrive_get_file filesystem.read_file filesystem.read_media_file filesystem.list_directory filesystem.read_text_file filesystem.directory_tree filesystem.list_directory_with_sizes figma.figma_get_file_nodes github.get_file_contents +q148,Write a new local file,paraphrase,filesystem.write_file,1.0,1.0,1.0,1.0000,1.0000,1,1,280,1.0,1,421,1.0,1,850,1.0,filesystem.write_file filesystem.edit_file slack.create_file filesystem.read_text_file filesystem.create_directory google_drive.gdrive_create_file filesystem.read_file filesystem.read_media_file filesystem.move_file github.create_or_update_file +q149,Search GitHub issues,verb_product,github.search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,314,1.0,1,724,1.0,github.search_issues github.search_users github.search_orgs github.search_pull_requests github.search_commits github.search_repositories github.search_code github.list_issues gitlab.gitlab_list_projects github.list_code_scanning_alerts +q150,Get the authenticated GitHub user,paraphrase,github.get_me,1.0,1.0,1.0,1.0000,1.0000,1,1,179,1.0,1,343,1.0,1,670,1.0,github.get_me github.get_teams github.list_notifications github.get_secret_scanning_alert github.projects_get github.list_gists github.get_code_scanning_alert github.get_latest_release github.get_notification_details slack.slack_get_user_profile diff --git a/experiments/mcp-tool-search/results/embed_bge_small.json b/experiments/mcp-tool-search/results/embed_bge_small.json new file mode 100644 index 00000000..c47a6d4d --- /dev/null +++ b/experiments/mcp-tool-search/results/embed_bge_small.json @@ -0,0 +1,7013 @@ +{ + "summary": { + "n": 150, + "index_s": 1.9597746540000571, + "rss_mb": 0.4609375, + "p50_ms": 2.1447405000003528, + "p95_ms": 2.6409478999823923, + "recall@1": 0.9266666666666666, + "recall@1_lo": 0.88, + "recall@1_hi": 0.9666666666666667, + "recall@3": 0.9933333333333333, + "recall@3_lo": 0.98, + "recall@3_hi": 1.0, + "recall@5": 0.9933333333333333, + "recall@5_lo": 0.98, + "recall@5_hi": 1.0, + "mrr@10": 0.9566666666666667, + "mrr@10_lo": 0.928888888888889, + "mrr@10_hi": 0.98, + "ndcg@10": 0.959883370284404, + "ndcg@10_lo": 0.9374330811792257, + "ndcg@10_hi": 0.979001377697449 + }, + "meta": { + "name": "embed_bge_small", + "needs_model": true, + "needs_api": false, + "notes": "fastembed BAAI/bge-small-en-v1.5" + }, + "rows": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.issue_write", + "github.create_issue", + "github.search_issues", + "github.create_repository", + "github.add_issue_comment", + "gitlab.gitlab_create_issue", + "github.assign_copilot_to_issue", + "github.create_gist", + "github.issue_read", + "github.list_issues" + ], + "hit1": "github.issue_write", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8597186998521972, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 375, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 560, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1034, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "kind": "verb_product", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_create_issue", + "jira.jira_create_sprint", + "jira.create_sprint", + "jira.get_sprint", + "jira.create_board", + "jira.create_version", + "jira.get_board", + "jira.create_webhook", + "jira.create_filter", + "jira.update_sprint" + ], + "hit1": "jira.jira_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 345, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 493, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 902, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "kind": "verb_product", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "ranking": [ + "slack.slack_post_message", + "slack.slack_add_reaction", + "slack.slack_reply_to_thread", + "slack.get_reminder", + "slack.slack_upload_file", + "slack.update_reminder", + "slack.slack_create_channel", + "slack.slack_get_thread_replies", + "slack.create_reminder", + "slack.update_pin" + ], + "hit1": "slack.slack_post_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 795, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "kind": "resource", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "ranking": [ + "github.list_pull_requests", + "bitbucket.bitbucket_list_prs", + "github.search_pull_requests", + "github.create_pull_request", + "github.pull_request_read", + "gitlab.list_snippets", + "bitbucket.bitbucket_list_repos", + "github.list_hooks", + "github.list_notifications", + "github.list_secret_scanning_alerts" + ], + "hit1": "github.list_pull_requests", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 477, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 828, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "ranking": [ + "github.create_pull_request", + "github.search_pull_requests", + "github.list_pull_requests", + "github.pull_request_review_write", + "github.pull_request_read", + "github.merge_pull_request", + "github.update_pull_request", + "github.update_pull_request_branch", + "github.add_comment_to_pending_review", + "github.get_hook" + ], + "hit1": "github.create_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 314, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 547, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1094, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "kind": "verb_product", + "gold": [ + "postgres.query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "mysql.mysql_query", + "sqlite.sqlite_query", + "salesforce.salesforce_query", + "postgres.explain_query", + "prometheus.prom_query", + "postgres.list_schemas", + "postgres.describe_table", + "prometheus.prom_query_range", + "postgres.list_tables" + ], + "hit1": "postgres.query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 596, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "kind": "paraphrase", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "ranking": [ + "filesystem.read_text_file", + "filesystem.read_file", + "filesystem.read_media_file", + "filesystem.get_file_info", + "github.get_file_contents", + "filesystem.list_directory", + "figma.figma_get_file", + "slack.get_file", + "filesystem.directory_tree", + "figma.figma_get_file_nodes" + ], + "hit1": "filesystem.read_text_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8597186998521972, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 677, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "kind": "paraphrase", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "ranking": [ + "filesystem.list_directory", + "filesystem.list_directory_with_sizes", + "filesystem.get_file_info", + "google_drive.gdrive_list_folder", + "filesystem.read_file", + "filesystem.directory_tree", + "filesystem.read_media_file", + "github.get_file_contents", + "git.git_add", + "filesystem.search_files" + ], + "hit1": "filesystem.list_directory", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9540734464808885, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 656, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "ranking": [ + "memory.read_graph", + "memory.create_relations", + "memory.add_observations", + "memory.open_nodes", + "memory.delete_entities", + "memory.search_nodes", + "memory.delete_relations", + "memory.create_entities", + "sequential_thinking.sequentialthinking", + "memory.delete_observations" + ], + "hit1": "memory.read_graph", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 164, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 266, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 585, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "kind": "verb_product", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "ranking": [ + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "grafana.grafana_search_dashboards", + "hubspot.hubspot_search_contacts", + "confluence.confluence_search", + "github.search_pull_requests", + "slack.slack_search_messages", + "github.search_orgs", + "linear.linear_search_issues", + "zendesk.zendesk_search" + ], + "hit1": "brave_search.brave_web_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 642, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "kind": "verb_product", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "ranking": [ + "fetch.fetch", + "playwright.browser_navigate", + "confluence.confluence_search", + "github.get_file_contents", + "puppeteer.puppeteer_navigate", + "gitlab.get_snippet", + "github.search_pull_requests", + "filesystem.read_file", + "slack.get_file", + "filesystem.get_file_info" + ], + "hit1": "fetch.fetch", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 369, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 643, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "kind": "identifier", + "gold": [ + "git.git_status" + ], + "alts": [], + "ranking": [ + "git.git_status", + "git.git_show", + "git.git_log", + "git.git_diff_unstaged", + "git.git_diff", + "github.list_code_scanning_alerts", + "github.get_notification_details", + "github.get_milestone", + "github.get_code_scanning_alert", + "github.list_milestones" + ], + "hit1": "git.git_status", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 696, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "kind": "paraphrase", + "gold": [ + "git.git_commit" + ], + "alts": [], + "ranking": [ + "git.git_commit", + "git.git_reset", + "git.git_diff_staged", + "github.update_milestone", + "filesystem.move_file", + "git.git_show", + "slack.update_reminder", + "gitlab.update_milestone", + "jira.jira_transition_issue", + "salesforce.salesforce_list_opportunities" + ], + "hit1": "git.git_commit", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 640, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_get_issue", + "linear.linear_create_issue", + "linear.linear_create_comment", + "linear.linear_search_issues", + "linear.linear_assign_issue", + "linear.linear_update_issue", + "linear.linear_list_issues", + "linear.linear_create_project", + "linear.create_label", + "linear.create_attachment" + ], + "hit1": "linear.linear_get_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 275, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 875, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "kind": "paraphrase", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_project", + "linear.linear_list_projects", + "linear.linear_create_issue", + "linear.linear_update_issue", + "linear.linear_list_issues", + "linear.create_attachment", + "linear.create_label", + "linear.linear_get_issue", + "linear.linear_create_comment", + "linear.linear_assign_issue" + ], + "hit1": "linear.linear_create_project", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 284, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 483, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 870, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "ranking": [ + "github.add_issue_comment", + "github.update_issue_comment", + "github.add_comment_to_pending_review", + "github.issue_write", + "github.get_discussion_comments", + "jira.jira_add_comment", + "github.pull_request_review_write", + "github.get_discussion", + "github.assign_copilot_to_issue", + "github.issue_read" + ], + "hit1": "github.add_issue_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 388, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 617, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1118, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "kind": "ambiguous", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "ranking": [ + "github.merge_pull_request", + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_get_merge_request", + "github.update_pull_request_branch", + "github.create_pull_request", + "gitlab.gitlab_list_merge_requests", + "gitlab.gitlab_create_merge_request", + "github.update_pull_request", + "github.search_pull_requests", + "github.pull_request_read" + ], + "hit1": "github.merge_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 528, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1003, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "kind": "verb_product", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "ranking": [ + "github.actions_run_trigger", + "github.actions_get", + "github.actions_list", + "github.actions_get_job_logs", + "github.update_check_run", + "github.dismiss_notification", + "github.get_check_run", + "github.create_check_run", + "github.update_hook", + "github.get_hook" + ], + "hit1": "github.actions_run_trigger", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 457, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 780, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "kind": "resource", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "ranking": [ + "github.get_dependabot_alert", + "github.list_dependabot_alerts", + "github.get_code_scanning_alert", + "github.get_secret_scanning_alert", + "github.list_code_scanning_alerts", + "github.list_secret_scanning_alerts", + "pagerduty.pagerduty_list_incidents", + "github.get_notification_details", + "slack.get_reminder", + "sentry.sentry_list_events" + ], + "hit1": "github.get_dependabot_alert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8597186998521972, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 286, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 481, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 819, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "kind": "verb_product", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_issues", + "sentry.sentry_list_events", + "sentry.sentry_list_projects", + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "linear.linear_list_issues", + "bitbucket.bitbucket_list_prs", + "pagerduty.pagerduty_list_incidents", + "slack.list_pins", + "hubspot.hubspot_list_contacts" + ], + "hit1": "sentry.sentry_list_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 766, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_create_customer", + "stripe.stripe_create_payment_intent", + "stripe.stripe_list_invoices", + "stripe.stripe_get_customer", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "stripe.stripe_create_refund", + "stripe.stripe_list_subscriptions", + "stripe.stripe_list_customers" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "kind": "paraphrase", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_refund", + "stripe.stripe_cancel_subscription", + "stripe.stripe_get_payment_intent", + "stripe.stripe_get_customer", + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_price", + "stripe.stripe_create_customer", + "stripe.stripe_create_product" + ], + "hit1": "stripe.stripe_create_refund", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 197, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "kind": "verb_product", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "ranking": [ + "gmail.gmail_send_email", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "gmail.gmail_create_draft", + "gmail.gmail_modify_email", + "google_drive.gdrive_share_file", + "gmail.gmail_search_emails", + "gmail.gmail_list_labels", + "hubspot.hubspot_create_contact", + "shopify.shopify_create_order" + ], + "hit1": "gmail.gmail_send_email", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 743, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "kind": "verb_product", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "ranking": [ + "gmail.gmail_search_emails", + "gmail.gmail_get_email", + "gmail.gmail_list_labels", + "hubspot.hubspot_search_contacts", + "gmail.gmail_trash_email", + "gmail.gmail_modify_email", + "gmail.gmail_create_draft", + "slack.slack_search_messages", + "gmail.gmail_send_email", + "github.search_users" + ], + "hit1": "gmail.gmail_search_emails", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 678, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "kind": "paraphrase", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_update_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_share_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "filesystem.write_file", + "slack.slack_upload_file", + "aws_s3.s3_put_object", + "filesystem.edit_file" + ], + "hit1": "google_drive.gdrive_update_file", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 307, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 806, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_create_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "google_calendar.gcal_delete_event", + "sentry.sentry_list_events", + "slack.create_reminder", + "pagerduty.pagerduty_list_oncalls", + "clickup.clickup_create_task", + "pagerduty.pagerduty_create_incident" + ], + "hit1": "google_calendar.gcal_create_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 317, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 858, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_list_pods", + "kubernetes.k8s_list_services", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_get_events", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_get_pod", + "docker.docker_images", + "kubernetes.k8s_apply", + "kubernetes.k8s_scale_deployment", + "okta.okta_list_users" + ], + "hit1": "kubernetes.k8s_list_pods", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 661, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_get_events", + "docker.docker_logs", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_list_services", + "kubernetes.k8s_list_deployments", + "github.actions_get_job_logs", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_apply" + ], + "hit1": "kubernetes.k8s_pod_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "ranking": [ + "docker.docker_run", + "docker.docker_exec", + "docker.docker_ps", + "docker.docker_stop", + "docker.docker_logs", + "docker.docker_build", + "docker.docker_images", + "docker.docker_rm", + "terraform.tf_plan", + "terraform.tf_apply" + ], + "hit1": "docker.docker_run", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 594, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "kind": "verb_product", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "ranking": [ + "docker.docker_logs", + "kubernetes.k8s_pod_logs", + "docker.docker_images", + "docker.docker_run", + "docker.docker_ps", + "git.git_log", + "docker.docker_stop", + "docker.docker_exec", + "datadog.datadog_list_logs", + "cloudflare.cf_list_dns" + ], + "hit1": "docker.docker_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 678, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "kind": "ambiguous", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "ranking": [ + "playwright.browser_navigate", + "puppeteer.puppeteer_navigate", + "playwright.browser_tab_new", + "playwright.browser_click", + "playwright.browser_press_key", + "fetch.fetch", + "playwright.browser_take_screenshot", + "playwright.browser_type", + "playwright.browser_tab_list", + "confluence.confluence_search" + ], + "hit1": "playwright.browser_navigate", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 613, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "kind": "ambiguous", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "ranking": [ + "playwright.browser_take_screenshot", + "puppeteer.puppeteer_screenshot", + "playwright.browser_snapshot", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_acknowledge_incident", + "playwright.browser_evaluate", + "playwright.browser_click", + "playwright.browser_press_key", + "pagerduty.pagerduty_resolve_incident", + "gitlab.get_snippet" + ], + "hit1": "playwright.browser_take_screenshot", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 346, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 633, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_create_merge_request", + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_get_merge_request", + "gitlab.gitlab_list_merge_requests", + "github.merge_pull_request", + "gitlab.create_release", + "gitlab.create_pipeline", + "gitlab.create_snippet", + "gitlab.gitlab_create_pipeline", + "gitlab.create_job" + ], + "hit1": "gitlab.gitlab_create_merge_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 943, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "kind": "verb_product", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "ranking": [ + "notion.notion_search", + "notion.notion_get_page", + "notion.notion_create_page", + "notion.notion_query_database", + "notion.notion_update_page", + "notion.notion_list_users", + "notion.notion_create_database", + "notion.notion_get_comments", + "notion.notion_get_block", + "memory.search_nodes" + ], + "hit1": "notion.notion_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 383, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 672, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "kind": "verb_product", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "ranking": [ + "notion.notion_create_page", + "notion.notion_get_page", + "notion.notion_create_database", + "notion.notion_search", + "notion.notion_update_page", + "notion.create_data_source", + "notion.create_users_token", + "notion.create_comment", + "notion.notion_list_users", + "notion.notion_get_comments" + ], + "hit1": "notion.notion_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "kind": "verb_product", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "ranking": [ + "notion.notion_query_database", + "notion.notion_search", + "notion.notion_create_database", + "notion.get_data_source", + "notion.notion_get_page", + "notion.notion_list_users", + "notion.notion_create_page", + "notion.notion_get_block", + "notion.get_users_token", + "notion.update_data_source" + ], + "hit1": "notion.notion_query_database", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_search", + "jira.jira_create_issue", + "jira.jira_create_sprint", + "google_calendar.gcal_create_event", + "jira.get_board", + "hubspot.hubspot_create_deal" + ], + "hit1": "zendesk.zendesk_get_ticket", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 855, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_search_contacts", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_create_deal", + "slack.create_usergroup", + "slack.get_usergroup", + "twilio.twilio_make_call", + "pagerduty.pagerduty_create_incident", + "slack.update_usergroup" + ], + "hit1": "hubspot.hubspot_create_contact", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 786, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "kind": "verb_product", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_create_lead", + "salesforce.salesforce_convert_lead", + "salesforce.salesforce_create_record", + "salesforce.salesforce_list_opportunities", + "slack.create_pin", + "linear.create_user", + "hubspot.hubspot_create_deal", + "salesforce.salesforce_get_record", + "slack.create_reminder", + "github.create_pull_request" + ], + "hit1": "salesforce.salesforce_create_lead", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 915, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "kind": "identifier", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_query", + "mysql.mysql_query", + "postgres.query", + "sqlite.sqlite_query", + "prometheus.prom_query", + "postgres.explain_query", + "prometheus.prom_query_range", + "datadog.datadog_list_monitors", + "supabase.supabase_select", + "hubspot.hubspot_search_contacts" + ], + "hit1": "salesforce.salesforce_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 187, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "kind": "verb_product", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "ranking": [ + "shopify.shopify_list_orders", + "shopify.shopify_get_order", + "shopify.shopify_create_order", + "shopify.shopify_list_products", + "shopify.shopify_list_customers", + "shopify.shopify_get_product", + "shopify.shopify_update_product", + "shopify.shopify_create_product", + "stripe.stripe_list_subscriptions", + "stripe.stripe_list_invoices" + ], + "hit1": "shopify.shopify_list_orders", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 210, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "sentry.sentry_list_events", + "playwright.browser_press_key", + "jira.jira_create_issue", + "google_calendar.gcal_create_event" + ], + "hit1": "pagerduty.pagerduty_get_incident", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 823, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "kind": "paraphrase", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_list_oncalls", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_resolve_incident", + "twilio.twilio_make_call", + "playwright.browser_press_key", + "hubspot.hubspot_get_contact", + "slack.get_bookmark" + ], + "hit1": "pagerduty.pagerduty_list_oncalls", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 253, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 719, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "kind": "verb_product", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "ranking": [ + "datadog.datadog_query_metrics", + "datadog.datadog_list_monitors", + "datadog.datadog_list_logs", + "datadog.datadog_create_monitor", + "datadog.datadog_get_monitor", + "memory.search_nodes", + "slack.slack_search_messages", + "notion.notion_search", + "github.search_pull_requests", + "prometheus.prom_list_metrics" + ], + "hit1": "datadog.datadog_query_metrics", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 284, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 756, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "kind": "verb_product", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "ranking": [ + "datadog.datadog_list_logs", + "datadog.datadog_list_monitors", + "datadog.datadog_create_monitor", + "datadog.datadog_get_monitor", + "datadog.datadog_query_metrics", + "github.search_pull_requests", + "memory.search_nodes", + "slack.slack_search_messages", + "github.search_users", + "github.search_issues" + ], + "hit1": "datadog.datadog_list_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 298, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 773, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_list_buckets", + "aws_s3.s3_list_objects", + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "bitbucket.bitbucket_list_repos", + "bitbucket.bitbucket_list_prs", + "aws_s3.s3_put_object", + "jira.jira_list_projects", + "salesforce.salesforce_list_opportunities", + "terraform.tf_state_list" + ], + "hit1": "aws_s3.s3_list_buckets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 624, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "kind": "paraphrase", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_put_object", + "aws_s3.s3_get_object", + "aws_s3.s3_list_objects", + "aws_s3.s3_delete_object", + "slack.slack_upload_file", + "twilio.twilio_send_sms", + "aws_s3.s3_list_buckets", + "jira.update_attachment", + "linear.update_attachment", + "gitlab.update_snippet" + ], + "hit1": "aws_s3.s3_put_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 734, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_purge_cache", + "slack.delete_bookmark", + "cloudflare.cf_list_zones", + "slack.delete_canvase", + "cloudflare.cf_list_dns", + "cloudflare.cf_create_dns", + "memory.delete_observations", + "redis.redis_delete", + "aws_s3.s3_delete_object", + "linear.delete_attachment" + ], + "hit1": "cloudflare.cf_purge_cache", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 618, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "cloudflare.cf_purge_cache", + "salesforce.salesforce_create_record", + "cloudflare.cf_list_zones", + "datadog.datadog_create_monitor", + "slack.create_file", + "salesforce.salesforce_get_record", + "docker.docker_logs", + "slack.create_bookmark" + ], + "hit1": "cloudflare.cf_create_dns", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 808, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_deployments", + "vercel.vercel_get_deployment", + "vercel.vercel_list_projects", + "kubernetes.k8s_list_deployments", + "vercel.vercel_create_project", + "github.list_packages", + "gitlab.list_releases", + "github.list_releases", + "docker.docker_images", + "kubernetes.k8s_list_services" + ], + "hit1": "vercel.vercel_list_deployments", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "ranking": [ + "supabase.supabase_insert", + "supabase.supabase_update", + "supabase.supabase_select", + "supabase.supabase_invoke_fn", + "supabase.supabase_list_tables", + "mongodb.mongo_insert", + "linear.update_cycle", + "linear.update_attachment", + "linear.update_user", + "linear.linear_create_comment" + ], + "hit1": "supabase.supabase_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 691, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "kind": "paraphrase", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "ranking": [ + "figma.figma_get_images", + "figma.figma_get_file", + "figma.figma_get_file_nodes", + "figma.figma_get_comments", + "figma.figma_post_comment", + "docker.docker_images", + "linear.update_attachment", + "linear.get_attachment", + "filesystem.read_media_file", + "linear.create_attachment" + ], + "hit1": "figma.figma_get_images", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 670, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "kind": "verb_product", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "ranking": [ + "twilio.twilio_send_sms", + "twilio.twilio_list_messages", + "twilio.twilio_make_call", + "slack.slack_reply_to_thread", + "slack.slack_add_reaction", + "discord.discord_send_message", + "slack.slack_post_message", + "slack.slack_get_thread_replies", + "gmail.gmail_send_email", + "grafana.grafana_create_annotation" + ], + "hit1": "twilio.twilio_send_sms", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 820, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "kind": "verb_product", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "ranking": [ + "okta.okta_create_user", + "okta.okta_get_user", + "okta.okta_list_users", + "okta.okta_deactivate_user", + "notion.create_users_token", + "linear.create_user", + "asana.asana_create_task", + "asana.asana_create_project", + "slack.create_usergroup", + "jira.create_attachment" + ], + "hit1": "okta.okta_create_user", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 806, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query", + "prometheus.prom_query_range", + "prometheus.prom_list_metrics", + "memory.search_nodes", + "datadog.datadog_query_metrics", + "github.search_pull_requests", + "hubspot.hubspot_search_contacts", + "elasticsearch.es_search", + "datadog.datadog_list_monitors", + "notion.notion_search" + ], + "hit1": "prometheus.prom_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "kind": "verb_product", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "ranking": [ + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "github.search_pull_requests", + "grafana.grafana_create_annotation", + "github.search_orgs", + "github.search_users", + "github.search_repositories", + "github.search_issues", + "zendesk.zendesk_search", + "hubspot.hubspot_search_contacts" + ], + "hit1": "grafana.grafana_search_dashboards", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "kind": "verb_product", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "ranking": [ + "circleci.circleci_trigger_pipeline", + "circleci.circleci_get_pipeline", + "circleci.circleci_list_pipelines", + "gitlab.list_pipelines", + "hubspot.hubspot_create_deal", + "gitlab.update_pipeline", + "github.actions_run_trigger", + "slack.slack_add_reaction", + "gitlab.get_pipeline", + "gitlab.gitlab_create_pipeline" + ], + "hit1": "circleci.circleci_trigger_pipeline", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 773, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "kind": "identifier", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "ranking": [ + "terraform.tf_plan", + "terraform.tf_apply", + "terraform.tf_state_list", + "github.update_check_run", + "hubspot.hubspot_create_deal", + "github.get_check_run", + "docker.docker_run", + "github.update_runner", + "github.create_check_run", + "github.update_environment" + ], + "hit1": "terraform.tf_plan", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 332, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 710, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "kind": "verb_product", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "ranking": [ + "asana.asana_create_task", + "asana.asana_create_project", + "asana.asana_list_tasks", + "asana.asana_get_task", + "asana.asana_update_task", + "clickup.clickup_create_task", + "slack.create_reminder", + "okta.okta_create_user", + "gitlab.create_snippet", + "notion.create_data_source" + ], + "hit1": "asana.asana_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 264, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 856, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "ranking": [ + "trello.trello_create_card", + "trello.trello_create_board", + "trello.trello_add_comment", + "trello.trello_list_cards", + "stripe.stripe_create_invoice", + "grafana.grafana_create_annotation", + "stripe.stripe_create_customer", + "stripe.stripe_create_product", + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_price" + ], + "hit1": "trello.trello_create_card", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 752, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_create_task", + "clickup.clickup_list_tasks", + "clickup.clickup_get_task", + "clickup.clickup_update_task", + "puppeteer.puppeteer_click", + "playwright.browser_click", + "puppeteer.puppeteer_fill", + "pagerduty.pagerduty_create_incident", + "puppeteer.puppeteer_screenshot", + "asana.asana_create_task" + ], + "hit1": "clickup.clickup_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 782, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "kind": "verb_product", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "ranking": [ + "confluence.confluence_create_page", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "confluence.confluence_search", + "notion.notion_create_page", + "github.create_artifact", + "notion.create_users_token", + "shopify.shopify_create_product", + "pagerduty.pagerduty_create_incident", + "github.search_code" + ], + "hit1": "confluence.confluence_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 866, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "kind": "verb_product", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "discord.discord_add_reaction", + "slack.slack_post_message", + "discord.discord_get_channel", + "slack.slack_add_reaction", + "slack.slack_reply_to_thread", + "gmail.gmail_trash_email", + "discord.discord_list_channels", + "twilio.twilio_send_sms", + "slack.slack_get_thread_replies" + ], + "hit1": "discord.discord_send_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_search", + "confluence.confluence_search", + "hubspot.hubspot_search_contacts", + "github.search_pull_requests", + "github.search_orgs", + "elasticsearch.es_get_doc", + "github.search_users", + "grafana.grafana_search_dashboards", + "memory.search_nodes", + "datadog.datadog_list_logs" + ], + "hit1": "elasticsearch.es_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 635, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "redis.redis_set", + "redis.redis_delete", + "redis.redis_keys", + "playwright.browser_press_key", + "slack.get_pin", + "circleci.circleci_get_pipeline", + "figma.figma_get_file", + "clickup.clickup_get_task", + "github.get_milestone" + ], + "hit1": "redis.redis_get", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_find", + "mongodb.mongo_list_collections", + "mongodb.mongo_delete", + "mongodb.mongo_update", + "mongodb.mongo_insert", + "elasticsearch.es_get_doc", + "filesystem.search_files", + "confluence.confluence_search", + "datadog.datadog_list_logs", + "hubspot.hubspot_search_contacts" + ], + "hit1": "mongodb.mongo_find", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "kind": "verb_product", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "ranking": [ + "sqlite.sqlite_list_tables", + "mysql.mysql_list_tables", + "supabase.supabase_list_tables", + "postgres.list_tables", + "sqlite.sqlite_describe_table", + "okta.okta_list_users", + "postgres.list_schemas", + "datadog.datadog_list_monitors", + "filesystem.list_directory", + "gitlab.list_wikis" + ], + "hit1": "sqlite.sqlite_list_tables", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 142, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 240, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 526, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "kind": "verb_product", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "ranking": [ + "postgres.describe_table", + "postgres.explain_query", + "sqlite.sqlite_describe_table", + "postgres.list_schemas", + "postgres.list_tables", + "mysql.mysql_list_tables", + "postgres.query", + "supabase.supabase_insert", + "supabase.supabase_select", + "notion.notion_query_database" + ], + "hit1": "postgres.describe_table", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 274, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 593, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_repository", + "github.create_issue", + "github.issue_write", + "github.create_gist", + "gitlab.gitlab_create_issue", + "github.create_environment", + "github.create_package", + "github.create_artifact", + "bitbucket.bitbucket_create_issue", + "github.create_runner" + ], + "hit1": "github.create_repository", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.66967181649423, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 367, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 536, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1012, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "ranking": [ + "linear.create_user", + "gmail.gmail_create_draft", + "notion.create_data_source", + "linear.create_cycle", + "notion.create_comment", + "linear.create_comment", + "linear.create_attachment", + "notion.create_users_token", + "stripe.stripe_create_product", + "okta.okta_create_user" + ], + "hit1": "linear.create_user", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 39, + "k3_calls": 13, + "k3_tokens": 3554, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 8, + "k5_tokens": 3600, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3528, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q071", + "query": "list-directory", + "kind": "identifier", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "ranking": [ + "filesystem.list_directory", + "filesystem.list_directory_with_sizes", + "okta.okta_list_users", + "mongodb.mongo_list_collections", + "filesystem.list_allowed_directories", + "clickup.clickup_list_tasks", + "terraform.tf_state_list", + "sentry.sentry_list_projects", + "docker.docker_images", + "gmail.gmail_list_labels" + ], + "hit1": "filesystem.list_directory", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 271, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 489, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "kind": "paraphrase", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "ranking": [ + "sequential_thinking.sequentialthinking", + "memory.create_relations", + "memory.add_observations", + "memory.search_nodes", + "memory.create_entities", + "memory.delete_relations", + "notion.update_comment", + "notion.notion_search", + "memory.open_nodes", + "notion.notion_append_block" + ], + "hit1": "sequential_thinking.sequentialthinking", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 650, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "kind": "paraphrase", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "ranking": [ + "time.get_current_time", + "zendesk.zendesk_list_tickets", + "google_calendar.gcal_list_events", + "time.convert_time", + "grafana.grafana_create_annotation", + "jira.jira_create_sprint", + "playwright.browser_wait_for", + "pagerduty.pagerduty_list_oncalls", + "slack.slack_add_reaction", + "pagerduty.pagerduty_list_incidents" + ], + "hit1": "time.get_current_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9238850086262381, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 785, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "kind": "verb_product", + "gold": [ + "time.convert_time" + ], + "alts": [], + "ranking": [ + "time.convert_time", + "time.get_current_time", + "grafana.grafana_create_annotation", + "playwright.browser_wait_for", + "datadog.datadog_query_metrics", + "cloudflare.cf_list_zones", + "linear.update_cycle", + "google_calendar.gcal_list_events", + "linear.get_cycle", + "cloudflare.cf_list_dns" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "kind": "verb_product", + "gold": [ + "github.star_repository" + ], + "alts": [], + "ranking": [ + "github.star_repository", + "github.unstar_repository", + "github.list_starred_repositories", + "github.create_or_update_file", + "github.create_gist", + "github.get_file_blame", + "github.create_repository", + "github.list_code_scanning_alerts", + "github.list_pull_requests", + "github.create_pull_request" + ], + "hit1": "github.star_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 950, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "kind": "resource", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "ranking": [ + "github.fork_repository", + "github.unstar_repository", + "github.merge_pull_request", + "github.star_repository", + "bitbucket.bitbucket_create_pr", + "github.delete_repository", + "github.get_commit", + "github.create_repository", + "github.create_or_update_file", + "github.get_repository_tree" + ], + "hit1": "github.fork_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 277, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 468, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 950, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "kind": "verb_product", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "ranking": [ + "github.get_latest_release", + "gitlab.get_release", + "github.get_release_by_tag", + "github.get_milestone", + "github.list_releases", + "github.get_notification_details", + "github.get_gist", + "github.get_check_run", + "github.get_code_scanning_alert", + "github.get_package" + ], + "hit1": "github.get_latest_release", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "kind": "resource", + "gold": [ + "github.create_gist" + ], + "alts": [], + "ranking": [ + "github.create_gist", + "github.update_gist", + "github.get_gist", + "gitlab.create_snippet", + "github.list_gists", + "github.create_milestone", + "notion.create_data_source", + "github.create_hook", + "gitlab.create_label", + "gitlab.create_wiki" + ], + "hit1": "github.create_gist", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 383, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 887, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "kind": "verb_product", + "gold": [ + "github.search_code" + ], + "alts": [], + "ranking": [ + "github.search_code", + "github.search_orgs", + "github.search_users", + "github.search_pull_requests", + "github.search_repositories", + "github.search_commits", + "github.get_code_scanning_alert", + "github.get_check_run", + "github.search_issues", + "github.list_code_scanning_alerts" + ], + "hit1": "github.search_code", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 722, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "kind": "verb_product", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "ranking": [ + "github.list_notifications", + "github.get_notification_details", + "github.mark_all_notifications_read", + "github.list_code_scanning_alerts", + "github.list_secret_scanning_alerts", + "github.list_dependabot_alerts", + "github.dismiss_notification", + "github.list_milestones", + "github.list_gists", + "github.get_code_scanning_alert" + ], + "hit1": "github.list_notifications", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 176, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "kind": "verb_product", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "ranking": [ + "github.assign_copilot_to_issue", + "linear.linear_assign_issue", + "github.request_copilot_review", + "jira.jira_assign_issue", + "pagerduty.pagerduty_resolve_incident", + "jira.jira_get_issue", + "linear.linear_create_comment", + "linear.linear_list_issues", + "github.create_issue", + "linear.linear_get_issue" + ], + "hit1": "github.assign_copilot_to_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 301, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 434, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 826, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "kind": "resource", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "ranking": [ + "github.get_secret_scanning_alert", + "github.list_secret_scanning_alerts", + "github.get_code_scanning_alert", + "github.list_code_scanning_alerts", + "github.get_dependabot_alert", + "github.list_dependabot_alerts", + "github.get_notification_details", + "github.mark_all_notifications_read", + "pagerduty.pagerduty_get_incident", + "github.list_notifications" + ], + "hit1": "github.get_secret_scanning_alert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8597186998521972, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 483, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "ranking": [ + "github.create_or_update_file", + "github.update_gist", + "github.create_gist", + "github.projects_write", + "github.update_environment", + "github.update_package", + "github.issue_write", + "github.update_hook", + "github.update_check_run", + "slack.update_file" + ], + "hit1": "github.create_or_update_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 327, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 476, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 878, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "kind": "resource", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "ranking": [ + "github.list_repository_collaborators", + "github.list_labels", + "github.list_starred_repositories", + "github.list_tags", + "github.list_releases", + "github.list_discussions", + "github.actions_list", + "github.get_team_members", + "bitbucket.bitbucket_list_repos", + "github.list_branches" + ], + "hit1": "github.list_repository_collaborators", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 365, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 696, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "kind": "verb_product", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "ranking": [ + "slack.slack_reply_to_thread", + "slack.slack_get_thread_replies", + "slack.slack_add_reaction", + "slack.slack_post_message", + "slack.get_reminder", + "slack.update_reminder", + "slack.slack_search_messages", + "slack.update_pin", + "slack.create_reminder", + "slack.update_file" + ], + "hit1": "slack.slack_reply_to_thread", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 277, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "kind": "verb_product", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "ranking": [ + "slack.slack_add_reaction", + "discord.discord_add_reaction", + "slack.slack_reply_to_thread", + "slack.slack_get_thread_replies", + "slack.update_canvase", + "slack.update_reminder", + "slack.get_reminder", + "slack.slack_post_message", + "slack.get_canvase", + "slack.create_reminder" + ], + "hit1": "slack.slack_add_reaction", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 289, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 440, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 795, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "kind": "verb_product", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "ranking": [ + "slack.slack_search_messages", + "slack.slack_get_channel_history", + "slack.slack_add_reaction", + "github.search_pull_requests", + "slack.slack_get_thread_replies", + "slack.get_reminder", + "slack.slack_reply_to_thread", + "slack.get_canvase", + "slack.slack_list_channels", + "slack.slack_post_message" + ], + "hit1": "slack.slack_search_messages", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 720, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "kind": "verb_product", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "ranking": [ + "slack.slack_create_channel", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_post_message", + "slack.slack_add_reaction", + "slack.slack_reply_to_thread", + "slack.slack_get_thread_replies", + "slack.slack_set_channel_topic", + "slack.create_usergroup", + "slack.slack_upload_file" + ], + "hit1": "slack.slack_create_channel", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "kind": "verb_product", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_transition_issue", + "jira.jira_update_issue", + "jira.jira_add_comment", + "jira.jira_assign_issue", + "jira.jira_create_issue", + "jira.jira_get_issue", + "jira.update_sprint", + "jira.jira_search_issues", + "jira.get_sprint", + "jira.create_version" + ], + "hit1": "jira.jira_transition_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 745, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "kind": "identifier", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "ranking": [ + "jira.jira_search_issues", + "jira.get_filter", + "jira.create_filter", + "jira.jira_get_issue", + "jira.update_filter", + "jira.get_webhook", + "jira.get_version", + "jira.jira_list_projects", + "jira.delete_filter", + "jira.list_filters" + ], + "hit1": "jira.jira_search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 618, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "kind": "verb_product", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "ranking": [ + "jira.jira_add_worklog", + "jira.jira_list_projects", + "jira.jira_create_issue", + "jira.jira_add_comment", + "jira.create_webhook", + "jira.create_sprint", + "jira.create_attachment", + "jira.create_version", + "jira.create_board", + "jira.create_filter" + ], + "hit1": "jira.jira_add_worklog", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 909, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "kind": "verb_product", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "ranking": [ + "linear.linear_list_teams", + "linear.linear_list_projects", + "linear.linear_list_issues", + "linear.linear_create_issue", + "linear.linear_create_project", + "github.get_team_members", + "linear.list_cycles", + "linear.linear_get_issue", + "linear.get_cycle", + "linear.list_users" + ], + "hit1": "linear.linear_list_teams", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 398, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "ranking": [ + "linear.linear_create_comment", + "linear.linear_get_issue", + "linear.linear_assign_issue", + "linear.linear_search_issues", + "linear.get_comment", + "linear.linear_update_issue", + "linear.update_comment", + "linear.linear_create_issue", + "linear.delete_comment", + "linear.linear_list_issues" + ], + "hit1": "linear.linear_create_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 771, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "kind": "paraphrase", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "ranking": [ + "filesystem.move_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "filesystem.write_file", + "filesystem.read_file", + "filesystem.edit_file", + "slack.update_file", + "slack.get_file", + "git.git_add", + "slack.create_file" + ], + "hit1": "filesystem.move_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 225, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "kind": "paraphrase", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "ranking": [ + "filesystem.search_files", + "filesystem.directory_tree", + "memory.search_nodes", + "google_drive.gdrive_search", + "redis.redis_keys", + "filesystem.get_file_info", + "filesystem.list_directory_with_sizes", + "filesystem.list_directory", + "notion.notion_search", + "github.search_code" + ], + "hit1": "filesystem.search_files", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 304, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 609, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "kind": "paraphrase", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "ranking": [ + "filesystem.edit_file", + "filesystem.move_file", + "filesystem.write_file", + "slack.update_file", + "linear.update_attachment", + "slack.create_file", + "linear.delete_attachment", + "linear.create_attachment", + "slack.update_bookmark", + "google_drive.gdrive_update_file" + ], + "hit1": "filesystem.edit_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 785, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "ranking": [ + "memory.read_graph", + "memory.open_nodes", + "memory.search_nodes", + "memory.add_observations", + "memory.delete_entities", + "memory.delete_relations", + "memory.create_relations", + "memory.create_entities", + "datadog.datadog_list_monitors", + "datadog.datadog_query_metrics" + ], + "hit1": "memory.read_graph", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 162, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 270, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 582, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "kind": "verb_product", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "memory.open_nodes", + "filesystem.search_files", + "hubspot.hubspot_search_contacts", + "memory.read_graph", + "github.search_orgs", + "github.search_pull_requests", + "datadog.datadog_list_logs", + "memory.delete_entities", + "github.search_repositories" + ], + "hit1": "memory.search_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 285, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "kind": "verb_product", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "ranking": [ + "git.git_diff_unstaged", + "git.git_diff", + "git.git_diff_staged", + "git.git_show", + "git.git_reset", + "git.git_status", + "git.git_log", + "git.git_commit", + "github.create_gist", + "git.git_add" + ], + "hit1": "git.git_diff_unstaged", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 620, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "kind": "identifier", + "gold": [ + "git.git_add" + ], + "alts": [], + "ranking": [ + "git.git_add", + "git.git_diff_staged", + "git.git_commit", + "github.update_gist", + "git.git_reset", + "github.create_or_update_file", + "github.create_gist", + "git.git_diff", + "git.git_show", + "git.git_branch" + ], + "hit1": "git.git_add", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 201, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 723, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "kind": "verb_product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "ranking": [ + "shopify.shopify_create_product", + "shopify.shopify_get_product", + "shopify.shopify_create_order", + "shopify.shopify_update_product", + "shopify.shopify_get_order", + "shopify.shopify_list_products", + "stripe.stripe_create_product", + "shopify.shopify_list_customers", + "shopify.shopify_list_orders", + "stripe.stripe_create_invoice" + ], + "hit1": "shopify.shopify_create_product", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 365, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 670, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "kind": "verb_product", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "ranking": [ + "stripe.stripe_cancel_subscription", + "stripe.stripe_list_subscriptions", + "stripe.stripe_create_refund", + "stripe.stripe_list_invoices", + "stripe.stripe_create_invoice", + "stripe.stripe_get_payment_intent", + "stripe.stripe_list_customers", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_customer", + "stripe.stripe_create_customer" + ], + "hit1": "stripe.stripe_cancel_subscription", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 629, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "kind": "resource", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_refund", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_invoice", + "stripe.stripe_create_price", + "hubspot.hubspot_create_deal", + "stripe.stripe_create_product", + "stripe.stripe_create_customer", + "slack.create_reminder", + "stripe.stripe_list_invoices" + ], + "hit1": "stripe.stripe_create_payment_intent", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "sentry.sentry_list_events", + "slack.get_bookmark", + "slack.get_reminder", + "notion.notion_get_page" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "kind": "verb_product", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "ranking": [ + "datadog.datadog_create_monitor", + "datadog.datadog_get_monitor", + "datadog.datadog_list_monitors", + "datadog.datadog_query_metrics", + "datadog.datadog_list_logs", + "memory.add_observations", + "notion.list_data_sources", + "github.list_runners", + "memory.create_entities", + "slack.list_bookmarks" + ], + "hit1": "datadog.datadog_create_monitor", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 799, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "kind": "paraphrase", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_issues", + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "sentry.sentry_list_events", + "sentry.sentry_list_projects", + "pagerduty.pagerduty_resolve_incident", + "jira.jira_assign_issue", + "jira.jira_get_issue", + "github.assign_copilot_to_issue", + "linear.linear_update_issue" + ], + "hit1": "sentry.sentry_list_issues", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 274, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 800, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_apply", + "kubernetes.k8s_list_services", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_delete", + "kubernetes.k8s_get_events", + "vercel.vercel_get_deployment" + ], + "hit1": "kubernetes.k8s_scale_deployment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 241, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_apply", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_get_events", + "kubernetes.k8s_pod_logs", + "terraform.tf_apply", + "kubernetes.k8s_delete" + ], + "hit1": "kubernetes.k8s_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "ranking": [ + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_run", + "docker.docker_logs", + "docker.docker_exec", + "docker.docker_ps", + "docker.docker_images", + "github.list_environments", + "docker.docker_build", + "cloudflare.cf_purge_cache" + ], + "hit1": "docker.docker_stop", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 636, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "kind": "verb_product", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "ranking": [ + "docker.docker_build", + "docker.docker_images", + "docker.docker_run", + "docker.docker_logs", + "docker.docker_ps", + "docker.docker_stop", + "github.create_environment", + "docker.docker_exec", + "github.create_or_update_file", + "figma.figma_get_images" + ], + "hit1": "docker.docker_build", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "kind": "verb_product", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "ranking": [ + "playwright.browser_click", + "playwright.browser_press_key", + "playwright.browser_type", + "playwright.browser_navigate", + "puppeteer.puppeteer_click", + "playwright.browser_tab_list", + "playwright.browser_tab_new", + "playwright.browser_wait_for", + "playwright.browser_tab_close", + "playwright.browser_take_screenshot" + ], + "hit1": "playwright.browser_click", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 318, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 594, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "kind": "paraphrase", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "ranking": [ + "playwright.browser_type", + "puppeteer.puppeteer_fill", + "playwright.browser_click", + "playwright.browser_press_key", + "playwright.browser_fill_form", + "playwright.browser_navigate", + "playwright.browser_tab_new", + "playwright.browser_evaluate", + "playwright.browser_wait_for", + "playwright.browser_take_screenshot" + ], + "hit1": "playwright.browser_type", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 624, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_list_projects", + "gitlab.gitlab_list_issues", + "gitlab.gitlab_list_merge_requests", + "gitlab.list_pipelines", + "gitlab.list_jobs", + "gitlab.list_snippets", + "gitlab.list_wikis", + "gitlab.list_milestones", + "gitlab.list_releases", + "vercel.vercel_list_projects" + ], + "hit1": "gitlab.gitlab_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 372, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 735, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "kind": "verb_product", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "ranking": [ + "bitbucket.bitbucket_create_pr", + "bitbucket.bitbucket_list_prs", + "github.create_pull_request", + "bitbucket.bitbucket_create_issue", + "github.pull_request_review_write", + "github.search_pull_requests", + "github.pull_request_read", + "github.update_pull_request_branch", + "github.merge_pull_request", + "github.update_pull_request" + ], + "hit1": "bitbucket.bitbucket_create_pr", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 372, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 606, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1091, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "kind": "verb_product", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "ranking": [ + "gmail.gmail_create_draft", + "gmail.gmail_modify_email", + "gmail.gmail_get_email", + "gmail.gmail_list_labels", + "gmail.gmail_send_email", + "gmail.gmail_trash_email", + "hubspot.hubspot_create_contact", + "stripe.stripe_create_customer", + "shopify.shopify_create_order", + "google_drive.gdrive_share_file" + ], + "hit1": "gmail.gmail_create_draft", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 739, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "kind": "verb_product", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_share_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_update_file", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "filesystem.move_file", + "filesystem.read_media_file", + "filesystem.get_file_info", + "filesystem.write_file" + ], + "hit1": "google_drive.gdrive_share_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 271, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "kind": "verb_product", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_list_calendars", + "google_calendar.gcal_list_events", + "google_calendar.gcal_create_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_delete_event", + "pagerduty.pagerduty_list_oncalls", + "gmail.gmail_list_labels", + "kubernetes.k8s_get_events", + "sentry.sentry_list_events", + "pagerduty.pagerduty_list_incidents" + ], + "hit1": "google_calendar.gcal_list_calendars", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "slack.delete_reminder", + "slack.delete_canvase", + "github.delete_milestone", + "gitlab.delete_milestone", + "linear.delete_cycle", + "slack.delete_bookmark", + "google_calendar.gcal_create_event", + "linear.delete_label", + "slack.delete_usergroup" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 318, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 651, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_projects", + "vercel.vercel_list_deployments", + "vercel.vercel_create_project", + "jira.jira_list_projects", + "sentry.sentry_list_projects", + "gitlab.gitlab_list_projects", + "vercel.vercel_get_deployment", + "linear.linear_list_projects", + "asana.asana_list_tasks", + "supabase.supabase_list_tables" + ], + "hit1": "vercel.vercel_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 175, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 260, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 524, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "ranking": [ + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update", + "supabase.supabase_list_tables", + "supabase.supabase_invoke_fn", + "sqlite.sqlite_query", + "salesforce.salesforce_query", + "mysql.mysql_query", + "postgres.query", + "okta.okta_list_users" + ], + "hit1": "supabase.supabase_select", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 623, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "kind": "verb_product", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "figma.figma_get_comments", + "figma.figma_get_file", + "linear.get_comment", + "linear.create_comment", + "notion.get_comment", + "linear.linear_create_comment", + "linear.update_comment", + "notion.create_comment", + "notion.update_comment" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "kind": "verb_product", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "ranking": [ + "okta.okta_list_users", + "okta.okta_get_user", + "okta.okta_create_user", + "notion.notion_list_users", + "okta.okta_deactivate_user", + "notion.list_users_tokens", + "github.list_repository_collaborators", + "slack.list_usergroups", + "github.list_starred_repositories", + "discord.discord_list_members" + ], + "hit1": "okta.okta_list_users", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 271, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 613, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query_range", + "prometheus.prom_query", + "prometheus.prom_list_metrics", + "datadog.datadog_query_metrics", + "datadog.datadog_list_monitors", + "gmail.gmail_search_emails", + "memory.search_nodes", + "github.search_pull_requests", + "datadog.datadog_list_logs", + "shopify.shopify_list_products" + ], + "hit1": "prometheus.prom_query_range", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 736, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "kind": "verb_product", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "ranking": [ + "grafana.grafana_create_annotation", + "grafana.grafana_get_dashboard", + "grafana.grafana_search_dashboards", + "slack.create_canvase", + "gitlab.create_snippet", + "github.create_artifact", + "linear.create_comment", + "linear.create_attachment", + "linear.create_label", + "notion.create_comment" + ], + "hit1": "grafana.grafana_create_annotation", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 911, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "kind": "identifier", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "ranking": [ + "terraform.tf_apply", + "terraform.tf_plan", + "terraform.tf_state_list", + "kubernetes.k8s_apply", + "github.update_environment", + "cloudflare.cf_create_dns", + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "playwright.browser_fill_form", + "hubspot.hubspot_create_deal" + ], + "hit1": "terraform.tf_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 653, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "kind": "verb_product", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "ranking": [ + "asana.asana_list_tasks", + "asana.asana_create_task", + "asana.asana_get_task", + "asana.asana_update_task", + "asana.asana_create_project", + "clickup.clickup_list_tasks", + "okta.okta_list_users", + "pagerduty.pagerduty_list_incidents", + "github.actions_list", + "salesforce.salesforce_list_opportunities" + ], + "hit1": "asana.asana_list_tasks", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "ranking": [ + "trello.trello_add_comment", + "trello.trello_create_card", + "trello.trello_list_cards", + "linear.get_comment", + "trello.trello_create_board", + "linear.update_comment", + "notion.get_comment", + "figma.figma_post_comment", + "linear.create_comment", + "linear.list_comments" + ], + "hit1": "trello.trello_add_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_update_task", + "clickup.clickup_list_tasks", + "clickup.clickup_get_task", + "clickup.clickup_create_task", + "puppeteer.puppeteer_click", + "gitlab.update_snippet", + "slack.update_bookmark", + "asana.asana_update_task", + "shopify.shopify_update_product", + "playwright.browser_click" + ], + "hit1": "clickup.clickup_update_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 731, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "kind": "verb_product", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "ranking": [ + "confluence.confluence_search", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "confluence.confluence_create_page", + "grafana.grafana_search_dashboards", + "github.search_orgs", + "github.search_pull_requests", + "hubspot.hubspot_search_contacts", + "github.search_users", + "notion.notion_search" + ], + "hit1": "confluence.confluence_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 652, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_deal", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_create_contact", + "hubspot.hubspot_search_contacts", + "hubspot.hubspot_list_contacts", + "slack.create_usergroup", + "slack.get_usergroup", + "stripe.stripe_create_refund", + "slack.update_usergroup", + "slack.slack_join_channel" + ], + "hit1": "hubspot.hubspot_create_deal", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 715, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_search", + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_update_ticket", + "google_calendar.gcal_list_calendars", + "jira.list_boards", + "salesforce.salesforce_list_opportunities", + "pagerduty.pagerduty_list_oncalls", + "jira.list_webhooks" + ], + "hit1": "zendesk.zendesk_list_tickets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 674, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "kind": "verb_product", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "ranking": [ + "discord.discord_list_channels", + "discord.discord_get_channel", + "slack.slack_list_channels", + "discord.discord_list_members", + "discord.discord_send_message", + "slack.list_usergroups", + "slack.slack_get_channel_history", + "github.list_notifications", + "twilio.twilio_list_messages", + "slack.slack_create_channel" + ], + "hit1": "discord.discord_list_channels", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 642, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "ranking": [ + "redis.redis_set", + "redis.redis_get", + "redis.redis_delete", + "redis.redis_keys", + "playwright.browser_press_key", + "slack.update_pin", + "clickup.clickup_update_task", + "slack.update_reminder", + "linear.update_user", + "gitlab.update_pipeline" + ], + "hit1": "redis.redis_set", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert", + "mongodb.mongo_update", + "supabase.supabase_insert", + "mongodb.mongo_list_collections", + "mongodb.mongo_find", + "mongodb.mongo_delete", + "aws_s3.s3_put_object", + "salesforce.salesforce_create_record", + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 374, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "kind": "verb_product", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "ranking": [ + "mysql.mysql_query", + "sqlite.sqlite_query", + "postgres.query", + "salesforce.salesforce_query", + "postgres.explain_query", + "supabase.supabase_select", + "mysql.mysql_list_tables", + "fetch.fetch", + "prometheus.prom_query", + "zendesk.zendesk_search" + ], + "hit1": "mysql.mysql_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 595, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "elasticsearch.es_search", + "mongodb.mongo_find", + "confluence.confluence_search", + "google_drive.gdrive_search", + "memory.search_nodes", + "filesystem.search_files", + "mongodb.mongo_insert", + "linear.linear_search_issues" + ], + "hit1": "elasticsearch.es_index_doc", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 372, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 681, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_get_object", + "aws_s3.s3_put_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_list_objects", + "aws_s3.s3_list_buckets", + "gitlab.get_snippet", + "linear.get_attachment", + "salesforce.salesforce_get_record", + "redis.redis_get", + "jira.get_attachment" + ], + "hit1": "aws_s3.s3_get_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones", + "cloudflare.cf_list_dns", + "cloudflare.cf_create_dns", + "cloudflare.cf_purge_cache", + "aws_s3.s3_list_buckets", + "kubernetes.k8s_list_services", + "kubernetes.k8s_list_deployments", + "circleci.circleci_list_pipelines", + "terraform.tf_state_list", + "docker.docker_images" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 326, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 570, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_customer", + "stripe.stripe_create_refund", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "stripe.stripe_get_payment_intent", + "stripe.stripe_get_customer", + "stripe.stripe_list_subscriptions" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "kind": "verb_product", + "gold": [ + "github.actions_list" + ], + "alts": [], + "ranking": [ + "github.actions_list", + "github.actions_get", + "github.actions_run_trigger", + "github.actions_get_job_logs", + "github.list_notifications", + "github.list_milestones", + "github.list_hooks", + "github.list_code_scanning_alerts", + "github.list_artifacts", + "github.list_check_runs" + ], + "hit1": "github.actions_list", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 452, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 859, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "kind": "verb_product", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "ranking": [ + "github.actions_get_job_logs", + "github.actions_get", + "github.actions_list", + "github.get_check_run", + "github.actions_run_trigger", + "github.get_notification_details", + "git.git_log", + "github.search_pull_requests", + "github.get_code_scanning_alert", + "github.list_code_scanning_alerts" + ], + "hit1": "github.actions_get_job_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 442, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 805, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "kind": "verb_product", + "gold": [ + "github.create_repository" + ], + "alts": [], + "ranking": [ + "github.create_repository", + "github.create_gist", + "github.create_environment", + "github.create_branch", + "github.create_or_update_file", + "github.create_package", + "github.create_pull_request", + "github.create_artifact", + "github.create_issue", + "github.create_hook" + ], + "hit1": "github.create_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 274, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 519, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1091, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "kind": "verb_product", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "ranking": [ + "git.git_checkout", + "github.get_check_run", + "git.git_branch", + "git.git_status", + "git.git_show", + "github.list_branches", + "github.pull_request_read", + "github.get_gist", + "git.git_create_branch", + "github.create_branch" + ], + "hit1": "git.git_checkout", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 693, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "kind": "identifier", + "gold": [ + "git.git_log" + ], + "alts": [], + "ranking": [ + "git.git_log", + "git.git_show", + "git.git_status", + "github.actions_get_job_logs", + "git.git_diff", + "git.git_diff_unstaged", + "git.git_commit", + "github.get_check_run", + "git.git_branch", + "github.list_code_scanning_alerts" + ], + "hit1": "git.git_log", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 668, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "kind": "verb_product", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "ranking": [ + "postgres.list_schemas", + "postgres.list_tables", + "mysql.mysql_list_tables", + "postgres.describe_table", + "vercel.vercel_list_projects", + "supabase.supabase_list_tables", + "vercel.vercel_list_deployments", + "mongodb.mongo_list_collections", + "sentry.sentry_list_projects", + "aws_s3.s3_list_buckets" + ], + "hit1": "postgres.list_schemas", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 254, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 493, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "kind": "verb_product", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "ranking": [ + "memory.open_nodes", + "memory.search_nodes", + "memory.read_graph", + "memory.create_relations", + "memory.create_entities", + "memory.add_observations", + "memory.delete_entities", + "memory.delete_relations", + "notion.notion_search", + "sequential_thinking.sequentialthinking" + ], + "hit1": "memory.open_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 599, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "kind": "paraphrase", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "ranking": [ + "filesystem.get_file_info", + "google_drive.gdrive_get_file", + "filesystem.read_file", + "filesystem.read_media_file", + "filesystem.list_directory", + "filesystem.read_text_file", + "filesystem.directory_tree", + "filesystem.list_directory_with_sizes", + "figma.figma_get_file_nodes", + "github.get_file_contents" + ], + "hit1": "filesystem.get_file_info", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 685, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "kind": "paraphrase", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "ranking": [ + "filesystem.write_file", + "filesystem.edit_file", + "slack.create_file", + "filesystem.read_text_file", + "filesystem.create_directory", + "google_drive.gdrive_create_file", + "filesystem.read_file", + "filesystem.read_media_file", + "filesystem.move_file", + "github.create_or_update_file" + ], + "hit1": "filesystem.write_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 280, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 421, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 850, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "kind": "verb_product", + "gold": [ + "github.search_issues" + ], + "alts": [], + "ranking": [ + "github.search_issues", + "github.search_users", + "github.search_orgs", + "github.search_pull_requests", + "github.search_commits", + "github.search_repositories", + "github.search_code", + "github.list_issues", + "gitlab.gitlab_list_projects", + "github.list_code_scanning_alerts" + ], + "hit1": "github.search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "kind": "paraphrase", + "gold": [ + "github.get_me" + ], + "alts": [], + "ranking": [ + "github.get_me", + "github.get_teams", + "github.list_notifications", + "github.get_secret_scanning_alert", + "github.projects_get", + "github.list_gists", + "github.get_code_scanning_alert", + "github.get_latest_release", + "github.get_notification_details", + "slack.slack_get_user_profile" + ], + "hit1": "github.get_me", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 670, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "cost": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.08, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 253.48, + "p95_tokens": 315.65, + "total_tokens": 38022, + "tokens_per_successful_query": 253.48, + "total_tokens_successful": 38022 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.0466666666666666, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 393.7, + "p95_tokens": 507.2999999999996, + "total_tokens": 59055, + "tokens_per_successful_query": 393.7, + "total_tokens_successful": 59055 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.02, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 751.1466666666666, + "p95_tokens": 979.1499999999992, + "total_tokens": 112672, + "tokens_per_successful_query": 751.1466666666666, + "total_tokens_successful": 112672 + } + } +} diff --git a/experiments/mcp-tool-search/results/hashed_ngram_embed.csv b/experiments/mcp-tool-search/results/hashed_ngram_embed.csv new file mode 100644 index 00000000..b5f7d80b --- /dev/null +++ b/experiments/mcp-tool-search/results/hashed_ngram_embed.csv @@ -0,0 +1,151 @@ +id,query,kind,hit1,recall@1,recall@3,recall@5,mrr@10,ndcg@10,hit_rank,k3_calls,k3_tokens,k3_success1,k5_calls,k5_tokens,k5_success1,k10_calls,k10_tokens,k10_success1,ranking +q001,Create a GitHub issue,verb_product,github.issue_write,1.0,1.0,1.0,1.0000,0.6509,1,1,334,1.0,1,509,1.0,1,1047,1.0,github.issue_write github.issue_read github.create_gist github.projects_write gitlab.gitlab_create_issue github.create_issue github.update_issue_comment github.label_write github.add_issue_comment github.update_gist +q002,Create a Jira ticket,verb_product,jira.jira_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,345,1.0,1,535,1.0,1,940,1.0,jira.jira_create_issue jira.jira_create_sprint jira.create_webhook jira.create_filter jira.create_board jira.create_sprint jira.jira_get_issue jira.create_attachment clickup.clickup_create_task jira.jira_assign_issue +q003,Send a Slack message,verb_product,slack.slack_post_message,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,394,1.0,1,709,1.0,slack.slack_post_message slack.slack_search_messages slack.slack_add_reaction slack.slack_get_channel_history slack.slack_create_channel discord.discord_send_message slack.slack_join_channel slack.slack_leave_channel slack.update_pin slack.slack_get_thread_replies +q004,List open pull requests,resource,github.create_pull_request,0.0,1.0,1.0,0.5000,0.6267,2,1,363,1.0,1,517,1.0,1,1044,1.0,github.create_pull_request github.list_pull_requests github.merge_pull_request github.request_copilot_review github.search_pull_requests github.pull_request_read github.pull_request_review_write gitlab.gitlab_get_merge_request gitlab.gitlab_list_merge_requests github.update_pull_request +q005,Open a pull request on GitHub,verb_product,github.create_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,327,1.0,1,520,1.0,1,1144,1.0,github.create_pull_request github.request_copilot_review github.search_pull_requests github.merge_pull_request github.list_pull_requests github.pull_request_read github.add_comment_to_pending_review github.update_pull_request github.pull_request_review_write github.update_pull_request_branch +q006,Run a SQL query against postgres,verb_product,postgres.query,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,295,1.0,1,575,1.0,postgres.query mysql.mysql_query postgres.explain_query sqlite.sqlite_query salesforce.salesforce_query slack.slack_post_message postgres.list_schemas memory.search_nodes postgres.describe_table prometheus.prom_query +q007,Read a local file from disk,paraphrase,filesystem.read_media_file,0.0,1.0,1.0,0.5000,0.6697,2,1,223,1.0,1,389,1.0,1,766,1.0,filesystem.read_media_file filesystem.read_file filesystem.read_text_file filesystem.edit_file playwright.browser_take_screenshot github.get_file_contents gitlab.gitlab_get_file filesystem.write_file slack.get_file google_drive.gdrive_get_file +q008,What files are in this folder,paraphrase,google_drive.gdrive_list_folder,0.0,0.0,1.0,0.2000,0.4434,5,2,464,0.0,1,388,1.0,1,715,1.0,google_drive.gdrive_list_folder github.push_files filesystem.move_file slack.list_files filesystem.list_directory_with_sizes filesystem.list_directory slack.delete_file github.get_file_blame filesystem.search_files filesystem.directory_tree +q009,Remember this fact in the knowledge graph,paraphrase,memory.read_graph,0.0,0.0,1.0,0.2500,0.4628,4,2,343,0.0,1,272,1.0,1,554,1.0,memory.read_graph memory.open_nodes memory.search_nodes memory.add_observations memory.delete_entities memory.create_entities memory.delete_relations gitlab.list_releases memory.create_relations github.get_me +q010,Search the web with Brave,verb_product,brave_search.brave_web_search,1.0,1.0,1.0,1.0000,1.0000,1,1,236,1.0,1,359,1.0,1,684,1.0,brave_search.brave_web_search brave_search.brave_local_search github.search_users elasticsearch.es_search memory.search_nodes github.search_orgs jira.jira_search_issues github.search_commits github.search_repositories hubspot.hubspot_search_contacts +q011,Fetch the contents of a URL,verb_product,fetch.fetch,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,336,1.0,1,705,1.0,fetch.fetch filesystem.read_file postgres.describe_table stripe.stripe_get_payment_intent memory.search_nodes stripe.stripe_create_payment_intent playwright.browser_take_screenshot playwright.browser_navigate memory.open_nodes github.get_file_contents +q012,Show git status,identifier,git.git_show,0.0,1.0,1.0,0.5000,0.6309,2,1,196,1.0,1,324,1.0,1,637,1.0,git.git_show git.git_status git.git_diff github.list_runners git.git_diff_staged git.git_diff_unstaged git.git_log git.git_branch github.list_packages github.list_check_runs +q013,Commit staged changes,paraphrase,git.git_reset,0.0,1.0,1.0,0.5000,0.6309,2,1,182,1.0,1,320,1.0,1,754,1.0,git.git_reset git.git_commit git.git_diff_staged github.get_commit git.git_diff_unstaged filesystem.read_media_file filesystem.read_text_file github.delete_file git.git_show github.push_files +q014,Create a Linear issue,verb_product,linear.linear_create_comment,0.0,1.0,1.0,0.5000,0.6309,2,1,333,1.0,1,471,1.0,1,895,1.0,linear.linear_create_comment linear.linear_create_issue linear.linear_update_issue linear.linear_get_issue linear.linear_create_project linear.linear_list_issues jira.jira_create_issue linear.linear_assign_issue linear.linear_search_issues linear.create_label +q015,Make a task in my project tracker Linear,paraphrase,linear.linear_create_project,0.0,0.0,0.0,0.1667,0.3562,6,2,580,0.0,2,849,0.0,1,825,1.0,linear.linear_create_project linear.linear_list_projects jira.jira_create_issue linear.list_users github.projects_write linear.linear_create_issue linear.list_cycles github.projects_get linear.create_user jira.jira_list_projects +q016,Post a comment on a GitHub issue,verb_product,github.update_issue_comment,0.0,1.0,1.0,0.5000,0.6309,2,1,337,1.0,1,509,1.0,1,1018,1.0,github.update_issue_comment github.add_issue_comment github.list_commits figma.figma_post_comment github.issue_read github.list_issues github.get_gist github.issue_write github.add_comment_to_pending_review linear.update_comment +q017,Merge the pull request,ambiguous,github.merge_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,283,1.0,1,474,1.0,1,933,1.0,github.merge_pull_request gitlab.gitlab_merge_merge_request gitlab.gitlab_get_merge_request github.request_copilot_review github.pull_request_read github.search_pull_requests github.create_pull_request github.update_pull_request_branch github.list_pull_requests gitlab.gitlab_list_merge_requests +q018,Trigger a GitHub Actions workflow,verb_product,github.actions_run_trigger,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,491,1.0,1,862,1.0,github.actions_run_trigger github.actions_get github.actions_list github.actions_get_job_logs github.list_commits github.list_branches github.list_discussions github.list_tags github.list_pull_requests github.list_labels +q019,Get Dependabot alerts,resource,github.get_dependabot_alert,1.0,1.0,1.0,1.0000,0.8597,1,1,286,1.0,1,455,1.0,1,773,1.0,github.get_dependabot_alert github.list_dependabot_alerts github.get_code_scanning_alert github.get_secret_scanning_alert google_calendar.gcal_delete_event gitlab.gitlab_get_issue jira.jira_get_issue linear.linear_get_issue google_calendar.gcal_list_calendars github.issue_read +q020,List Sentry issues,verb_product,sentry.sentry_list_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,396,1.0,1,845,1.0,sentry.sentry_list_issues sentry.sentry_list_events sentry.sentry_get_issue sentry.sentry_update_issue sentry.sentry_list_projects gitlab.gitlab_list_issues github.list_issues gitlab.gitlab_get_issue jira.jira_update_issue github.issue_write +q021,Create a Stripe customer,verb_product,stripe.stripe_create_customer,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,353,1.0,1,669,1.0,stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_create_invoice stripe.stripe_create_payment_intent stripe.stripe_create_price stripe.stripe_list_customers stripe.stripe_create_refund stripe.stripe_create_product stripe.stripe_list_invoices stripe.stripe_list_subscriptions +q022,Refund a Stripe payment,paraphrase,stripe.stripe_create_refund,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,350,1.0,1,662,1.0,stripe.stripe_create_refund stripe.stripe_get_payment_intent stripe.stripe_create_payment_intent stripe.stripe_create_customer stripe.stripe_create_invoice stripe.stripe_create_price stripe.stripe_get_customer stripe.stripe_create_product stripe.stripe_list_invoices stripe.stripe_list_customers +q023,Send an email with Gmail,verb_product,gmail.gmail_send_email,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,328,1.0,1,733,1.0,gmail.gmail_send_email gmail.gmail_search_emails gmail.gmail_list_labels gmail.gmail_get_email gmail.gmail_trash_email gmail.gmail_modify_email gmail.gmail_create_draft google_drive.gdrive_share_file filesystem.read_text_file filesystem.read_media_file +q024,Search my Gmail inbox,verb_product,gmail.gmail_search_emails,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,373,1.0,1,726,1.0,gmail.gmail_search_emails elasticsearch.es_search gmail.gmail_send_email notion.notion_search gmail.gmail_get_email gmail.gmail_trash_email brave_search.brave_web_search gmail.gmail_create_draft gmail.gmail_modify_email gmail.gmail_list_labels +q025,Upload a file to Google Drive,paraphrase,google_drive.gdrive_create_file,1.0,1.0,1.0,1.0000,1.0000,1,1,272,1.0,1,418,1.0,1,826,1.0,google_drive.gdrive_create_file google_drive.gdrive_get_file google_drive.gdrive_share_file google_drive.gdrive_update_file google_drive.gdrive_search google_drive.gdrive_list_folder filesystem.edit_file slack.slack_upload_file filesystem.read_text_file gitlab.gitlab_get_file +q026,Create a calendar event,resource,google_calendar.gcal_create_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,427,1.0,1,837,1.0,google_calendar.gcal_create_event google_calendar.gcal_update_event google_calendar.gcal_delete_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars github.projects_write notion.notion_create_page confluence.confluence_create_page trello.trello_create_card trello.trello_create_board +q027,List Kubernetes pods,verb_product,kubernetes.k8s_list_pods,1.0,1.0,1.0,1.0000,1.0000,1,1,206,1.0,1,314,1.0,1,749,1.0,kubernetes.k8s_list_pods kubernetes.k8s_list_services kubernetes.k8s_get_pod kubernetes.k8s_get_events kubernetes.k8s_list_deployments kubernetes.k8s_pod_logs kubernetes.k8s_apply kubernetes.k8s_delete kubernetes.k8s_scale_deployment slack.list_canvases +q028,Get logs from a k8s pod,verb_product,kubernetes.k8s_pod_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,397,1.0,1,783,1.0,kubernetes.k8s_pod_logs github.get_teams git.git_create_branch docker.docker_logs gitlab.gitlab_get_file gitlab.get_snippet gitlab.create_snippet gitlab.update_snippet github.create_branch gitlab.delete_snippet +q029,Run a docker container,verb_product,docker.docker_stop,0.0,0.0,1.0,0.2500,0.4307,4,2,411,0.0,1,345,1.0,1,600,1.0,docker.docker_stop docker.docker_exec docker.docker_rm docker.docker_run docker.docker_logs docker.docker_ps docker.docker_images docker.docker_build mysql.mysql_query hubspot.hubspot_get_contact +q030,Show docker container logs,verb_product,docker.docker_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,298,1.0,1,609,1.0,docker.docker_logs docker.docker_rm docker.docker_stop docker.docker_exec docker.docker_ps docker.docker_run git.git_log docker.docker_images docker.docker_build hubspot.hubspot_get_contact +q031,Navigate the browser to a URL,ambiguous,playwright.browser_navigate,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,265,1.0,1,658,1.0,playwright.browser_navigate puppeteer.puppeteer_navigate playwright.browser_tab_new playwright.browser_tab_close playwright.browser_tab_list playwright.browser_wait_for playwright.browser_evaluate github.create_runner notion.create_users_token playwright.browser_type +q032,Take a screenshot of the page,ambiguous,playwright.browser_take_screenshot,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,408,1.0,1,707,1.0,playwright.browser_take_screenshot puppeteer.puppeteer_screenshot notion.notion_update_page pagerduty.pagerduty_list_oncalls pagerduty.pagerduty_create_incident playwright.browser_snapshot filesystem.read_file filesystem.read_media_file linear.update_cycle github.update_gist +q033,Create a GitLab merge request,verb_product,gitlab.gitlab_create_merge_request,1.0,1.0,1.0,1.0000,1.0000,1,1,297,1.0,1,475,1.0,1,883,1.0,gitlab.gitlab_create_merge_request gitlab.gitlab_merge_merge_request gitlab.gitlab_get_merge_request gitlab.gitlab_list_merge_requests github.merge_pull_request gitlab.gitlab_create_issue gitlab.gitlab_create_pipeline github.request_copilot_review gitlab.gitlab_get_issue gitlab.gitlab_list_issues +q034,Search Notion pages,verb_product,notion.notion_search,1.0,1.0,1.0,1.0000,1.0000,1,1,229,1.0,1,364,1.0,1,708,1.0,notion.notion_search notion.notion_get_page notion.notion_create_page notion.notion_update_page elasticsearch.es_search notion.notion_list_users brave_search.brave_web_search notion.notion_get_comments notion.notion_create_database brave_search.brave_local_search +q035,Create a Notion page,verb_product,notion.notion_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,230,1.0,1,419,1.0,1,815,1.0,notion.notion_create_page notion.notion_get_page notion.notion_update_page notion.notion_create_database notion.create_data_source notion.create_comment notion.notion_search github.create_repository notion.notion_get_comments github.create_package +q036,Query a Notion database,verb_product,notion.notion_query_database,1.0,1.0,1.0,1.0000,1.0000,1,1,261,1.0,1,373,1.0,1,691,1.0,notion.notion_query_database notion.notion_create_database notion.notion_search notion.delete_data_source notion.get_data_source notion.notion_get_block notion.create_data_source notion.notion_get_page notion.notion_list_users notion.update_data_source +q037,Create a Zendesk ticket,verb_product,zendesk.zendesk_get_ticket,0.0,1.0,1.0,0.5000,0.6309,2,1,257,1.0,1,368,1.0,1,881,1.0,zendesk.zendesk_get_ticket zendesk.zendesk_create_ticket zendesk.zendesk_update_ticket zendesk.zendesk_list_tickets zendesk.zendesk_search jira.create_sprint jira.jira_create_sprint clickup.clickup_create_task notion.create_comment jira.jira_create_issue +q038,Create a HubSpot contact,verb_product,hubspot.hubspot_create_contact,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,364,1.0,1,805,1.0,hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_search_contacts hubspot.hubspot_list_contacts hubspot.hubspot_create_deal confluence.confluence_create_page github.create_gist gmail.gmail_create_draft github.create_artifact clickup.clickup_create_task +q039,Create a Salesforce lead,verb_product,salesforce.salesforce_create_lead,1.0,1.0,1.0,1.0000,1.0000,1,1,252,1.0,1,416,1.0,1,766,1.0,salesforce.salesforce_create_lead salesforce.salesforce_create_record salesforce.salesforce_convert_lead salesforce.salesforce_update_record salesforce.salesforce_delete_record salesforce.salesforce_get_record salesforce.salesforce_query salesforce.salesforce_list_opportunities pagerduty.pagerduty_create_incident vercel.vercel_create_project +q040,Run a SOQL query,identifier,salesforce.salesforce_query,1.0,1.0,1.0,1.0000,1.0000,1,1,187,1.0,1,296,1.0,1,600,1.0,salesforce.salesforce_query postgres.query mysql.mysql_query memory.search_nodes sqlite.sqlite_query prometheus.prom_query linear.update_label postgres.explain_query discord.discord_send_message linear.update_cycle +q041,List Shopify orders,verb_product,shopify.shopify_list_orders,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,333,1.0,1,691,1.0,shopify.shopify_list_orders shopify.shopify_get_order shopify.shopify_list_products shopify.shopify_create_order shopify.shopify_list_customers shopify.shopify_get_product shopify.shopify_update_product shopify.shopify_create_product github.projects_list github.list_discussions +q042,Create a PagerDuty incident,verb_product,pagerduty.pagerduty_create_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,403,1.0,1,818,1.0,pagerduty.pagerduty_create_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls notion.notion_create_page github.create_gist confluence.confluence_create_page slack.create_pin +q043,Who is on call in PagerDuty,paraphrase,pagerduty.pagerduty_acknowledge_incident,0.0,1.0,1.0,0.3333,0.5000,3,1,219,1.0,1,389,1.0,1,749,1.0,pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_list_oncalls pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_list_incidents notion.notion_create_page memory.open_nodes twilio.twilio_make_call memory.search_nodes +q044,Query Datadog metrics,verb_product,datadog.datadog_query_metrics,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,458,1.0,1,753,1.0,datadog.datadog_query_metrics datadog.datadog_create_monitor datadog.datadog_list_monitors datadog.datadog_list_logs datadog.datadog_get_monitor notion.notion_query_database prometheus.prom_query mysql.mysql_query filesystem.get_file_info prometheus.prom_list_metrics +q045,Search Datadog logs,verb_product,datadog.datadog_list_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,422,1.0,1,736,1.0,datadog.datadog_list_logs elasticsearch.es_search notion.notion_search github.search_orgs datadog.datadog_create_monitor github.search_users datadog.datadog_get_monitor brave_search.brave_web_search datadog.datadog_list_monitors github.search_commits +q046,List S3 buckets,verb_product,aws_s3.s3_list_buckets,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,316,1.0,1,613,1.0,aws_s3.s3_list_buckets aws_s3.s3_list_objects bitbucket.bitbucket_list_repos bitbucket.bitbucket_list_prs zendesk.zendesk_list_tickets terraform.tf_state_list vercel.vercel_list_deployments vercel.vercel_list_projects gitlab.list_snippets github.list_packages +q047,Upload an object to S3,paraphrase,aws_s3.s3_put_object,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,380,1.0,1,733,1.0,aws_s3.s3_put_object aws_s3.s3_get_object aws_s3.s3_delete_object aws_s3.s3_list_objects vercel.vercel_create_project vercel.vercel_list_projects github.projects_write twilio.twilio_send_sms gmail.gmail_send_email gitlab.gitlab_list_projects +q048,Purge Cloudflare cache,verb_product,cloudflare.cf_purge_cache,1.0,1.0,1.0,1.0000,1.0000,1,1,231,1.0,1,338,1.0,1,732,1.0,cloudflare.cf_purge_cache cloudflare.cf_list_zones cloudflare.cf_create_dns cloudflare.cf_list_dns github.search_orgs gmail.gmail_search_emails slack.list_reminders gitlab.get_release gitlab.gitlab_create_merge_request gitlab.gitlab_list_merge_requests +q049,Create a Cloudflare DNS record,verb_product,cloudflare.cf_create_dns,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,378,1.0,1,782,1.0,cloudflare.cf_create_dns cloudflare.cf_list_dns cloudflare.cf_list_zones cloudflare.cf_purge_cache trello.trello_create_card trello.trello_create_board github.create_repository notion.create_data_source salesforce.salesforce_create_record jira.create_attachment +q050,List Vercel deployments,verb_product,vercel.vercel_list_deployments,1.0,1.0,1.0,1.0000,1.0000,1,1,195,1.0,1,291,1.0,1,637,1.0,vercel.vercel_list_deployments vercel.vercel_get_deployment kubernetes.k8s_list_deployments vercel.vercel_list_projects vercel.vercel_create_project kubernetes.k8s_scale_deployment github.list_environments okta.okta_list_users asana.asana_list_tasks clickup.clickup_list_tasks +q051,Insert a row into Supabase,verb_product,supabase.supabase_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,355,1.0,1,741,1.0,supabase.supabase_insert supabase.supabase_update supabase.supabase_select supabase.supabase_invoke_fn supabase.supabase_list_tables mongodb.mongo_insert jira.jira_create_issue jira.get_version jira.create_version playwright.browser_navigate +q052,Export images from Figma,paraphrase,figma.figma_get_images,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,322,1.0,1,667,1.0,figma.figma_get_images docker.docker_images figma.figma_get_file_nodes memory.delete_entities salesforce.salesforce_list_opportunities gitlab.gitlab_get_file figma.figma_get_comments github.list_secret_scanning_alerts figma.figma_get_file git.git_commit +q053,Send an SMS with Twilio,verb_product,twilio.twilio_send_sms,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,407,1.0,1,848,1.0,twilio.twilio_send_sms twilio.twilio_list_messages twilio.twilio_make_call docker.docker_build gmail.gmail_send_email github.update_pull_request_branch okta.okta_deactivate_user git.git_checkout google_calendar.gcal_create_event google_drive.gdrive_create_file +q054,Create an Okta user,verb_product,okta.okta_create_user,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,344,1.0,1,744,1.0,okta.okta_create_user okta.okta_deactivate_user okta.okta_get_user vercel.vercel_create_project slack.create_usergroup asana.asana_create_project okta.okta_list_users linear.create_user notion.create_users_token stripe.stripe_create_price +q055,Instant query Prometheus,verb_product,prometheus.prom_query,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,329,1.0,1,708,1.0,prometheus.prom_query prometheus.prom_query_range prometheus.prom_list_metrics postgres.explain_query postgres.query datadog.datadog_query_metrics shopify.shopify_list_products sentry.sentry_list_issues sqlite.sqlite_query salesforce.salesforce_query +q056,Search Grafana dashboards,verb_product,grafana.grafana_search_dashboards,1.0,1.0,1.0,1.0000,1.0000,1,1,198,1.0,1,352,1.0,1,677,1.0,grafana.grafana_search_dashboards grafana.grafana_get_dashboard elasticsearch.es_search grafana.grafana_create_annotation notion.notion_search brave_search.brave_local_search memory.search_nodes brave_search.brave_web_search github.search_orgs gitlab.gitlab_list_projects +q057,Trigger a CircleCI pipeline,verb_product,circleci.circleci_trigger_pipeline,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,325,1.0,1,676,1.0,circleci.circleci_trigger_pipeline circleci.circleci_get_pipeline circleci.circleci_list_pipelines gitlab.delete_pipeline gitlab.gitlab_create_pipeline gitlab.update_pipeline gitlab.get_pipeline gitlab.create_pipeline gitlab.list_pipelines linear.delete_cycle +q058,Run terraform plan,identifier,terraform.tf_plan,1.0,1.0,1.0,1.0000,1.0000,1,1,172,1.0,1,293,1.0,1,582,1.0,terraform.tf_plan terraform.tf_apply terraform.tf_state_list playwright.browser_fill_form github.list_releases playwright.browser_click datadog.datadog_get_monitor postgres.explain_query memory.delete_entities salesforce.salesforce_query +q059,Create an Asana task,verb_product,asana.asana_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,405,1.0,1,872,1.0,asana.asana_create_task asana.asana_get_task asana.asana_update_task asana.asana_create_project asana.asana_list_tasks grafana.grafana_create_annotation clickup.clickup_create_task hubspot.hubspot_create_deal notion.create_users_token slack.create_pin +q060,Create a Trello card,verb_product,trello.trello_create_card,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,411,1.0,1,895,1.0,trello.trello_create_card trello.trello_create_board trello.trello_add_comment trello.trello_list_cards jira.create_board github.create_check_run github.create_gist slack.create_canvase slack.create_reminder linear.create_user +q061,Create a ClickUp task,verb_product,clickup.clickup_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,407,1.0,1,858,1.0,clickup.clickup_create_task clickup.clickup_get_task clickup.clickup_update_task clickup.clickup_list_tasks notion.create_comment github.create_artifact trello.trello_create_board jira.create_attachment github.create_check_run gitlab.create_release +q062,Create a Confluence page,verb_product,confluence.confluence_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,409,1.0,1,943,1.0,confluence.confluence_create_page confluence.confluence_update_page confluence.confluence_get_page confluence.confluence_search jira.create_board jira.create_attachment pagerduty.pagerduty_create_incident github.create_or_update_file jira.create_version jira.create_filter +q063,Send a Discord message,verb_product,discord.discord_send_message,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,322,1.0,1,692,1.0,discord.discord_send_message discord.discord_add_reaction discord.discord_get_channel discord.discord_list_members discord.discord_list_channels slack.slack_search_messages git.git_commit slack.slack_post_message twilio.twilio_list_messages datadog.datadog_create_monitor +q064,Search Elasticsearch,verb_product,elasticsearch.es_search,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,404,1.0,1,696,1.0,elasticsearch.es_search elasticsearch.es_get_doc elasticsearch.es_index_doc brave_search.brave_web_search brave_search.brave_local_search github.search_users memory.search_nodes grafana.grafana_search_dashboards notion.notion_search zendesk.zendesk_search +q065,Get a Redis key,verb_product,redis.redis_get,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,312,1.0,1,622,1.0,redis.redis_get redis.redis_set redis.redis_delete redis.redis_keys github.get_label github.get_teams github.search_commits github.get_gist github.label_write linear.get_attachment +q066,Find MongoDB documents,verb_product,mongodb.mongo_insert,0.0,1.0,1.0,0.5000,0.6309,2,1,244,1.0,1,381,1.0,1,788,1.0,mongodb.mongo_insert mongodb.mongo_find mongodb.mongo_delete mongodb.mongo_update mongodb.mongo_list_collections elasticsearch.es_index_doc github.list_environments zendesk.zendesk_create_ticket elasticsearch.es_get_doc stripe.stripe_create_payment_intent +q067,List SQLite tables,verb_product,sqlite.sqlite_list_tables,1.0,1.0,1.0,1.0000,1.0000,1,1,156,1.0,1,243,1.0,1,591,1.0,sqlite.sqlite_list_tables sqlite.sqlite_describe_table sqlite.sqlite_query mysql.mysql_list_tables supabase.supabase_list_tables supabase.supabase_select postgres.list_tables github.list_milestones gitlab.list_milestones supabase.supabase_insert +q068,Describe a postgres table,verb_product,postgres.describe_table,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,290,1.0,1,705,1.0,postgres.describe_table postgres.query sqlite.sqlite_describe_table postgres.list_tables postgres.explain_query postgres.list_schemas hubspot.hubspot_create_deal confluence.confluence_update_page github.create_check_run jira.create_board +q069,github_create_issue,identifier,gitlab.gitlab_create_issue,0.0,1.0,1.0,0.5000,0.5995,2,1,347,1.0,1,567,1.0,1,1062,1.0,gitlab.gitlab_create_issue github.create_issue github.update_issue_comment github.add_issue_comment jira.jira_create_issue github.search_issues github.create_gist github.issue_write github.assign_copilot_to_issue bitbucket.bitbucket_create_issue +q070,createIssue,identifier,gitlab.gitlab_create_issue,0.0,1.0,1.0,0.5000,0.2015,2,1,319,1.0,1,501,1.0,1,917,1.0,gitlab.gitlab_create_issue jira.jira_create_issue jira.create_version clickup.clickup_create_task bitbucket.bitbucket_create_issue gitlab.gitlab_get_issue slack.slack_create_channel gitlab.create_label github.create_gist jira.create_board +q071,list-directory,identifier,filesystem.create_directory,0.0,1.0,1.0,0.5000,0.6309,2,1,186,1.0,1,281,1.0,1,621,1.0,filesystem.create_directory filesystem.list_directory filesystem.list_directory_with_sizes filesystem.list_allowed_directories filesystem.directory_tree github.get_file_contents filesystem.move_file github.list_repository_collaborators linear.linear_list_projects terraform.tf_state_list +q072,sequential thinking next thought,paraphrase,sequential_thinking.sequentialthinking,1.0,1.0,1.0,1.0000,1.0000,1,1,302,1.0,1,430,1.0,1,746,1.0,sequential_thinking.sequentialthinking filesystem.edit_file kubernetes.k8s_get_pod filesystem.list_allowed_directories github.list_pull_requests stripe.stripe_list_subscriptions playwright.browser_wait_for kubernetes.k8s_get_events notion.list_data_sources git.git_status +q073,What time is it in Tokyo,paraphrase,time.convert_time,1.0,1.0,1.0,1.0000,0.8597,1,1,217,1.0,1,352,1.0,1,725,1.0,time.convert_time time.get_current_time playwright.browser_wait_for slack.list_pins gitlab.delete_snippet slack.get_pin gitlab.get_snippet slack.delete_pin slack.list_bookmarks github.create_issue +q074,Convert time between timezones,verb_product,time.convert_time,1.0,1.0,1.0,1.0000,1.0000,1,1,213,1.0,1,378,1.0,1,779,1.0,time.convert_time time.get_current_time git.git_diff docker.docker_logs notion.create_users_token slack.create_usergroup google_calendar.gcal_list_events slack.slack_add_reaction playwright.browser_wait_for okta.okta_create_user +q075,Star a GitHub repo,verb_product,github.star_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,414,1.0,1,861,1.0,github.star_repository github.unstar_repository github.get_tag github.list_commits github.delete_repository github.update_issue_comment github.list_issues github.list_tags github.fork_repository github.list_pull_requests +q076,Fork a repository,resource,github.fork_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,235,1.0,1,386,1.0,1,824,1.0,github.fork_repository github.delete_repository github.star_repository github.unstar_repository github.create_repository github.get_repository_tree github.list_repository_collaborators github.get_tag github.get_file_blame github.list_releases +q077,Get the latest GitHub release,verb_product,github.get_latest_release,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,401,1.0,1,707,1.0,github.get_latest_release github.list_releases github.get_release_by_tag github.search_repositories github.search_commits github.search_orgs github.search_code github.search_users github.get_label github.search_pull_requests +q078,Create a gist,resource,github.create_gist,1.0,1.0,1.0,1.0000,1.0000,1,1,236,1.0,1,387,1.0,1,824,1.0,github.create_gist vercel.vercel_create_project gmail.gmail_create_draft gitlab.gitlab_create_pipeline github.update_gist slack.create_canvase stripe.stripe_create_refund github.projects_write hubspot.hubspot_create_contact trello.trello_create_card +q079,Search GitHub code,verb_product,github.search_users,0.0,1.0,1.0,0.3333,0.5000,3,1,190,1.0,1,329,1.0,1,757,1.0,github.search_users github.search_orgs github.search_code github.search_issues github.search_commits github.search_repositories github.search_pull_requests brave_search.brave_web_search github.get_code_scanning_alert github.list_code_scanning_alerts +q080,List GitHub notifications,verb_product,github.list_notifications,1.0,1.0,1.0,1.0000,1.0000,1,1,175,1.0,1,280,1.0,1,601,1.0,github.list_notifications github.mark_all_notifications_read github.dismiss_notification github.get_notification_details github.get_gist github.search_orgs github.list_gists github.list_packages github.actions_list github.create_gist +q081,Assign Copilot to an issue,verb_product,github.assign_copilot_to_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,468,1.0,1,852,1.0,github.assign_copilot_to_issue jira.jira_assign_issue jira.jira_update_issue gitlab.gitlab_get_issue github.issue_write jira.jira_transition_issue jira.jira_add_comment jira.jira_add_worklog jira.jira_get_issue github.add_issue_comment +q082,Get secret scanning alerts,resource,github.get_secret_scanning_alert,1.0,1.0,1.0,1.0000,0.8597,1,1,294,1.0,1,440,1.0,1,756,1.0,github.get_secret_scanning_alert github.list_secret_scanning_alerts github.get_code_scanning_alert github.list_code_scanning_alerts memory.create_entities memory.delete_entities zendesk.zendesk_get_ticket docker.docker_exec zendesk.zendesk_list_tickets zendesk.zendesk_create_ticket +q083,Create or update a file on GitHub,verb_product,github.create_or_update_file,1.0,1.0,1.0,1.0000,0.7602,1,1,314,1.0,1,491,1.0,1,909,1.0,github.create_or_update_file github.create_gist slack.update_file github.update_gist slack.create_file github.projects_write github.update_artifact github.update_runner github.issue_write github.update_environment +q084,List collaborators on a repo,resource,github.list_repository_collaborators,1.0,1.0,1.0,1.0000,1.0000,1,1,272,1.0,1,413,1.0,1,789,1.0,github.list_repository_collaborators github.list_commits github.list_tags github.list_labels github.list_releases github.list_branches github.list_starred_repositories github.star_repository github.list_issues github.list_secret_scanning_alerts +q085,Reply in a Slack thread,verb_product,slack.slack_reply_to_thread,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,385,1.0,1,698,1.0,slack.slack_reply_to_thread slack.slack_get_thread_replies slack.slack_get_user_profile slack.slack_add_reaction slack.get_pin slack.slack_leave_channel slack.slack_create_channel slack.slack_join_channel slack.update_pin slack.slack_post_message +q086,Add an emoji reaction on Slack,verb_product,slack.slack_add_reaction,1.0,1.0,1.0,1.0000,1.0000,1,1,289,1.0,1,500,1.0,1,945,1.0,slack.slack_add_reaction discord.discord_add_reaction slack.slack_reply_to_thread slack.create_canvase slack.create_usergroup slack.create_reminder slack.slack_get_thread_replies slack.create_pin slack.create_file slack.slack_create_channel +q087,Search Slack messages,verb_product,slack.slack_search_messages,1.0,1.0,1.0,1.0000,1.0000,1,1,248,1.0,1,379,1.0,1,757,1.0,slack.slack_search_messages slack.slack_post_message slack.slack_add_reaction elasticsearch.es_search slack.slack_list_users slack.list_usergroups slack.slack_list_channels slack.slack_get_channel_history notion.notion_search brave_search.brave_web_search +q088,Create a Slack channel,verb_product,slack.slack_create_channel,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,339,1.0,1,791,1.0,slack.slack_create_channel slack.slack_join_channel slack.slack_leave_channel slack.slack_post_message slack.slack_get_channel_history slack.slack_set_channel_topic slack.create_usergroup slack.create_pin slack.slack_reply_to_thread slack.slack_add_reaction +q089,Transition a Jira issue,verb_product,jira.jira_transition_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,271,1.0,1,389,1.0,1,772,1.0,jira.jira_transition_issue jira.jira_assign_issue jira.jira_create_issue jira.jira_update_issue jira.jira_get_issue jira.jira_add_worklog jira.jira_add_comment jira.jira_search_issues sentry.sentry_update_issue jira.update_version +q090,Search Jira with JQL,identifier,jira.jira_search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,193,1.0,1,305,1.0,1,749,1.0,jira.jira_search_issues elasticsearch.es_search jira.jira_list_projects google_drive.gdrive_search zendesk.zendesk_search linear.linear_search_issues brave_search.brave_web_search jira.jira_create_sprint jira.create_sprint jira.jira_add_worklog +q091,Add a worklog to Jira,verb_product,jira.jira_add_worklog,1.0,1.0,1.0,1.0000,1.0000,1,1,233,1.0,1,401,1.0,1,862,1.0,jira.jira_add_worklog jira.jira_add_comment bitbucket.bitbucket_list_repos bitbucket.bitbucket_create_issue jira.list_attachments jira.list_sprints jira.jira_create_sprint jira.jira_list_projects bitbucket.bitbucket_create_pr jira.jira_create_issue +q092,List Linear teams,verb_product,linear.linear_list_teams,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,389,1.0,1,713,1.0,linear.linear_list_teams linear.linear_list_projects linear.linear_list_issues linear.linear_create_issue linear.list_labels linear.get_label linear.list_users linear.linear_create_project github.get_team_members linear.get_user +q093,Comment on a Linear issue,verb_product,linear.linear_create_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,195,1.0,1,355,1.0,1,824,1.0,linear.linear_create_comment linear.get_comment linear.linear_get_issue github.update_issue_comment linear.linear_search_issues linear.linear_update_issue linear.update_comment linear.list_comments linear.linear_create_issue linear.linear_list_issues +q094,Move a local file,paraphrase,filesystem.write_file,0.0,1.0,1.0,0.5000,0.6309,2,1,209,1.0,1,352,1.0,1,778,1.0,filesystem.write_file filesystem.move_file filesystem.read_media_file filesystem.read_file filesystem.read_text_file google_drive.gdrive_share_file google_drive.gdrive_update_file figma.figma_get_file gitlab.create_pipeline slack.create_file +q095,Recursively search files by pattern,paraphrase,filesystem.search_files,1.0,1.0,1.0,1.0000,1.0000,1,1,225,1.0,1,334,1.0,1,649,1.0,filesystem.search_files notion.notion_search elasticsearch.es_search filesystem.directory_tree filesystem.list_directory google_drive.gdrive_search git.git_add filesystem.list_directory_with_sizes slack.list_files filesystem.move_file +q096,Edit a file in place with old and new string,paraphrase,filesystem.edit_file,1.0,1.0,1.0,1.0000,1.0000,1,1,282,1.0,1,524,1.0,1,864,1.0,filesystem.edit_file google_drive.gdrive_list_folder google_drive.gdrive_create_file github.get_file_blame github.create_or_update_file filesystem.write_file google_drive.gdrive_update_file filesystem.list_directory_with_sizes slack.update_file slack.delete_file +q097,Read the whole knowledge graph,paraphrase,memory.read_graph,1.0,1.0,1.0,1.0000,1.0000,1,1,162,1.0,1,262,1.0,1,631,1.0,memory.read_graph memory.open_nodes memory.search_nodes memory.delete_entities memory.create_entities memory.add_observations pagerduty.pagerduty_acknowledge_incident memory.delete_relations gitlab.gitlab_create_issue gitlab.create_snippet +q098,Search memory nodes,verb_product,memory.search_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,192,1.0,1,350,1.0,1,662,1.0,memory.search_nodes memory.open_nodes elasticsearch.es_search notion.notion_search brave_search.brave_web_search github.search_orgs brave_search.brave_local_search zendesk.zendesk_search jira.jira_search_issues linear.linear_search_issues +q099,Show unstaged git diff,verb_product,git.git_diff_unstaged,1.0,1.0,1.0,1.0000,1.0000,1,1,174,1.0,1,299,1.0,1,629,1.0,git.git_diff_unstaged git.git_diff_staged git.git_reset git.git_diff git.git_show trello.trello_add_comment git.git_commit git.git_checkout github.unstar_repository git.git_log +q100,Stage files with git add,identifier,filesystem.list_directory_with_sizes,0.0,1.0,1.0,0.5000,0.6309,2,1,260,1.0,1,400,1.0,1,774,1.0,filesystem.list_directory_with_sizes git.git_add github.push_files filesystem.read_text_file filesystem.list_directory filesystem.move_file google_drive.gdrive_share_file filesystem.directory_tree filesystem.read_media_file github.delete_file +q101,Create a Shopify product,verb_product,shopify.shopify_create_product,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,380,1.0,1,719,1.0,shopify.shopify_create_product shopify.shopify_update_product shopify.shopify_get_product shopify.shopify_create_order shopify.shopify_list_products shopify.shopify_get_order stripe.stripe_create_product shopify.shopify_list_orders trello.trello_create_card vercel.vercel_create_project +q102,Cancel a Stripe subscription,verb_product,stripe.stripe_cancel_subscription,1.0,1.0,1.0,1.0000,1.0000,1,1,203,1.0,1,300,1.0,1,640,1.0,stripe.stripe_cancel_subscription stripe.stripe_list_subscriptions stripe.stripe_create_product stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_create_price stripe.stripe_list_customers stripe.stripe_get_payment_intent stripe.stripe_create_refund stripe.stripe_create_customer +q103,Create a payment intent,resource,stripe.stripe_create_payment_intent,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,466,1.0,1,917,1.0,stripe.stripe_create_payment_intent stripe.stripe_create_refund stripe.stripe_get_payment_intent notion.notion_create_page github.create_or_update_file notion.create_comment jira.create_attachment salesforce.salesforce_create_record linear.create_comment linear.create_attachment +q104,Acknowledge a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,405,1.0,1,765,1.0,pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls github.update_package notion.notion_create_page github.delete_package github.create_package +q105,Create a Datadog monitor,verb_product,datadog.datadog_create_monitor,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,461,1.0,1,905,1.0,datadog.datadog_create_monitor datadog.datadog_get_monitor datadog.datadog_list_monitors notion.create_data_source datadog.datadog_list_logs trello.trello_create_board trello.trello_create_card jira.jira_create_sprint datadog.datadog_query_metrics hubspot.hubspot_create_contact +q106,Resolve a Sentry issue,paraphrase,sentry.sentry_get_issue,0.0,1.0,1.0,0.5000,0.6309,2,1,251,1.0,1,397,1.0,1,892,1.0,sentry.sentry_get_issue sentry.sentry_update_issue sentry.sentry_list_events sentry.sentry_list_issues sentry.sentry_list_projects github.issue_write gitlab.gitlab_get_issue jira.jira_update_issue github.update_issue_comment jira.jira_create_issue +q107,Scale a Kubernetes deployment,verb_product,kubernetes.k8s_scale_deployment,1.0,1.0,1.0,1.0000,1.0000,1,1,262,1.0,1,438,1.0,1,726,1.0,kubernetes.k8s_scale_deployment kubernetes.k8s_list_deployments kubernetes.k8s_delete kubernetes.k8s_pod_logs kubernetes.k8s_get_pod kubernetes.k8s_apply vercel.vercel_get_deployment kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_list_pods +q108,Apply a Kubernetes manifest,verb_product,kubernetes.k8s_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,263,1.0,1,403,1.0,1,755,1.0,kubernetes.k8s_apply kubernetes.k8s_pod_logs kubernetes.k8s_get_pod kubernetes.k8s_delete kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_scale_deployment kubernetes.k8s_list_pods kubernetes.k8s_list_deployments slack.slack_reply_to_thread +q109,Stop a docker container,verb_product,docker.docker_stop,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,345,1.0,1,604,1.0,docker.docker_stop docker.docker_rm docker.docker_logs docker.docker_exec docker.docker_run docker.docker_ps docker.docker_images docker.docker_build hubspot.hubspot_get_contact hubspot.hubspot_search_contacts +q110,Build a docker image,verb_product,docker.docker_build,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,317,1.0,1,590,1.0,docker.docker_build docker.docker_images docker.docker_run docker.docker_stop docker.docker_rm docker.docker_ps docker.docker_logs docker.docker_exec discord.discord_send_message memory.read_graph +q111,Click an element in Playwright,verb_product,playwright.browser_click,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,343,1.0,1,646,1.0,playwright.browser_click playwright.browser_type puppeteer.puppeteer_click playwright.browser_take_screenshot notion.delete_comment jira.delete_attachment notion.get_comment linear.delete_attachment notion.list_comments jira.list_versions +q112,Type text into a browser input,paraphrase,playwright.browser_type,1.0,1.0,1.0,1.0000,0.7602,1,1,217,1.0,1,316,1.0,1,693,1.0,playwright.browser_type playwright.browser_navigate playwright.browser_wait_for playwright.browser_tab_new playwright.browser_tab_close playwright.browser_tab_list jira.create_filter jira.create_webhook playwright.browser_evaluate github.create_runner +q113,List GitLab projects,verb_product,gitlab.gitlab_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,209,1.0,1,373,1.0,1,709,1.0,gitlab.gitlab_list_projects gitlab.gitlab_list_issues github.projects_list gitlab.gitlab_list_merge_requests gitlab.gitlab_get_file gitlab.gitlab_create_issue gitlab.gitlab_get_issue jira.jira_list_projects github.projects_get gitlab.gitlab_create_pipeline +q114,Create a Bitbucket pull request,verb_product,bitbucket.bitbucket_create_pr,1.0,1.0,1.0,1.0000,1.0000,1,1,313,1.0,1,564,1.0,1,1080,1.0,bitbucket.bitbucket_create_pr bitbucket.bitbucket_list_prs bitbucket.bitbucket_create_issue github.request_copilot_review github.create_pull_request github.pull_request_review_write github.update_pull_request github.pull_request_read bitbucket.bitbucket_list_repos zendesk.zendesk_create_ticket +q115,Create a Gmail draft,verb_product,gmail.gmail_create_draft,1.0,1.0,1.0,1.0000,1.0000,1,1,235,1.0,1,374,1.0,1,814,1.0,gmail.gmail_create_draft gmail.gmail_trash_email github.create_gist hubspot.hubspot_create_contact gmail.gmail_get_email gmail.gmail_send_email stripe.stripe_create_customer gmail.gmail_search_emails vercel.vercel_create_project github.create_pull_request +q116,Share a Google Drive file,verb_product,google_drive.gdrive_share_file,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,418,1.0,1,833,1.0,google_drive.gdrive_share_file google_drive.gdrive_update_file google_drive.gdrive_get_file google_drive.gdrive_list_folder google_drive.gdrive_create_file google_drive.gdrive_search github.delete_file filesystem.write_file github.push_files filesystem.move_file +q117,List Google Calendar events,verb_product,google_calendar.gcal_list_events,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,427,1.0,1,825,1.0,google_calendar.gcal_list_events google_calendar.gcal_list_calendars google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event sentry.sentry_list_events aws_s3.s3_list_objects github.list_commits github.list_dependabot_alerts linear.list_comments +q118,Delete a calendar event,resource,google_calendar.gcal_delete_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,427,1.0,1,687,1.0,google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars github.delete_check_run linear.delete_comment notion.delete_comment linear.delete_attachment jira.delete_attachment +q119,List Vercel projects,verb_product,vercel.vercel_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,175,1.0,1,269,1.0,1,593,1.0,vercel.vercel_list_projects vercel.vercel_create_project vercel.vercel_list_deployments jira.jira_list_projects vercel.vercel_get_deployment github.projects_write asana.asana_create_project gitlab.gitlab_list_projects github.projects_get sentry.sentry_list_projects +q120,Select rows from Supabase,verb_product,supabase.supabase_select,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,354,1.0,1,659,1.0,supabase.supabase_select supabase.supabase_insert supabase.supabase_update supabase.supabase_list_tables supabase.supabase_invoke_fn mongodb.mongo_delete redis.redis_keys figma.figma_get_images memory.delete_entities redis.redis_delete +q121,Post a Figma comment,verb_product,figma.figma_post_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,368,1.0,1,715,1.0,figma.figma_post_comment figma.figma_get_comments figma.figma_get_file figma.figma_get_file_nodes figma.figma_get_images linear.get_comment linear.update_comment notion.get_comment github.update_issue_comment linear.list_comments +q122,List Okta users,verb_product,okta.okta_list_users,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,284,1.0,1,636,1.0,okta.okta_list_users okta.okta_get_user okta.okta_create_user okta.okta_deactivate_user github.list_gists notion.create_users_token slack.slack_get_users slack.slack_list_users slack.slack_get_user_profile pagerduty.pagerduty_list_oncalls +q123,Range query Prometheus,verb_product,prometheus.prom_query_range,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,347,1.0,1,702,1.0,prometheus.prom_query_range prometheus.prom_query prometheus.prom_list_metrics shopify.shopify_list_products postgres.explain_query shopify.shopify_list_customers datadog.datadog_query_metrics google_drive.gdrive_search shopify.shopify_get_product github.search_repositories +q124,Create a Grafana annotation,verb_product,grafana.grafana_create_annotation,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,434,1.0,1,813,1.0,grafana.grafana_create_annotation grafana.grafana_get_dashboard gmail.gmail_create_draft clickup.clickup_create_task notion.create_data_source github.create_gist memory.create_relations jira.create_attachment notion.create_comment vercel.vercel_create_project +q125,Apply terraform,identifier,terraform.tf_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,171,1.0,1,303,1.0,1,673,1.0,terraform.tf_apply terraform.tf_plan terraform.tf_state_list kubernetes.k8s_apply github.update_environment github.list_environments slack.slack_reply_to_thread memory.delete_entities github.update_gist github.list_releases +q126,List Asana tasks,verb_product,asana.asana_list_tasks,1.0,1.0,1.0,1.0000,1.0000,1,1,228,1.0,1,386,1.0,1,712,1.0,asana.asana_list_tasks asana.asana_get_task asana.asana_update_task asana.asana_create_task clickup.clickup_list_tasks asana.asana_create_project notion.notion_list_users trello.trello_list_cards grafana.grafana_create_annotation kubernetes.k8s_list_deployments +q127,Comment on a Trello card,verb_product,trello.trello_add_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,412,1.0,1,787,1.0,trello.trello_add_comment trello.trello_create_card trello.trello_list_cards trello.trello_create_board linear.create_comment github.update_issue_comment linear.get_comment figma.figma_post_comment notion.get_comment github.get_discussion_comments +q128,Update a ClickUp task,verb_product,clickup.clickup_update_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,401,1.0,1,787,1.0,clickup.clickup_update_task clickup.clickup_get_task clickup.clickup_create_task clickup.clickup_list_tasks confluence.confluence_update_page asana.asana_update_task gitlab.update_wiki github.projects_write notion.update_data_source gitlab.update_snippet +q129,Search Confluence,verb_product,confluence.confluence_search,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,369,1.0,1,703,1.0,confluence.confluence_search confluence.confluence_get_page confluence.confluence_create_page confluence.confluence_update_page hubspot.hubspot_search_contacts brave_search.brave_web_search brave_search.brave_local_search elasticsearch.es_search github.search_users github.search_commits +q130,Create a HubSpot deal,verb_product,hubspot.hubspot_create_deal,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,819,1.0,hubspot.hubspot_create_deal hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_search_contacts hubspot.hubspot_list_contacts github.create_artifact github.create_hook github.create_gist confluence.confluence_create_page clickup.clickup_create_task +q131,List Zendesk tickets,verb_product,zendesk.zendesk_list_tickets,1.0,1.0,1.0,1.0000,1.0000,1,1,222,1.0,1,367,1.0,1,739,1.0,zendesk.zendesk_list_tickets zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_search zendesk.zendesk_create_ticket aws_s3.s3_list_buckets notion.list_comments github.list_code_scanning_alerts github.list_secret_scanning_alerts jira.list_sprints +q132,List Discord channels,verb_product,discord.discord_list_channels,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,321,1.0,1,635,1.0,discord.discord_list_channels discord.discord_get_channel discord.discord_send_message discord.discord_add_reaction discord.discord_list_members slack.slack_get_channel_history slack.slack_leave_channel slack.slack_post_message slack.slack_join_channel slack.slack_list_channels +q133,Set a Redis key,verb_product,redis.redis_get,0.0,1.0,1.0,0.5000,0.6309,2,1,182,1.0,1,321,1.0,1,663,1.0,redis.redis_get redis.redis_set redis.redis_delete redis.redis_keys filesystem.edit_file github.get_label github.label_write github.get_teams github.search_commits github.get_gist +q134,Insert MongoDB documents,verb_product,mongodb.mongo_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,381,1.0,1,779,1.0,mongodb.mongo_insert mongodb.mongo_delete mongodb.mongo_find mongodb.mongo_update mongodb.mongo_list_collections elasticsearch.es_index_doc stripe.stripe_create_payment_intent github.create_environment okta.okta_list_users stripe.stripe_create_price +q135,Run a MySQL query,verb_product,mysql.mysql_query,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,281,1.0,1,625,1.0,mysql.mysql_query postgres.query postgres.explain_query sqlite.sqlite_query mysql.mysql_list_tables memory.search_nodes prometheus.prom_query salesforce.salesforce_query linear.linear_assign_issue datadog.datadog_query_metrics +q136,Index a document in Elasticsearch,verb_product,elasticsearch.es_index_doc,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,392,1.0,1,869,1.0,elasticsearch.es_index_doc elasticsearch.es_get_doc elasticsearch.es_search mongodb.mongo_insert mongodb.mongo_find notion.update_comment slack.list_usergroups github.create_or_update_file notion.create_comment mongodb.mongo_update +q137,Get an S3 object,verb_product,aws_s3.s3_delete_object,0.0,1.0,1.0,0.5000,0.6309,2,1,234,1.0,1,392,1.0,1,714,1.0,aws_s3.s3_delete_object aws_s3.s3_get_object aws_s3.s3_list_objects aws_s3.s3_put_object github.get_latest_release github.get_label gitlab.get_label github.get_milestone salesforce.salesforce_get_record okta.okta_get_user +q138,List Cloudflare zones,verb_product,cloudflare.cf_list_zones,1.0,1.0,1.0,1.0000,1.0000,1,1,215,1.0,1,341,1.0,1,722,1.0,cloudflare.cf_list_zones cloudflare.cf_list_dns cloudflare.cf_create_dns cloudflare.cf_purge_cache kubernetes.k8s_list_deployments sequential_thinking.sequentialthinking kubernetes.k8s_scale_deployment gitlab.list_milestones notion.notion_list_users prometheus.prom_list_metrics +q139,Create a Stripe invoice,verb_product,stripe.stripe_create_invoice,1.0,1.0,1.0,1.0000,1.0000,1,1,185,1.0,1,334,1.0,1,676,1.0,stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_create_product stripe.stripe_create_price stripe.stripe_create_refund stripe.stripe_create_customer stripe.stripe_create_payment_intent stripe.stripe_cancel_subscription stripe.stripe_list_subscriptions slack.slack_create_channel +q140,List GitHub Actions workflows,verb_product,github.actions_list,1.0,1.0,1.0,1.0000,1.0000,1,1,293,1.0,1,435,1.0,1,824,1.0,github.actions_list github.actions_get github.actions_run_trigger github.list_discussions github.list_tags github.actions_get_job_logs github.list_gists github.list_commits github.list_branches github.list_labels +q141,Get GitHub Actions job logs,verb_product,github.actions_get_job_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,460,1.0,1,815,1.0,github.actions_get_job_logs github.actions_get github.actions_list github.actions_run_trigger github.list_discussions github.get_label github.search_orgs github.get_runner github.search_commits github.label_write +q142,Create a GitHub repository,verb_product,github.create_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,235,1.0,1,388,1.0,1,928,1.0,github.create_repository github.delete_repository github.star_repository github.fork_repository github.unstar_repository github.create_branch github.create_or_update_file github.create_issue github.get_tag github.list_repository_collaborators +q143,Checkout a git branch,verb_product,git.git_checkout,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,384,1.0,1,737,1.0,git.git_checkout git.git_create_branch git.git_branch github.create_branch github.list_branches github.list_check_runs github.delete_check_run git.git_diff github.get_check_run github.list_commits +q144,Show git log,identifier,git.git_log,1.0,1.0,1.0,1.0000,1.0000,1,1,206,1.0,1,306,1.0,1,606,1.0,git.git_log git.git_show git.git_diff git.git_branch git.git_diff_unstaged git.git_diff_staged git.git_status github.list_gists git.git_checkout github.get_commit +q145,List postgres schemas,verb_product,postgres.list_schemas,1.0,1.0,1.0,1.0000,1.0000,1,1,158,1.0,1,277,1.0,1,606,1.0,postgres.list_schemas postgres.list_tables postgres.query postgres.describe_table postgres.explain_query mysql.mysql_list_tables github.projects_list github.actions_list github.list_secret_scanning_alerts github.list_labels +q146,Open nodes in the knowledge graph,verb_product,memory.open_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,163,1.0,1,267,1.0,1,575,1.0,memory.open_nodes memory.search_nodes memory.read_graph memory.create_entities memory.add_observations memory.create_relations memory.delete_entities memory.delete_relations gitlab.list_jobs gitlab.list_releases +q147,Get file metadata from the filesystem,paraphrase,filesystem.read_media_file,0.0,0.0,1.0,0.2500,0.4307,4,2,410,0.0,1,334,1.0,1,683,1.0,filesystem.read_media_file filesystem.read_file filesystem.read_text_file filesystem.get_file_info google_drive.gdrive_get_file figma.figma_get_file figma.figma_get_images filesystem.write_file filesystem.move_file figma.figma_get_file_nodes +q148,Write a new local file,paraphrase,filesystem.write_file,1.0,1.0,1.0,1.0000,1.0000,1,1,199,1.0,1,392,1.0,1,875,1.0,filesystem.write_file filesystem.read_media_file playwright.browser_tab_new github.issue_write filesystem.read_file filesystem.read_text_file github.create_or_update_file google_drive.gdrive_list_folder google_drive.gdrive_share_file jira.jira_create_sprint +q149,Search GitHub issues,verb_product,github.search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,360,1.0,1,830,1.0,github.search_issues github.search_orgs github.search_users github.search_pull_requests github.issue_read github.issue_write github.assign_copilot_to_issue github.search_commits github.add_issue_comment jira.jira_search_issues +q150,Get the authenticated GitHub user,paraphrase,github.get_teams,0.0,1.0,1.0,0.5000,0.6309,2,1,179,1.0,1,300,1.0,1,603,1.0,github.get_teams github.get_me github.list_notifications github.search_users github.get_latest_release github.search_commits github.get_gist github.update_gist github.get_team_members github.list_gists diff --git a/experiments/mcp-tool-search/results/hashed_ngram_embed.json b/experiments/mcp-tool-search/results/hashed_ngram_embed.json new file mode 100644 index 00000000..41fe8e43 --- /dev/null +++ b/experiments/mcp-tool-search/results/hashed_ngram_embed.json @@ -0,0 +1,7013 @@ +{ + "summary": { + "n": 150, + "index_s": 0.021400491000008515, + "rss_mb": 0.19140625, + "p50_ms": 4.2002120000574905, + "p95_ms": 4.377745999943272, + "recall@1": 0.84, + "recall@1_lo": 0.78, + "recall@1_hi": 0.9, + "recall@3": 0.9666666666666667, + "recall@3_lo": 0.94, + "recall@3_hi": 0.9933333333333333, + "recall@5": 0.9933333333333333, + "recall@5_lo": 0.98, + "recall@5_hi": 1.0, + "mrr@10": 0.9085555555555556, + "mrr@10_lo": 0.8743333333333334, + "mrr@10_hi": 0.9422222222222223, + "ndcg@10": 0.9211576774050081, + "ndcg@10_lo": 0.8934063236877832, + "ndcg@10_hi": 0.9471845982817189 + }, + "meta": { + "name": "hashed_ngram_embed", + "needs_model": false, + "needs_api": false, + "notes": "fallback sparse hashed n-gram embedding, not neural" + }, + "rows": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.issue_write", + "github.issue_read", + "github.create_gist", + "github.projects_write", + "gitlab.gitlab_create_issue", + "github.create_issue", + "github.update_issue_comment", + "github.label_write", + "github.add_issue_comment", + "github.update_gist" + ], + "hit1": "github.issue_write", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.6508780296742857, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 334, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 509, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1047, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "kind": "verb_product", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_create_issue", + "jira.jira_create_sprint", + "jira.create_webhook", + "jira.create_filter", + "jira.create_board", + "jira.create_sprint", + "jira.jira_get_issue", + "jira.create_attachment", + "clickup.clickup_create_task", + "jira.jira_assign_issue" + ], + "hit1": "jira.jira_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 345, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 535, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 940, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "kind": "verb_product", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "ranking": [ + "slack.slack_post_message", + "slack.slack_search_messages", + "slack.slack_add_reaction", + "slack.slack_get_channel_history", + "slack.slack_create_channel", + "discord.discord_send_message", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.update_pin", + "slack.slack_get_thread_replies" + ], + "hit1": "slack.slack_post_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 394, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "kind": "resource", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "ranking": [ + "github.create_pull_request", + "github.list_pull_requests", + "github.merge_pull_request", + "github.request_copilot_review", + "github.search_pull_requests", + "github.pull_request_read", + "github.pull_request_review_write", + "gitlab.gitlab_get_merge_request", + "gitlab.gitlab_list_merge_requests", + "github.update_pull_request" + ], + "hit1": "github.create_pull_request", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.626665273802673, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 363, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 517, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1044, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "ranking": [ + "github.create_pull_request", + "github.request_copilot_review", + "github.search_pull_requests", + "github.merge_pull_request", + "github.list_pull_requests", + "github.pull_request_read", + "github.add_comment_to_pending_review", + "github.update_pull_request", + "github.pull_request_review_write", + "github.update_pull_request_branch" + ], + "hit1": "github.create_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 327, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 520, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1144, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "kind": "verb_product", + "gold": [ + "postgres.query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "mysql.mysql_query", + "postgres.explain_query", + "sqlite.sqlite_query", + "salesforce.salesforce_query", + "slack.slack_post_message", + "postgres.list_schemas", + "memory.search_nodes", + "postgres.describe_table", + "prometheus.prom_query" + ], + "hit1": "postgres.query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 575, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "kind": "paraphrase", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "ranking": [ + "filesystem.read_media_file", + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.edit_file", + "playwright.browser_take_screenshot", + "github.get_file_contents", + "gitlab.gitlab_get_file", + "filesystem.write_file", + "slack.get_file", + "google_drive.gdrive_get_file" + ], + "hit1": "filesystem.read_media_file", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.66967181649423, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 766, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "kind": "paraphrase", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "ranking": [ + "google_drive.gdrive_list_folder", + "github.push_files", + "filesystem.move_file", + "slack.list_files", + "filesystem.list_directory_with_sizes", + "filesystem.list_directory", + "slack.delete_file", + "github.get_file_blame", + "filesystem.search_files", + "filesystem.directory_tree" + ], + "hit1": "google_drive.gdrive_list_folder", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.2, + "ndcg@10": 0.44342483448720016, + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 464, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 715, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "ranking": [ + "memory.read_graph", + "memory.open_nodes", + "memory.search_nodes", + "memory.add_observations", + "memory.delete_entities", + "memory.create_entities", + "memory.delete_relations", + "gitlab.list_releases", + "memory.create_relations", + "github.get_me" + ], + "hit1": "memory.read_graph", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.4627870818679153, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 343, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 272, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 554, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "kind": "verb_product", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "ranking": [ + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "github.search_users", + "elasticsearch.es_search", + "memory.search_nodes", + "github.search_orgs", + "jira.jira_search_issues", + "github.search_commits", + "github.search_repositories", + "hubspot.hubspot_search_contacts" + ], + "hit1": "brave_search.brave_web_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 236, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 359, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "kind": "verb_product", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "ranking": [ + "fetch.fetch", + "filesystem.read_file", + "postgres.describe_table", + "stripe.stripe_get_payment_intent", + "memory.search_nodes", + "stripe.stripe_create_payment_intent", + "playwright.browser_take_screenshot", + "playwright.browser_navigate", + "memory.open_nodes", + "github.get_file_contents" + ], + "hit1": "fetch.fetch", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 336, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "kind": "identifier", + "gold": [ + "git.git_status" + ], + "alts": [], + "ranking": [ + "git.git_show", + "git.git_status", + "git.git_diff", + "github.list_runners", + "git.git_diff_staged", + "git.git_diff_unstaged", + "git.git_log", + "git.git_branch", + "github.list_packages", + "github.list_check_runs" + ], + "hit1": "git.git_show", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 637, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "kind": "paraphrase", + "gold": [ + "git.git_commit" + ], + "alts": [], + "ranking": [ + "git.git_reset", + "git.git_commit", + "git.git_diff_staged", + "github.get_commit", + "git.git_diff_unstaged", + "filesystem.read_media_file", + "filesystem.read_text_file", + "github.delete_file", + "git.git_show", + "github.push_files" + ], + "hit1": "git.git_reset", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 754, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_comment", + "linear.linear_create_issue", + "linear.linear_update_issue", + "linear.linear_get_issue", + "linear.linear_create_project", + "linear.linear_list_issues", + "jira.jira_create_issue", + "linear.linear_assign_issue", + "linear.linear_search_issues", + "linear.create_label" + ], + "hit1": "linear.linear_create_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 471, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 895, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "kind": "paraphrase", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_project", + "linear.linear_list_projects", + "jira.jira_create_issue", + "linear.list_users", + "github.projects_write", + "linear.linear_create_issue", + "linear.list_cycles", + "github.projects_get", + "linear.create_user", + "jira.jira_list_projects" + ], + "hit1": "linear.linear_create_project", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.16666666666666666, + "ndcg@10": 0.3562071871080222, + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 580, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 849, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "ranking": [ + "github.update_issue_comment", + "github.add_issue_comment", + "github.list_commits", + "figma.figma_post_comment", + "github.issue_read", + "github.list_issues", + "github.get_gist", + "github.issue_write", + "github.add_comment_to_pending_review", + "linear.update_comment" + ], + "hit1": "github.update_issue_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 337, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 509, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1018, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "kind": "ambiguous", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "ranking": [ + "github.merge_pull_request", + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_get_merge_request", + "github.request_copilot_review", + "github.pull_request_read", + "github.search_pull_requests", + "github.create_pull_request", + "github.update_pull_request_branch", + "github.list_pull_requests", + "gitlab.gitlab_list_merge_requests" + ], + "hit1": "github.merge_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 474, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 933, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "kind": "verb_product", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "ranking": [ + "github.actions_run_trigger", + "github.actions_get", + "github.actions_list", + "github.actions_get_job_logs", + "github.list_commits", + "github.list_branches", + "github.list_discussions", + "github.list_tags", + "github.list_pull_requests", + "github.list_labels" + ], + "hit1": "github.actions_run_trigger", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 491, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 862, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "kind": "resource", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "ranking": [ + "github.get_dependabot_alert", + "github.list_dependabot_alerts", + "github.get_code_scanning_alert", + "github.get_secret_scanning_alert", + "google_calendar.gcal_delete_event", + "gitlab.gitlab_get_issue", + "jira.jira_get_issue", + "linear.linear_get_issue", + "google_calendar.gcal_list_calendars", + "github.issue_read" + ], + "hit1": "github.get_dependabot_alert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8597186998521972, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 286, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 773, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "kind": "verb_product", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_issues", + "sentry.sentry_list_events", + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "sentry.sentry_list_projects", + "gitlab.gitlab_list_issues", + "github.list_issues", + "gitlab.gitlab_get_issue", + "jira.jira_update_issue", + "github.issue_write" + ], + "hit1": "sentry.sentry_list_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 845, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_create_invoice", + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_price", + "stripe.stripe_list_customers", + "stripe.stripe_create_refund", + "stripe.stripe_create_product", + "stripe.stripe_list_invoices", + "stripe.stripe_list_subscriptions" + ], + "hit1": "stripe.stripe_create_customer", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "kind": "paraphrase", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_refund", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_customer", + "stripe.stripe_create_invoice", + "stripe.stripe_create_price", + "stripe.stripe_get_customer", + "stripe.stripe_create_product", + "stripe.stripe_list_invoices", + "stripe.stripe_list_customers" + ], + "hit1": "stripe.stripe_create_refund", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "kind": "verb_product", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "ranking": [ + "gmail.gmail_send_email", + "gmail.gmail_search_emails", + "gmail.gmail_list_labels", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "gmail.gmail_modify_email", + "gmail.gmail_create_draft", + "google_drive.gdrive_share_file", + "filesystem.read_text_file", + "filesystem.read_media_file" + ], + "hit1": "gmail.gmail_send_email", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "kind": "verb_product", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "ranking": [ + "gmail.gmail_search_emails", + "elasticsearch.es_search", + "gmail.gmail_send_email", + "notion.notion_search", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "brave_search.brave_web_search", + "gmail.gmail_create_draft", + "gmail.gmail_modify_email", + "gmail.gmail_list_labels" + ], + "hit1": "gmail.gmail_search_emails", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "kind": "paraphrase", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_create_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_share_file", + "google_drive.gdrive_update_file", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "filesystem.edit_file", + "slack.slack_upload_file", + "filesystem.read_text_file", + "gitlab.gitlab_get_file" + ], + "hit1": "google_drive.gdrive_create_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 272, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 826, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_create_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "github.projects_write", + "notion.notion_create_page", + "confluence.confluence_create_page", + "trello.trello_create_card", + "trello.trello_create_board" + ], + "hit1": "google_calendar.gcal_create_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 837, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_list_pods", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_get_events", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_apply", + "kubernetes.k8s_delete", + "kubernetes.k8s_scale_deployment", + "slack.list_canvases" + ], + "hit1": "kubernetes.k8s_list_pods", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 749, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_pod_logs", + "github.get_teams", + "git.git_create_branch", + "docker.docker_logs", + "gitlab.gitlab_get_file", + "gitlab.get_snippet", + "gitlab.create_snippet", + "gitlab.update_snippet", + "github.create_branch", + "gitlab.delete_snippet" + ], + "hit1": "kubernetes.k8s_pod_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 783, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "ranking": [ + "docker.docker_stop", + "docker.docker_exec", + "docker.docker_rm", + "docker.docker_run", + "docker.docker_logs", + "docker.docker_ps", + "docker.docker_images", + "docker.docker_build", + "mysql.mysql_query", + "hubspot.hubspot_get_contact" + ], + "hit1": "docker.docker_stop", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 411, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "kind": "verb_product", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "ranking": [ + "docker.docker_logs", + "docker.docker_rm", + "docker.docker_stop", + "docker.docker_exec", + "docker.docker_ps", + "docker.docker_run", + "git.git_log", + "docker.docker_images", + "docker.docker_build", + "hubspot.hubspot_get_contact" + ], + "hit1": "docker.docker_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 298, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 609, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "kind": "ambiguous", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "ranking": [ + "playwright.browser_navigate", + "puppeteer.puppeteer_navigate", + "playwright.browser_tab_new", + "playwright.browser_tab_close", + "playwright.browser_tab_list", + "playwright.browser_wait_for", + "playwright.browser_evaluate", + "github.create_runner", + "notion.create_users_token", + "playwright.browser_type" + ], + "hit1": "playwright.browser_navigate", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 265, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "kind": "ambiguous", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "ranking": [ + "playwright.browser_take_screenshot", + "puppeteer.puppeteer_screenshot", + "notion.notion_update_page", + "pagerduty.pagerduty_list_oncalls", + "pagerduty.pagerduty_create_incident", + "playwright.browser_snapshot", + "filesystem.read_file", + "filesystem.read_media_file", + "linear.update_cycle", + "github.update_gist" + ], + "hit1": "playwright.browser_take_screenshot", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 408, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_create_merge_request", + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_get_merge_request", + "gitlab.gitlab_list_merge_requests", + "github.merge_pull_request", + "gitlab.gitlab_create_issue", + "gitlab.gitlab_create_pipeline", + "github.request_copilot_review", + "gitlab.gitlab_get_issue", + "gitlab.gitlab_list_issues" + ], + "hit1": "gitlab.gitlab_create_merge_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 883, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "kind": "verb_product", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "ranking": [ + "notion.notion_search", + "notion.notion_get_page", + "notion.notion_create_page", + "notion.notion_update_page", + "elasticsearch.es_search", + "notion.notion_list_users", + "brave_search.brave_web_search", + "notion.notion_get_comments", + "notion.notion_create_database", + "brave_search.brave_local_search" + ], + "hit1": "notion.notion_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 708, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "kind": "verb_product", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "ranking": [ + "notion.notion_create_page", + "notion.notion_get_page", + "notion.notion_update_page", + "notion.notion_create_database", + "notion.create_data_source", + "notion.create_comment", + "notion.notion_search", + "github.create_repository", + "notion.notion_get_comments", + "github.create_package" + ], + "hit1": "notion.notion_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 419, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 815, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "kind": "verb_product", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "ranking": [ + "notion.notion_query_database", + "notion.notion_create_database", + "notion.notion_search", + "notion.delete_data_source", + "notion.get_data_source", + "notion.notion_get_block", + "notion.create_data_source", + "notion.notion_get_page", + "notion.notion_list_users", + "notion.update_data_source" + ], + "hit1": "notion.notion_query_database", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 691, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_search", + "jira.create_sprint", + "jira.jira_create_sprint", + "clickup.clickup_create_task", + "notion.create_comment", + "jira.jira_create_issue" + ], + "hit1": "zendesk.zendesk_get_ticket", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 881, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_search_contacts", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_create_deal", + "confluence.confluence_create_page", + "github.create_gist", + "gmail.gmail_create_draft", + "github.create_artifact", + "clickup.clickup_create_task" + ], + "hit1": "hubspot.hubspot_create_contact", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 805, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "kind": "verb_product", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_create_lead", + "salesforce.salesforce_create_record", + "salesforce.salesforce_convert_lead", + "salesforce.salesforce_update_record", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_get_record", + "salesforce.salesforce_query", + "salesforce.salesforce_list_opportunities", + "pagerduty.pagerduty_create_incident", + "vercel.vercel_create_project" + ], + "hit1": "salesforce.salesforce_create_lead", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 766, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "kind": "identifier", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_query", + "postgres.query", + "mysql.mysql_query", + "memory.search_nodes", + "sqlite.sqlite_query", + "prometheus.prom_query", + "linear.update_label", + "postgres.explain_query", + "discord.discord_send_message", + "linear.update_cycle" + ], + "hit1": "salesforce.salesforce_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 187, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "kind": "verb_product", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "ranking": [ + "shopify.shopify_list_orders", + "shopify.shopify_get_order", + "shopify.shopify_list_products", + "shopify.shopify_create_order", + "shopify.shopify_list_customers", + "shopify.shopify_get_product", + "shopify.shopify_update_product", + "shopify.shopify_create_product", + "github.projects_list", + "github.list_discussions" + ], + "hit1": "shopify.shopify_list_orders", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 691, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "notion.notion_create_page", + "github.create_gist", + "confluence.confluence_create_page", + "slack.create_pin" + ], + "hit1": "pagerduty.pagerduty_create_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 818, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "kind": "paraphrase", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_list_oncalls", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_list_incidents", + "notion.notion_create_page", + "memory.open_nodes", + "twilio.twilio_make_call", + "memory.search_nodes" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 749, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "kind": "verb_product", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "ranking": [ + "datadog.datadog_query_metrics", + "datadog.datadog_create_monitor", + "datadog.datadog_list_monitors", + "datadog.datadog_list_logs", + "datadog.datadog_get_monitor", + "notion.notion_query_database", + "prometheus.prom_query", + "mysql.mysql_query", + "filesystem.get_file_info", + "prometheus.prom_list_metrics" + ], + "hit1": "datadog.datadog_query_metrics", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "kind": "verb_product", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "ranking": [ + "datadog.datadog_list_logs", + "elasticsearch.es_search", + "notion.notion_search", + "github.search_orgs", + "datadog.datadog_create_monitor", + "github.search_users", + "datadog.datadog_get_monitor", + "brave_search.brave_web_search", + "datadog.datadog_list_monitors", + "github.search_commits" + ], + "hit1": "datadog.datadog_list_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 422, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 736, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_list_buckets", + "aws_s3.s3_list_objects", + "bitbucket.bitbucket_list_repos", + "bitbucket.bitbucket_list_prs", + "zendesk.zendesk_list_tickets", + "terraform.tf_state_list", + "vercel.vercel_list_deployments", + "vercel.vercel_list_projects", + "gitlab.list_snippets", + "github.list_packages" + ], + "hit1": "aws_s3.s3_list_buckets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 613, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "kind": "paraphrase", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_put_object", + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_list_objects", + "vercel.vercel_create_project", + "vercel.vercel_list_projects", + "github.projects_write", + "twilio.twilio_send_sms", + "gmail.gmail_send_email", + "gitlab.gitlab_list_projects" + ], + "hit1": "aws_s3.s3_put_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "github.search_orgs", + "gmail.gmail_search_emails", + "slack.list_reminders", + "gitlab.get_release", + "gitlab.gitlab_create_merge_request", + "gitlab.gitlab_list_merge_requests" + ], + "hit1": "cloudflare.cf_purge_cache", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 732, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "cloudflare.cf_list_zones", + "cloudflare.cf_purge_cache", + "trello.trello_create_card", + "trello.trello_create_board", + "github.create_repository", + "notion.create_data_source", + "salesforce.salesforce_create_record", + "jira.create_attachment" + ], + "hit1": "cloudflare.cf_create_dns", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 782, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_deployments", + "vercel.vercel_get_deployment", + "kubernetes.k8s_list_deployments", + "vercel.vercel_list_projects", + "vercel.vercel_create_project", + "kubernetes.k8s_scale_deployment", + "github.list_environments", + "okta.okta_list_users", + "asana.asana_list_tasks", + "clickup.clickup_list_tasks" + ], + "hit1": "vercel.vercel_list_deployments", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 637, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "ranking": [ + "supabase.supabase_insert", + "supabase.supabase_update", + "supabase.supabase_select", + "supabase.supabase_invoke_fn", + "supabase.supabase_list_tables", + "mongodb.mongo_insert", + "jira.jira_create_issue", + "jira.get_version", + "jira.create_version", + "playwright.browser_navigate" + ], + "hit1": "supabase.supabase_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 741, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "kind": "paraphrase", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "ranking": [ + "figma.figma_get_images", + "docker.docker_images", + "figma.figma_get_file_nodes", + "memory.delete_entities", + "salesforce.salesforce_list_opportunities", + "gitlab.gitlab_get_file", + "figma.figma_get_comments", + "github.list_secret_scanning_alerts", + "figma.figma_get_file", + "git.git_commit" + ], + "hit1": "figma.figma_get_images", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "kind": "verb_product", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "ranking": [ + "twilio.twilio_send_sms", + "twilio.twilio_list_messages", + "twilio.twilio_make_call", + "docker.docker_build", + "gmail.gmail_send_email", + "github.update_pull_request_branch", + "okta.okta_deactivate_user", + "git.git_checkout", + "google_calendar.gcal_create_event", + "google_drive.gdrive_create_file" + ], + "hit1": "twilio.twilio_send_sms", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 848, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "kind": "verb_product", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "ranking": [ + "okta.okta_create_user", + "okta.okta_deactivate_user", + "okta.okta_get_user", + "vercel.vercel_create_project", + "slack.create_usergroup", + "asana.asana_create_project", + "okta.okta_list_users", + "linear.create_user", + "notion.create_users_token", + "stripe.stripe_create_price" + ], + "hit1": "okta.okta_create_user", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 744, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query", + "prometheus.prom_query_range", + "prometheus.prom_list_metrics", + "postgres.explain_query", + "postgres.query", + "datadog.datadog_query_metrics", + "shopify.shopify_list_products", + "sentry.sentry_list_issues", + "sqlite.sqlite_query", + "salesforce.salesforce_query" + ], + "hit1": "prometheus.prom_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 708, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "kind": "verb_product", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "ranking": [ + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "elasticsearch.es_search", + "grafana.grafana_create_annotation", + "notion.notion_search", + "brave_search.brave_local_search", + "memory.search_nodes", + "brave_search.brave_web_search", + "github.search_orgs", + "gitlab.gitlab_list_projects" + ], + "hit1": "grafana.grafana_search_dashboards", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 198, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 677, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "kind": "verb_product", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "ranking": [ + "circleci.circleci_trigger_pipeline", + "circleci.circleci_get_pipeline", + "circleci.circleci_list_pipelines", + "gitlab.delete_pipeline", + "gitlab.gitlab_create_pipeline", + "gitlab.update_pipeline", + "gitlab.get_pipeline", + "gitlab.create_pipeline", + "gitlab.list_pipelines", + "linear.delete_cycle" + ], + "hit1": "circleci.circleci_trigger_pipeline", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 325, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 676, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "kind": "identifier", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "ranking": [ + "terraform.tf_plan", + "terraform.tf_apply", + "terraform.tf_state_list", + "playwright.browser_fill_form", + "github.list_releases", + "playwright.browser_click", + "datadog.datadog_get_monitor", + "postgres.explain_query", + "memory.delete_entities", + "salesforce.salesforce_query" + ], + "hit1": "terraform.tf_plan", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 293, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 582, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "kind": "verb_product", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "ranking": [ + "asana.asana_create_task", + "asana.asana_get_task", + "asana.asana_update_task", + "asana.asana_create_project", + "asana.asana_list_tasks", + "grafana.grafana_create_annotation", + "clickup.clickup_create_task", + "hubspot.hubspot_create_deal", + "notion.create_users_token", + "slack.create_pin" + ], + "hit1": "asana.asana_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 872, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "ranking": [ + "trello.trello_create_card", + "trello.trello_create_board", + "trello.trello_add_comment", + "trello.trello_list_cards", + "jira.create_board", + "github.create_check_run", + "github.create_gist", + "slack.create_canvase", + "slack.create_reminder", + "linear.create_user" + ], + "hit1": "trello.trello_create_card", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 411, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 895, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_create_task", + "clickup.clickup_get_task", + "clickup.clickup_update_task", + "clickup.clickup_list_tasks", + "notion.create_comment", + "github.create_artifact", + "trello.trello_create_board", + "jira.create_attachment", + "github.create_check_run", + "gitlab.create_release" + ], + "hit1": "clickup.clickup_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 858, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "kind": "verb_product", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "ranking": [ + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "confluence.confluence_get_page", + "confluence.confluence_search", + "jira.create_board", + "jira.create_attachment", + "pagerduty.pagerduty_create_incident", + "github.create_or_update_file", + "jira.create_version", + "jira.create_filter" + ], + "hit1": "confluence.confluence_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 943, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "kind": "verb_product", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "discord.discord_add_reaction", + "discord.discord_get_channel", + "discord.discord_list_members", + "discord.discord_list_channels", + "slack.slack_search_messages", + "git.git_commit", + "slack.slack_post_message", + "twilio.twilio_list_messages", + "datadog.datadog_create_monitor" + ], + "hit1": "discord.discord_send_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 692, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_search", + "elasticsearch.es_get_doc", + "elasticsearch.es_index_doc", + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "github.search_users", + "memory.search_nodes", + "grafana.grafana_search_dashboards", + "notion.notion_search", + "zendesk.zendesk_search" + ], + "hit1": "elasticsearch.es_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 696, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "redis.redis_set", + "redis.redis_delete", + "redis.redis_keys", + "github.get_label", + "github.get_teams", + "github.search_commits", + "github.get_gist", + "github.label_write", + "linear.get_attachment" + ], + "hit1": "redis.redis_get", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 622, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert", + "mongodb.mongo_find", + "mongodb.mongo_delete", + "mongodb.mongo_update", + "mongodb.mongo_list_collections", + "elasticsearch.es_index_doc", + "github.list_environments", + "zendesk.zendesk_create_ticket", + "elasticsearch.es_get_doc", + "stripe.stripe_create_payment_intent" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 788, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "kind": "verb_product", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "ranking": [ + "sqlite.sqlite_list_tables", + "sqlite.sqlite_describe_table", + "sqlite.sqlite_query", + "mysql.mysql_list_tables", + "supabase.supabase_list_tables", + "supabase.supabase_select", + "postgres.list_tables", + "github.list_milestones", + "gitlab.list_milestones", + "supabase.supabase_insert" + ], + "hit1": "sqlite.sqlite_list_tables", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 156, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 243, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 591, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "kind": "verb_product", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "ranking": [ + "postgres.describe_table", + "postgres.query", + "sqlite.sqlite_describe_table", + "postgres.list_tables", + "postgres.explain_query", + "postgres.list_schemas", + "hubspot.hubspot_create_deal", + "confluence.confluence_update_page", + "github.create_check_run", + "jira.create_board" + ], + "hit1": "postgres.describe_table", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 290, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "gitlab.gitlab_create_issue", + "github.create_issue", + "github.update_issue_comment", + "github.add_issue_comment", + "jira.jira_create_issue", + "github.search_issues", + "github.create_gist", + "github.issue_write", + "github.assign_copilot_to_issue", + "bitbucket.bitbucket_create_issue" + ], + "hit1": "gitlab.gitlab_create_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.5995311664203287, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 347, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 567, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1062, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "ranking": [ + "gitlab.gitlab_create_issue", + "jira.jira_create_issue", + "jira.create_version", + "clickup.clickup_create_task", + "bitbucket.bitbucket_create_issue", + "gitlab.gitlab_get_issue", + "slack.slack_create_channel", + "gitlab.create_label", + "github.create_gist", + "jira.create_board" + ], + "hit1": "gitlab.gitlab_create_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.20151514190050246, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 319, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 501, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 917, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "kind": "identifier", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "ranking": [ + "filesystem.create_directory", + "filesystem.list_directory", + "filesystem.list_directory_with_sizes", + "filesystem.list_allowed_directories", + "filesystem.directory_tree", + "github.get_file_contents", + "filesystem.move_file", + "github.list_repository_collaborators", + "linear.linear_list_projects", + "terraform.tf_state_list" + ], + "hit1": "filesystem.create_directory", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 621, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "kind": "paraphrase", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "ranking": [ + "sequential_thinking.sequentialthinking", + "filesystem.edit_file", + "kubernetes.k8s_get_pod", + "filesystem.list_allowed_directories", + "github.list_pull_requests", + "stripe.stripe_list_subscriptions", + "playwright.browser_wait_for", + "kubernetes.k8s_get_events", + "notion.list_data_sources", + "git.git_status" + ], + "hit1": "sequential_thinking.sequentialthinking", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 302, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 430, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "kind": "paraphrase", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "ranking": [ + "time.convert_time", + "time.get_current_time", + "playwright.browser_wait_for", + "slack.list_pins", + "gitlab.delete_snippet", + "slack.get_pin", + "gitlab.get_snippet", + "slack.delete_pin", + "slack.list_bookmarks", + "github.create_issue" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8597186998521972, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "kind": "verb_product", + "gold": [ + "time.convert_time" + ], + "alts": [], + "ranking": [ + "time.convert_time", + "time.get_current_time", + "git.git_diff", + "docker.docker_logs", + "notion.create_users_token", + "slack.create_usergroup", + "google_calendar.gcal_list_events", + "slack.slack_add_reaction", + "playwright.browser_wait_for", + "okta.okta_create_user" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 779, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "kind": "verb_product", + "gold": [ + "github.star_repository" + ], + "alts": [], + "ranking": [ + "github.star_repository", + "github.unstar_repository", + "github.get_tag", + "github.list_commits", + "github.delete_repository", + "github.update_issue_comment", + "github.list_issues", + "github.list_tags", + "github.fork_repository", + "github.list_pull_requests" + ], + "hit1": "github.star_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 414, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 861, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "kind": "resource", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "ranking": [ + "github.fork_repository", + "github.delete_repository", + "github.star_repository", + "github.unstar_repository", + "github.create_repository", + "github.get_repository_tree", + "github.list_repository_collaborators", + "github.get_tag", + "github.get_file_blame", + "github.list_releases" + ], + "hit1": "github.fork_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 824, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "kind": "verb_product", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "ranking": [ + "github.get_latest_release", + "github.list_releases", + "github.get_release_by_tag", + "github.search_repositories", + "github.search_commits", + "github.search_orgs", + "github.search_code", + "github.search_users", + "github.get_label", + "github.search_pull_requests" + ], + "hit1": "github.get_latest_release", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "kind": "resource", + "gold": [ + "github.create_gist" + ], + "alts": [], + "ranking": [ + "github.create_gist", + "vercel.vercel_create_project", + "gmail.gmail_create_draft", + "gitlab.gitlab_create_pipeline", + "github.update_gist", + "slack.create_canvase", + "stripe.stripe_create_refund", + "github.projects_write", + "hubspot.hubspot_create_contact", + "trello.trello_create_card" + ], + "hit1": "github.create_gist", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 236, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 387, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 824, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "kind": "verb_product", + "gold": [ + "github.search_code" + ], + "alts": [], + "ranking": [ + "github.search_users", + "github.search_orgs", + "github.search_code", + "github.search_issues", + "github.search_commits", + "github.search_repositories", + "github.search_pull_requests", + "brave_search.brave_web_search", + "github.get_code_scanning_alert", + "github.list_code_scanning_alerts" + ], + "hit1": "github.search_users", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "kind": "verb_product", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "ranking": [ + "github.list_notifications", + "github.mark_all_notifications_read", + "github.dismiss_notification", + "github.get_notification_details", + "github.get_gist", + "github.search_orgs", + "github.list_gists", + "github.list_packages", + "github.actions_list", + "github.create_gist" + ], + "hit1": "github.list_notifications", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 175, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "kind": "verb_product", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "ranking": [ + "github.assign_copilot_to_issue", + "jira.jira_assign_issue", + "jira.jira_update_issue", + "gitlab.gitlab_get_issue", + "github.issue_write", + "jira.jira_transition_issue", + "jira.jira_add_comment", + "jira.jira_add_worklog", + "jira.jira_get_issue", + "github.add_issue_comment" + ], + "hit1": "github.assign_copilot_to_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 468, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 852, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "kind": "resource", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "ranking": [ + "github.get_secret_scanning_alert", + "github.list_secret_scanning_alerts", + "github.get_code_scanning_alert", + "github.list_code_scanning_alerts", + "memory.create_entities", + "memory.delete_entities", + "zendesk.zendesk_get_ticket", + "docker.docker_exec", + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_create_ticket" + ], + "hit1": "github.get_secret_scanning_alert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8597186998521972, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 440, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 756, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "ranking": [ + "github.create_or_update_file", + "github.create_gist", + "slack.update_file", + "github.update_gist", + "slack.create_file", + "github.projects_write", + "github.update_artifact", + "github.update_runner", + "github.issue_write", + "github.update_environment" + ], + "hit1": "github.create_or_update_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 314, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 491, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 909, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "kind": "resource", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "ranking": [ + "github.list_repository_collaborators", + "github.list_commits", + "github.list_tags", + "github.list_labels", + "github.list_releases", + "github.list_branches", + "github.list_starred_repositories", + "github.star_repository", + "github.list_issues", + "github.list_secret_scanning_alerts" + ], + "hit1": "github.list_repository_collaborators", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 272, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 413, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 789, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "kind": "verb_product", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "ranking": [ + "slack.slack_reply_to_thread", + "slack.slack_get_thread_replies", + "slack.slack_get_user_profile", + "slack.slack_add_reaction", + "slack.get_pin", + "slack.slack_leave_channel", + "slack.slack_create_channel", + "slack.slack_join_channel", + "slack.update_pin", + "slack.slack_post_message" + ], + "hit1": "slack.slack_reply_to_thread", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "kind": "verb_product", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "ranking": [ + "slack.slack_add_reaction", + "discord.discord_add_reaction", + "slack.slack_reply_to_thread", + "slack.create_canvase", + "slack.create_usergroup", + "slack.create_reminder", + "slack.slack_get_thread_replies", + "slack.create_pin", + "slack.create_file", + "slack.slack_create_channel" + ], + "hit1": "slack.slack_add_reaction", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 289, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 500, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 945, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "kind": "verb_product", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "ranking": [ + "slack.slack_search_messages", + "slack.slack_post_message", + "slack.slack_add_reaction", + "elasticsearch.es_search", + "slack.slack_list_users", + "slack.list_usergroups", + "slack.slack_list_channels", + "slack.slack_get_channel_history", + "notion.notion_search", + "brave_search.brave_web_search" + ], + "hit1": "slack.slack_search_messages", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 248, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "kind": "verb_product", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "ranking": [ + "slack.slack_create_channel", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_post_message", + "slack.slack_get_channel_history", + "slack.slack_set_channel_topic", + "slack.create_usergroup", + "slack.create_pin", + "slack.slack_reply_to_thread", + "slack.slack_add_reaction" + ], + "hit1": "slack.slack_create_channel", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 791, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "kind": "verb_product", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_transition_issue", + "jira.jira_assign_issue", + "jira.jira_create_issue", + "jira.jira_update_issue", + "jira.jira_get_issue", + "jira.jira_add_worklog", + "jira.jira_add_comment", + "jira.jira_search_issues", + "sentry.sentry_update_issue", + "jira.update_version" + ], + "hit1": "jira.jira_transition_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 271, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 772, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "kind": "identifier", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "ranking": [ + "jira.jira_search_issues", + "elasticsearch.es_search", + "jira.jira_list_projects", + "google_drive.gdrive_search", + "zendesk.zendesk_search", + "linear.linear_search_issues", + "brave_search.brave_web_search", + "jira.jira_create_sprint", + "jira.create_sprint", + "jira.jira_add_worklog" + ], + "hit1": "jira.jira_search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 305, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 749, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "kind": "verb_product", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "ranking": [ + "jira.jira_add_worklog", + "jira.jira_add_comment", + "bitbucket.bitbucket_list_repos", + "bitbucket.bitbucket_create_issue", + "jira.list_attachments", + "jira.list_sprints", + "jira.jira_create_sprint", + "jira.jira_list_projects", + "bitbucket.bitbucket_create_pr", + "jira.jira_create_issue" + ], + "hit1": "jira.jira_add_worklog", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 233, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 862, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "kind": "verb_product", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "ranking": [ + "linear.linear_list_teams", + "linear.linear_list_projects", + "linear.linear_list_issues", + "linear.linear_create_issue", + "linear.list_labels", + "linear.get_label", + "linear.list_users", + "linear.linear_create_project", + "github.get_team_members", + "linear.get_user" + ], + "hit1": "linear.linear_list_teams", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "ranking": [ + "linear.linear_create_comment", + "linear.get_comment", + "linear.linear_get_issue", + "github.update_issue_comment", + "linear.linear_search_issues", + "linear.linear_update_issue", + "linear.update_comment", + "linear.list_comments", + "linear.linear_create_issue", + "linear.linear_list_issues" + ], + "hit1": "linear.linear_create_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 824, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "kind": "paraphrase", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "ranking": [ + "filesystem.write_file", + "filesystem.move_file", + "filesystem.read_media_file", + "filesystem.read_file", + "filesystem.read_text_file", + "google_drive.gdrive_share_file", + "google_drive.gdrive_update_file", + "figma.figma_get_file", + "gitlab.create_pipeline", + "slack.create_file" + ], + "hit1": "filesystem.write_file", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 209, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 778, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "kind": "paraphrase", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "ranking": [ + "filesystem.search_files", + "notion.notion_search", + "elasticsearch.es_search", + "filesystem.directory_tree", + "filesystem.list_directory", + "google_drive.gdrive_search", + "git.git_add", + "filesystem.list_directory_with_sizes", + "slack.list_files", + "filesystem.move_file" + ], + "hit1": "filesystem.search_files", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 225, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 649, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "kind": "paraphrase", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "ranking": [ + "filesystem.edit_file", + "google_drive.gdrive_list_folder", + "google_drive.gdrive_create_file", + "github.get_file_blame", + "github.create_or_update_file", + "filesystem.write_file", + "google_drive.gdrive_update_file", + "filesystem.list_directory_with_sizes", + "slack.update_file", + "slack.delete_file" + ], + "hit1": "filesystem.edit_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 282, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 524, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 864, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "ranking": [ + "memory.read_graph", + "memory.open_nodes", + "memory.search_nodes", + "memory.delete_entities", + "memory.create_entities", + "memory.add_observations", + "pagerduty.pagerduty_acknowledge_incident", + "memory.delete_relations", + "gitlab.gitlab_create_issue", + "gitlab.create_snippet" + ], + "hit1": "memory.read_graph", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 162, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 631, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "kind": "verb_product", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "memory.open_nodes", + "elasticsearch.es_search", + "notion.notion_search", + "brave_search.brave_web_search", + "github.search_orgs", + "brave_search.brave_local_search", + "zendesk.zendesk_search", + "jira.jira_search_issues", + "linear.linear_search_issues" + ], + "hit1": "memory.search_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "kind": "verb_product", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "ranking": [ + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_reset", + "git.git_diff", + "git.git_show", + "trello.trello_add_comment", + "git.git_commit", + "git.git_checkout", + "github.unstar_repository", + "git.git_log" + ], + "hit1": "git.git_diff_unstaged", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 174, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 629, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "kind": "identifier", + "gold": [ + "git.git_add" + ], + "alts": [], + "ranking": [ + "filesystem.list_directory_with_sizes", + "git.git_add", + "github.push_files", + "filesystem.read_text_file", + "filesystem.list_directory", + "filesystem.move_file", + "google_drive.gdrive_share_file", + "filesystem.directory_tree", + "filesystem.read_media_file", + "github.delete_file" + ], + "hit1": "filesystem.list_directory_with_sizes", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 400, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "kind": "verb_product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "ranking": [ + "shopify.shopify_create_product", + "shopify.shopify_update_product", + "shopify.shopify_get_product", + "shopify.shopify_create_order", + "shopify.shopify_list_products", + "shopify.shopify_get_order", + "stripe.stripe_create_product", + "shopify.shopify_list_orders", + "trello.trello_create_card", + "vercel.vercel_create_project" + ], + "hit1": "shopify.shopify_create_product", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 719, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "kind": "verb_product", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "ranking": [ + "stripe.stripe_cancel_subscription", + "stripe.stripe_list_subscriptions", + "stripe.stripe_create_product", + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_create_price", + "stripe.stripe_list_customers", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_refund", + "stripe.stripe_create_customer" + ], + "hit1": "stripe.stripe_cancel_subscription", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 203, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 640, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "kind": "resource", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_refund", + "stripe.stripe_get_payment_intent", + "notion.notion_create_page", + "github.create_or_update_file", + "notion.create_comment", + "jira.create_attachment", + "salesforce.salesforce_create_record", + "linear.create_comment", + "linear.create_attachment" + ], + "hit1": "stripe.stripe_create_payment_intent", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 466, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 917, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "github.update_package", + "notion.notion_create_page", + "github.delete_package", + "github.create_package" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "kind": "verb_product", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "ranking": [ + "datadog.datadog_create_monitor", + "datadog.datadog_get_monitor", + "datadog.datadog_list_monitors", + "notion.create_data_source", + "datadog.datadog_list_logs", + "trello.trello_create_board", + "trello.trello_create_card", + "jira.jira_create_sprint", + "datadog.datadog_query_metrics", + "hubspot.hubspot_create_contact" + ], + "hit1": "datadog.datadog_create_monitor", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 461, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 905, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "kind": "paraphrase", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "ranking": [ + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "sentry.sentry_list_events", + "sentry.sentry_list_issues", + "sentry.sentry_list_projects", + "github.issue_write", + "gitlab.gitlab_get_issue", + "jira.jira_update_issue", + "github.update_issue_comment", + "jira.jira_create_issue" + ], + "hit1": "sentry.sentry_get_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 892, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_delete", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_apply", + "vercel.vercel_get_deployment", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_list_pods" + ], + "hit1": "kubernetes.k8s_scale_deployment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 438, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_apply", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_delete", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_list_deployments", + "slack.slack_reply_to_thread" + ], + "hit1": "kubernetes.k8s_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 263, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "ranking": [ + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_exec", + "docker.docker_run", + "docker.docker_ps", + "docker.docker_images", + "docker.docker_build", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_search_contacts" + ], + "hit1": "docker.docker_stop", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 604, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "kind": "verb_product", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "ranking": [ + "docker.docker_build", + "docker.docker_images", + "docker.docker_run", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_ps", + "docker.docker_logs", + "docker.docker_exec", + "discord.discord_send_message", + "memory.read_graph" + ], + "hit1": "docker.docker_build", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 590, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "kind": "verb_product", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "ranking": [ + "playwright.browser_click", + "playwright.browser_type", + "puppeteer.puppeteer_click", + "playwright.browser_take_screenshot", + "notion.delete_comment", + "jira.delete_attachment", + "notion.get_comment", + "linear.delete_attachment", + "notion.list_comments", + "jira.list_versions" + ], + "hit1": "playwright.browser_click", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 646, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "kind": "paraphrase", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "ranking": [ + "playwright.browser_type", + "playwright.browser_navigate", + "playwright.browser_wait_for", + "playwright.browser_tab_new", + "playwright.browser_tab_close", + "playwright.browser_tab_list", + "jira.create_filter", + "jira.create_webhook", + "playwright.browser_evaluate", + "github.create_runner" + ], + "hit1": "playwright.browser_type", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 693, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_list_projects", + "gitlab.gitlab_list_issues", + "github.projects_list", + "gitlab.gitlab_list_merge_requests", + "gitlab.gitlab_get_file", + "gitlab.gitlab_create_issue", + "gitlab.gitlab_get_issue", + "jira.jira_list_projects", + "github.projects_get", + "gitlab.gitlab_create_pipeline" + ], + "hit1": "gitlab.gitlab_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 209, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "kind": "verb_product", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "ranking": [ + "bitbucket.bitbucket_create_pr", + "bitbucket.bitbucket_list_prs", + "bitbucket.bitbucket_create_issue", + "github.request_copilot_review", + "github.create_pull_request", + "github.pull_request_review_write", + "github.update_pull_request", + "github.pull_request_read", + "bitbucket.bitbucket_list_repos", + "zendesk.zendesk_create_ticket" + ], + "hit1": "bitbucket.bitbucket_create_pr", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 564, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1080, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "kind": "verb_product", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "ranking": [ + "gmail.gmail_create_draft", + "gmail.gmail_trash_email", + "github.create_gist", + "hubspot.hubspot_create_contact", + "gmail.gmail_get_email", + "gmail.gmail_send_email", + "stripe.stripe_create_customer", + "gmail.gmail_search_emails", + "vercel.vercel_create_project", + "github.create_pull_request" + ], + "hit1": "gmail.gmail_create_draft", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 374, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 814, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "kind": "verb_product", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_share_file", + "google_drive.gdrive_update_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_list_folder", + "google_drive.gdrive_create_file", + "google_drive.gdrive_search", + "github.delete_file", + "filesystem.write_file", + "github.push_files", + "filesystem.move_file" + ], + "hit1": "google_drive.gdrive_share_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 833, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "kind": "verb_product", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "sentry.sentry_list_events", + "aws_s3.s3_list_objects", + "github.list_commits", + "github.list_dependabot_alerts", + "linear.list_comments" + ], + "hit1": "google_calendar.gcal_list_events", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "github.delete_check_run", + "linear.delete_comment", + "notion.delete_comment", + "linear.delete_attachment", + "jira.delete_attachment" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_projects", + "vercel.vercel_create_project", + "vercel.vercel_list_deployments", + "jira.jira_list_projects", + "vercel.vercel_get_deployment", + "github.projects_write", + "asana.asana_create_project", + "gitlab.gitlab_list_projects", + "github.projects_get", + "sentry.sentry_list_projects" + ], + "hit1": "vercel.vercel_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 175, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 269, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 593, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "ranking": [ + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update", + "supabase.supabase_list_tables", + "supabase.supabase_invoke_fn", + "mongodb.mongo_delete", + "redis.redis_keys", + "figma.figma_get_images", + "memory.delete_entities", + "redis.redis_delete" + ], + "hit1": "supabase.supabase_select", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 659, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "kind": "verb_product", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "figma.figma_get_comments", + "figma.figma_get_file", + "figma.figma_get_file_nodes", + "figma.figma_get_images", + "linear.get_comment", + "linear.update_comment", + "notion.get_comment", + "github.update_issue_comment", + "linear.list_comments" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 715, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "kind": "verb_product", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "ranking": [ + "okta.okta_list_users", + "okta.okta_get_user", + "okta.okta_create_user", + "okta.okta_deactivate_user", + "github.list_gists", + "notion.create_users_token", + "slack.slack_get_users", + "slack.slack_list_users", + "slack.slack_get_user_profile", + "pagerduty.pagerduty_list_oncalls" + ], + "hit1": "okta.okta_list_users", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 284, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 636, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query_range", + "prometheus.prom_query", + "prometheus.prom_list_metrics", + "shopify.shopify_list_products", + "postgres.explain_query", + "shopify.shopify_list_customers", + "datadog.datadog_query_metrics", + "google_drive.gdrive_search", + "shopify.shopify_get_product", + "github.search_repositories" + ], + "hit1": "prometheus.prom_query_range", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 347, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "kind": "verb_product", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "ranking": [ + "grafana.grafana_create_annotation", + "grafana.grafana_get_dashboard", + "gmail.gmail_create_draft", + "clickup.clickup_create_task", + "notion.create_data_source", + "github.create_gist", + "memory.create_relations", + "jira.create_attachment", + "notion.create_comment", + "vercel.vercel_create_project" + ], + "hit1": "grafana.grafana_create_annotation", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 434, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 813, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "kind": "identifier", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "ranking": [ + "terraform.tf_apply", + "terraform.tf_plan", + "terraform.tf_state_list", + "kubernetes.k8s_apply", + "github.update_environment", + "github.list_environments", + "slack.slack_reply_to_thread", + "memory.delete_entities", + "github.update_gist", + "github.list_releases" + ], + "hit1": "terraform.tf_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "kind": "verb_product", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "ranking": [ + "asana.asana_list_tasks", + "asana.asana_get_task", + "asana.asana_update_task", + "asana.asana_create_task", + "clickup.clickup_list_tasks", + "asana.asana_create_project", + "notion.notion_list_users", + "trello.trello_list_cards", + "grafana.grafana_create_annotation", + "kubernetes.k8s_list_deployments" + ], + "hit1": "asana.asana_list_tasks", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 228, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 712, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "ranking": [ + "trello.trello_add_comment", + "trello.trello_create_card", + "trello.trello_list_cards", + "trello.trello_create_board", + "linear.create_comment", + "github.update_issue_comment", + "linear.get_comment", + "figma.figma_post_comment", + "notion.get_comment", + "github.get_discussion_comments" + ], + "hit1": "trello.trello_add_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 787, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_update_task", + "clickup.clickup_get_task", + "clickup.clickup_create_task", + "clickup.clickup_list_tasks", + "confluence.confluence_update_page", + "asana.asana_update_task", + "gitlab.update_wiki", + "github.projects_write", + "notion.update_data_source", + "gitlab.update_snippet" + ], + "hit1": "clickup.clickup_update_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 787, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "kind": "verb_product", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "ranking": [ + "confluence.confluence_search", + "confluence.confluence_get_page", + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "hubspot.hubspot_search_contacts", + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "elasticsearch.es_search", + "github.search_users", + "github.search_commits" + ], + "hit1": "confluence.confluence_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 369, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_deal", + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_search_contacts", + "hubspot.hubspot_list_contacts", + "github.create_artifact", + "github.create_hook", + "github.create_gist", + "confluence.confluence_create_page", + "clickup.clickup_create_task" + ], + "hit1": "hubspot.hubspot_create_deal", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 819, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_search", + "zendesk.zendesk_create_ticket", + "aws_s3.s3_list_buckets", + "notion.list_comments", + "github.list_code_scanning_alerts", + "github.list_secret_scanning_alerts", + "jira.list_sprints" + ], + "hit1": "zendesk.zendesk_list_tickets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 222, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 739, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "kind": "verb_product", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "ranking": [ + "discord.discord_list_channels", + "discord.discord_get_channel", + "discord.discord_send_message", + "discord.discord_add_reaction", + "discord.discord_list_members", + "slack.slack_get_channel_history", + "slack.slack_leave_channel", + "slack.slack_post_message", + "slack.slack_join_channel", + "slack.slack_list_channels" + ], + "hit1": "discord.discord_list_channels", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 635, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "redis.redis_set", + "redis.redis_delete", + "redis.redis_keys", + "filesystem.edit_file", + "github.get_label", + "github.label_write", + "github.get_teams", + "github.search_commits", + "github.get_gist" + ], + "hit1": "redis.redis_get", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert", + "mongodb.mongo_delete", + "mongodb.mongo_find", + "mongodb.mongo_update", + "mongodb.mongo_list_collections", + "elasticsearch.es_index_doc", + "stripe.stripe_create_payment_intent", + "github.create_environment", + "okta.okta_list_users", + "stripe.stripe_create_price" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 779, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "kind": "verb_product", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "ranking": [ + "mysql.mysql_query", + "postgres.query", + "postgres.explain_query", + "sqlite.sqlite_query", + "mysql.mysql_list_tables", + "memory.search_nodes", + "prometheus.prom_query", + "salesforce.salesforce_query", + "linear.linear_assign_issue", + "datadog.datadog_query_metrics" + ], + "hit1": "mysql.mysql_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 625, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "elasticsearch.es_search", + "mongodb.mongo_insert", + "mongodb.mongo_find", + "notion.update_comment", + "slack.list_usergroups", + "github.create_or_update_file", + "notion.create_comment", + "mongodb.mongo_update" + ], + "hit1": "elasticsearch.es_index_doc", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 392, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 869, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_delete_object", + "aws_s3.s3_get_object", + "aws_s3.s3_list_objects", + "aws_s3.s3_put_object", + "github.get_latest_release", + "github.get_label", + "gitlab.get_label", + "github.get_milestone", + "salesforce.salesforce_get_record", + "okta.okta_get_user" + ], + "hit1": "aws_s3.s3_delete_object", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 392, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones", + "cloudflare.cf_list_dns", + "cloudflare.cf_create_dns", + "cloudflare.cf_purge_cache", + "kubernetes.k8s_list_deployments", + "sequential_thinking.sequentialthinking", + "kubernetes.k8s_scale_deployment", + "gitlab.list_milestones", + "notion.notion_list_users", + "prometheus.prom_list_metrics" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 722, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "stripe.stripe_create_refund", + "stripe.stripe_create_customer", + "stripe.stripe_create_payment_intent", + "stripe.stripe_cancel_subscription", + "stripe.stripe_list_subscriptions", + "slack.slack_create_channel" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 676, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "kind": "verb_product", + "gold": [ + "github.actions_list" + ], + "alts": [], + "ranking": [ + "github.actions_list", + "github.actions_get", + "github.actions_run_trigger", + "github.list_discussions", + "github.list_tags", + "github.actions_get_job_logs", + "github.list_gists", + "github.list_commits", + "github.list_branches", + "github.list_labels" + ], + "hit1": "github.actions_list", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 435, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 824, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "kind": "verb_product", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "ranking": [ + "github.actions_get_job_logs", + "github.actions_get", + "github.actions_list", + "github.actions_run_trigger", + "github.list_discussions", + "github.get_label", + "github.search_orgs", + "github.get_runner", + "github.search_commits", + "github.label_write" + ], + "hit1": "github.actions_get_job_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 460, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 815, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "kind": "verb_product", + "gold": [ + "github.create_repository" + ], + "alts": [], + "ranking": [ + "github.create_repository", + "github.delete_repository", + "github.star_repository", + "github.fork_repository", + "github.unstar_repository", + "github.create_branch", + "github.create_or_update_file", + "github.create_issue", + "github.get_tag", + "github.list_repository_collaborators" + ], + "hit1": "github.create_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 928, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "kind": "verb_product", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "ranking": [ + "git.git_checkout", + "git.git_create_branch", + "git.git_branch", + "github.create_branch", + "github.list_branches", + "github.list_check_runs", + "github.delete_check_run", + "git.git_diff", + "github.get_check_run", + "github.list_commits" + ], + "hit1": "git.git_checkout", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "kind": "identifier", + "gold": [ + "git.git_log" + ], + "alts": [], + "ranking": [ + "git.git_log", + "git.git_show", + "git.git_diff", + "git.git_branch", + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_status", + "github.list_gists", + "git.git_checkout", + "github.get_commit" + ], + "hit1": "git.git_log", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 306, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "kind": "verb_product", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "ranking": [ + "postgres.list_schemas", + "postgres.list_tables", + "postgres.query", + "postgres.describe_table", + "postgres.explain_query", + "mysql.mysql_list_tables", + "github.projects_list", + "github.actions_list", + "github.list_secret_scanning_alerts", + "github.list_labels" + ], + "hit1": "postgres.list_schemas", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 158, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "kind": "verb_product", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "ranking": [ + "memory.open_nodes", + "memory.search_nodes", + "memory.read_graph", + "memory.create_entities", + "memory.add_observations", + "memory.create_relations", + "memory.delete_entities", + "memory.delete_relations", + "gitlab.list_jobs", + "gitlab.list_releases" + ], + "hit1": "memory.open_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 267, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 575, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "kind": "paraphrase", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "ranking": [ + "filesystem.read_media_file", + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.get_file_info", + "google_drive.gdrive_get_file", + "figma.figma_get_file", + "figma.figma_get_images", + "filesystem.write_file", + "filesystem.move_file", + "figma.figma_get_file_nodes" + ], + "hit1": "filesystem.read_media_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 410, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 683, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "kind": "paraphrase", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "ranking": [ + "filesystem.write_file", + "filesystem.read_media_file", + "playwright.browser_tab_new", + "github.issue_write", + "filesystem.read_file", + "filesystem.read_text_file", + "github.create_or_update_file", + "google_drive.gdrive_list_folder", + "google_drive.gdrive_share_file", + "jira.jira_create_sprint" + ], + "hit1": "filesystem.write_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 199, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 392, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 875, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "kind": "verb_product", + "gold": [ + "github.search_issues" + ], + "alts": [], + "ranking": [ + "github.search_issues", + "github.search_orgs", + "github.search_users", + "github.search_pull_requests", + "github.issue_read", + "github.issue_write", + "github.assign_copilot_to_issue", + "github.search_commits", + "github.add_issue_comment", + "jira.jira_search_issues" + ], + "hit1": "github.search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 830, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "kind": "paraphrase", + "gold": [ + "github.get_me" + ], + "alts": [], + "ranking": [ + "github.get_teams", + "github.get_me", + "github.list_notifications", + "github.search_users", + "github.get_latest_release", + "github.search_commits", + "github.get_gist", + "github.update_gist", + "github.get_team_members", + "github.list_gists" + ], + "hit1": "github.get_teams", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 603, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "cost": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.0333333333333334, + "p95_calls": 1.0, + "success_within_1": 0.9666666666666667, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 240.9, + "p95_tokens": 340.2999999999999, + "total_tokens": 36135, + "tokens_per_successful_query": 240.9, + "total_tokens_successful": 36135 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.0066666666666666, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 380.88, + "p95_tokens": 509.0, + "total_tokens": 57132, + "tokens_per_successful_query": 380.88, + "total_tokens_successful": 57132 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.0, + "p95_calls": 1.0, + "success_within_1": 1.0, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 756.0866666666667, + "p95_tokens": 941.65, + "total_tokens": 113413, + "tokens_per_successful_query": 756.0866666666667, + "total_tokens_successful": 113413 + } + } +} diff --git a/experiments/mcp-tool-search/results/hybrid_codedb_embed.csv b/experiments/mcp-tool-search/results/hybrid_codedb_embed.csv new file mode 100644 index 00000000..841ddaa8 --- /dev/null +++ b/experiments/mcp-tool-search/results/hybrid_codedb_embed.csv @@ -0,0 +1,151 @@ +id,query,kind,hit1,recall@1,recall@3,recall@5,mrr@10,ndcg@10,hit_rank,k3_calls,k3_tokens,k3_success1,k5_calls,k5_tokens,k5_success1,k10_calls,k10_tokens,k10_success1,ranking +q001,Create a GitHub issue,verb_product,github.create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,380,1.0,1,584,1.0,1,1088,1.0,github.create_issue github.issue_write gitlab.gitlab_create_issue github.add_issue_comment github.issue_read bitbucket.bitbucket_create_issue github.search_issues linear.linear_create_issue github.create_repository jira.jira_create_issue +q002,Create a Jira ticket,verb_product,jira.jira_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,345,1.0,1,540,1.0,1,924,1.0,jira.jira_create_issue jira.jira_create_sprint jira.create_sprint jira.create_webhook jira.create_version jira.create_filter zendesk.zendesk_create_ticket zendesk.zendesk_get_ticket jira.get_sprint zendesk.zendesk_update_ticket +q003,Send a Slack message,verb_product,slack.slack_post_message,1.0,1.0,1.0,1.0000,1.0000,1,1,252,1.0,1,408,1.0,1,778,1.0,slack.slack_post_message slack.slack_add_reaction slack.slack_create_channel discord.discord_send_message slack.slack_reply_to_thread twilio.twilio_send_sms slack.get_reminder gmail.gmail_send_email slack.slack_upload_file slack.slack_leave_channel +q004,List open pull requests,resource,github.list_pull_requests,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,477,1.0,1,769,1.0,github.list_pull_requests github.search_pull_requests bitbucket.bitbucket_list_prs github.create_pull_request github.pull_request_read gitlab.gitlab_list_merge_requests playwright.browser_tab_list gitlab.list_snippets memory.open_nodes bitbucket.bitbucket_list_repos +q005,Open a pull request on GitHub,verb_product,github.create_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,368,1.0,1,564,1.0,1,1144,1.0,github.create_pull_request github.merge_pull_request github.pull_request_read github.pull_request_review_write github.search_pull_requests github.update_pull_request github.list_pull_requests github.update_pull_request_branch github.add_comment_to_pending_review github.request_copilot_review +q006,Run a SQL query against postgres,verb_product,postgres.query,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,295,1.0,1,551,1.0,postgres.query mysql.mysql_query sqlite.sqlite_query postgres.explain_query salesforce.salesforce_query postgres.list_schemas postgres.describe_table postgres.list_tables prometheus.prom_query github.delete_check_run +q007,Read a local file from disk,paraphrase,filesystem.read_media_file,0.0,1.0,1.0,0.5000,0.6199,2,1,223,1.0,1,406,1.0,1,774,1.0,filesystem.read_media_file filesystem.read_text_file filesystem.read_file github.get_file_contents figma.figma_get_file_nodes filesystem.get_file_info gitlab.gitlab_get_file github.delete_file filesystem.list_directory figma.figma_get_file +q008,What files are in this folder,paraphrase,google_drive.gdrive_list_folder,0.0,1.0,1.0,0.5000,0.6863,2,1,191,1.0,1,314,1.0,1,686,1.0,google_drive.gdrive_list_folder filesystem.list_directory filesystem.list_directory_with_sizes filesystem.search_files filesystem.directory_tree slack.list_files github.push_files filesystem.get_file_info cloudflare.cf_purge_cache filesystem.read_file +q009,Remember this fact in the knowledge graph,paraphrase,memory.read_graph,0.0,1.0,1.0,0.3333,0.5271,3,1,164,1.0,1,262,1.0,1,568,1.0,memory.read_graph memory.create_relations memory.add_observations memory.open_nodes memory.create_entities memory.search_nodes memory.delete_entities memory.delete_relations postgres.list_schemas sequential_thinking.sequentialthinking +q010,Search the web with Brave,verb_product,brave_search.brave_web_search,1.0,1.0,1.0,1.0000,1.0000,1,1,245,1.0,1,354,1.0,1,691,1.0,brave_search.brave_web_search brave_search.brave_local_search grafana.grafana_search_dashboards hubspot.hubspot_search_contacts github.search_orgs jira.jira_search_issues memory.search_nodes filesystem.list_directory_with_sizes confluence.confluence_search github.update_pull_request_branch +q011,Fetch the contents of a URL,verb_product,fetch.fetch,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,369,1.0,1,659,1.0,fetch.fetch playwright.browser_navigate github.get_file_contents filesystem.read_file confluence.confluence_search git.git_add puppeteer.puppeteer_navigate playwright.browser_snapshot gitlab.get_snippet slack.slack_set_channel_topic +q012,Show git status,identifier,git.git_status,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,312,1.0,1,675,1.0,git.git_status git.git_show git.git_diff_unstaged git.git_log git.git_diff github.list_milestones git.git_diff_staged github.list_code_scanning_alerts gitlab.list_releases github.get_notification_details +q013,Commit staged changes,paraphrase,git.git_commit,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,327,1.0,1,706,1.0,git.git_commit git.git_reset git.git_diff_staged git.git_show github.get_commit github.update_milestone filesystem.move_file git.git_log github.push_files slack.update_reminder +q014,Create a Linear issue,verb_product,linear.linear_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,275,1.0,1,463,1.0,1,867,1.0,linear.linear_create_issue linear.linear_get_issue linear.linear_create_comment linear.linear_assign_issue linear.linear_update_issue linear.linear_create_project linear.linear_search_issues bitbucket.bitbucket_create_issue gitlab.gitlab_create_issue linear.linear_list_issues +q015,Make a task in my project tracker Linear,paraphrase,linear.linear_create_project,0.0,0.0,1.0,0.2000,0.3869,5,2,502,0.0,1,426,1.0,1,857,1.0,linear.linear_create_project twilio.twilio_make_call linear.linear_list_projects clickup.clickup_get_task linear.linear_create_issue asana.asana_get_task linear.linear_update_issue clickup.clickup_update_task linear.linear_list_issues asana.asana_update_task +q016,Post a comment on a GitHub issue,verb_product,github.update_issue_comment,0.0,1.0,1.0,0.5000,0.6309,2,1,388,1.0,1,628,1.0,1,1039,1.0,github.update_issue_comment github.add_issue_comment github.add_comment_to_pending_review github.assign_copilot_to_issue github.issue_write github.get_discussion_comments github.issue_read figma.figma_post_comment jira.jira_add_comment github.list_repository_collaborators +q017,Merge the pull request,ambiguous,github.merge_pull_request,1.0,1.0,1.0,1.0000,0.9502,1,1,299,1.0,1,495,1.0,1,1093,1.0,github.merge_pull_request github.update_pull_request_branch gitlab.gitlab_merge_merge_request gitlab.gitlab_get_merge_request gitlab.gitlab_create_merge_request github.create_pull_request github.pull_request_read github.update_pull_request gitlab.gitlab_list_merge_requests github.pull_request_review_write +q018,Trigger a GitHub Actions workflow,verb_product,github.actions_run_trigger,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,458,1.0,1,735,1.0,github.actions_run_trigger github.actions_get github.actions_list github.actions_get_job_logs circleci.circleci_trigger_pipeline github.update_check_run github.search_users github.dismiss_notification github.search_orgs github.get_check_run +q019,Get Dependabot alerts,resource,github.list_dependabot_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,464,1.0,1,748,1.0,github.list_dependabot_alerts github.get_dependabot_alert github.list_secret_scanning_alerts github.list_code_scanning_alerts github.get_code_scanning_alert github.get_secret_scanning_alert id_map.json discord.discord_get_channel vercel.vercel_get_deployment pagerduty.pagerduty_list_incidents +q020,List Sentry issues,verb_product,sentry.sentry_list_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,393,1.0,1,809,1.0,sentry.sentry_list_issues sentry.sentry_list_projects sentry.sentry_list_events sentry.sentry_get_issue linear.linear_list_issues sentry.sentry_update_issue gitlab.gitlab_list_issues github.list_issues bitbucket.bitbucket_list_prs pagerduty.pagerduty_list_incidents +q021,Create a Stripe customer,verb_product,stripe.stripe_create_customer,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,341,1.0,1,662,1.0,stripe.stripe_create_customer stripe.stripe_create_invoice stripe.stripe_create_payment_intent stripe.stripe_get_customer stripe.stripe_create_product stripe.stripe_list_invoices stripe.stripe_create_refund stripe.stripe_create_price stripe.stripe_list_subscriptions stripe.stripe_get_payment_intent +q022,Refund a Stripe payment,paraphrase,stripe.stripe_create_refund,1.0,1.0,1.0,1.0000,1.0000,1,1,197,1.0,1,295,1.0,1,644,1.0,stripe.stripe_create_refund stripe.stripe_get_payment_intent stripe.stripe_cancel_subscription stripe.stripe_create_invoice stripe.stripe_get_customer stripe.stripe_create_payment_intent stripe.stripe_list_invoices stripe.stripe_create_product stripe.stripe_list_customers stripe.stripe_create_price +q023,Send an email with Gmail,verb_product,gmail.gmail_send_email,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,399,1.0,1,743,1.0,gmail.gmail_send_email gmail.gmail_trash_email gmail.gmail_get_email gmail.gmail_modify_email gmail.gmail_create_draft gmail.gmail_list_labels gmail.gmail_search_emails twilio.twilio_send_sms discord.discord_send_message google_drive.gdrive_share_file +q024,Search my Gmail inbox,verb_product,gmail.gmail_search_emails,1.0,1.0,1.0,1.0000,1.0000,1,1,178,1.0,1,315,1.0,1,668,1.0,gmail.gmail_search_emails gmail.gmail_list_labels gmail.gmail_get_email gmail.gmail_trash_email gmail.gmail_create_draft gmail.gmail_modify_email hubspot.hubspot_search_contacts gmail.gmail_send_email github.search_users grafana.grafana_search_dashboards +q025,Upload a file to Google Drive,paraphrase,google_drive.gdrive_update_file,0.0,1.0,1.0,0.3333,0.5000,3,1,277,1.0,1,446,1.0,1,806,1.0,google_drive.gdrive_update_file google_drive.gdrive_get_file google_drive.gdrive_create_file google_drive.gdrive_share_file slack.slack_upload_file google_drive.gdrive_search google_drive.gdrive_list_folder filesystem.edit_file aws_s3.s3_put_object filesystem.write_file +q026,Create a calendar event,resource,google_calendar.gcal_create_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,427,1.0,1,801,1.0,google_calendar.gcal_create_event google_calendar.gcal_update_event google_calendar.gcal_delete_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars github.pull_request_review_write sentry.sentry_list_events id_map.json slack.create_reminder trello.trello_create_board +q027,List Kubernetes pods,verb_product,kubernetes.k8s_list_pods,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,314,1.0,1,698,1.0,kubernetes.k8s_list_pods kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_get_pod kubernetes.k8s_apply kubernetes.k8s_pod_logs kubernetes.k8s_scale_deployment docker.docker_images kubernetes.k8s_delete +q028,Get logs from a k8s pod,verb_product,kubernetes.k8s_pod_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,262,1.0,1,408,1.0,1,757,1.0,kubernetes.k8s_pod_logs kubernetes.k8s_get_pod docker.docker_logs github.actions_get_job_logs kubernetes.k8s_get_events datadog.datadog_list_logs kubernetes.k8s_list_pods elasticsearch.es_get_doc kubernetes.k8s_list_services aws_s3.s3_get_object +q029,Run a docker container,verb_product,docker.docker_run,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,331,1.0,1,587,1.0,docker.docker_run docker.docker_exec docker.docker_stop docker.docker_ps docker.docker_logs docker.docker_rm docker.docker_images docker.docker_build github.delete_check_run terraform.tf_plan +q030,Show docker container logs,verb_product,docker.docker_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,226,1.0,1,383,1.0,1,683,1.0,docker.docker_logs kubernetes.k8s_pod_logs docker.docker_stop docker.docker_run docker.docker_exec git.git_log datadog.datadog_list_logs docker.docker_rm docker.docker_images docker.docker_ps +q031,Navigate the browser to a URL,ambiguous,playwright.browser_navigate,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,294,1.0,1,603,1.0,playwright.browser_navigate puppeteer.puppeteer_navigate playwright.browser_tab_new playwright.browser_press_key playwright.browser_click fetch.fetch playwright.browser_take_screenshot playwright.browser_evaluate playwright.browser_snapshot playwright.browser_wait_for +q032,Take a screenshot of the page,ambiguous,playwright.browser_take_screenshot,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,316,1.0,1,667,1.0,playwright.browser_take_screenshot puppeteer.puppeteer_screenshot playwright.browser_snapshot playwright.browser_evaluate notion.notion_get_page pagerduty.pagerduty_get_incident notion.notion_update_page pagerduty.pagerduty_acknowledge_incident confluence.confluence_get_page confluence.confluence_create_page +q033,Create a GitLab merge request,verb_product,gitlab.gitlab_create_merge_request,1.0,1.0,1.0,1.0000,1.0000,1,1,297,1.0,1,475,1.0,1,983,1.0,gitlab.gitlab_create_merge_request gitlab.gitlab_get_merge_request gitlab.gitlab_merge_merge_request gitlab.gitlab_list_merge_requests github.merge_pull_request gitlab.gitlab_create_pipeline gitlab.create_release gitlab.gitlab_create_issue gitlab.create_pipeline github.create_pull_request +q034,Search Notion pages,verb_product,notion.notion_search,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,320,1.0,1,640,1.0,notion.notion_search notion.notion_get_page notion.notion_list_users notion.notion_get_block notion.notion_create_page notion.notion_query_database grafana.grafana_search_dashboards notion.notion_update_page github.search_users hubspot.hubspot_search_contacts +q035,Create a Notion page,verb_product,notion.notion_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,230,1.0,1,419,1.0,1,834,1.0,notion.notion_create_page notion.notion_get_page notion.notion_update_page notion.notion_create_database notion.create_data_source notion.create_comment notion.notion_get_comments notion.create_users_token confluence.confluence_create_page notion.notion_search +q036,Query a Notion database,verb_product,notion.notion_query_database,1.0,1.0,1.0,1.0000,1.0000,1,1,261,1.0,1,386,1.0,1,656,1.0,notion.notion_query_database notion.notion_create_database notion.notion_search notion.notion_create_page notion.notion_list_users notion.notion_get_page notion.notion_get_block notion.get_data_source postgres.query notion.notion_get_comments +q037,Create a Zendesk ticket,verb_product,zendesk.zendesk_create_ticket,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,426,1.0,1,770,1.0,zendesk.zendesk_create_ticket zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_list_tickets jira.jira_create_issue zendesk.zendesk_search id_map.json jira.jira_create_sprint trello.trello_create_board google_calendar.gcal_create_event +q038,Create a HubSpot contact,verb_product,hubspot.hubspot_create_contact,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,364,1.0,1,781,1.0,hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_search_contacts hubspot.hubspot_create_deal hubspot.hubspot_list_contacts github.create_gist slack.create_usergroup github.create_repository slack.get_usergroup github.create_branch +q039,Create a Salesforce lead,verb_product,salesforce.salesforce_create_lead,1.0,1.0,1.0,1.0000,1.0000,1,1,252,1.0,1,380,1.0,1,796,1.0,salesforce.salesforce_create_lead salesforce.salesforce_convert_lead salesforce.salesforce_create_record salesforce.salesforce_list_opportunities salesforce.salesforce_get_record salesforce.salesforce_delete_record slack.create_pin salesforce.salesforce_query linear.create_user salesforce.salesforce_update_record +q040,Run a SOQL query,identifier,salesforce.salesforce_query,1.0,1.0,1.0,1.0000,1.0000,1,1,187,1.0,1,293,1.0,1,622,1.0,salesforce.salesforce_query postgres.query mysql.mysql_query sqlite.sqlite_query github.delete_check_run prometheus.prom_query github.get_check_run postgres.explain_query github.update_check_run prometheus.prom_query_range +q041,List Shopify orders,verb_product,shopify.shopify_list_orders,1.0,1.0,1.0,1.0000,1.0000,1,1,193,1.0,1,333,1.0,1,618,1.0,shopify.shopify_list_orders shopify.shopify_list_customers shopify.shopify_get_order shopify.shopify_list_products shopify.shopify_create_order shopify.shopify_get_product shopify.shopify_update_product shopify.shopify_create_product id_map.json stripe.stripe_list_subscriptions +q042,Create a PagerDuty incident,verb_product,pagerduty.pagerduty_create_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,403,1.0,1,645,1.0,pagerduty.pagerduty_create_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls id_map.json sentry.sentry_list_events trello.trello_create_board playwright.browser_press_key +q043,Who is on call in PagerDuty,paraphrase,pagerduty.pagerduty_list_oncalls,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,389,1.0,1,718,1.0,pagerduty.pagerduty_list_oncalls pagerduty.pagerduty_get_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents twilio.twilio_make_call slack.slack_create_channel playwright.browser_press_key filesystem.list_allowed_directories +q044,Query Datadog metrics,verb_product,datadog.datadog_query_metrics,1.0,1.0,1.0,1.0000,1.0000,1,1,284,1.0,1,458,1.0,1,739,1.0,datadog.datadog_query_metrics datadog.datadog_list_monitors datadog.datadog_list_logs datadog.datadog_create_monitor datadog.datadog_get_monitor prometheus.prom_list_metrics memory.search_nodes postgres.query slack.slack_search_messages prometheus.prom_query +q045,Search Datadog logs,verb_product,datadog.datadog_list_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,458,1.0,1,778,1.0,datadog.datadog_list_logs datadog.datadog_list_monitors datadog.datadog_get_monitor datadog.datadog_create_monitor datadog.datadog_query_metrics github.search_users docker.docker_logs github.search_pull_requests kubernetes.k8s_pod_logs memory.search_nodes +q046,List S3 buckets,verb_product,aws_s3.s3_list_buckets,1.0,1.0,1.0,1.0000,1.0000,1,1,203,1.0,1,361,1.0,1,577,1.0,aws_s3.s3_list_buckets aws_s3.s3_list_objects aws_s3.s3_delete_object aws_s3.s3_get_object aws_s3.s3_put_object jira.jira_list_projects bitbucket.bitbucket_list_repos clickup.clickup_list_tasks bitbucket.bitbucket_list_prs id_map.json +q047,Upload an object to S3,paraphrase,aws_s3.s3_put_object,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,405,1.0,1,795,1.0,aws_s3.s3_put_object aws_s3.s3_get_object aws_s3.s3_delete_object aws_s3.s3_list_objects slack.slack_upload_file twilio.twilio_send_sms aws_s3.s3_list_buckets github.assign_copilot_to_issue gmail.gmail_send_email jira.update_attachment +q048,Purge Cloudflare cache,verb_product,cloudflare.cf_purge_cache,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,341,1.0,1,567,1.0,cloudflare.cf_purge_cache cloudflare.cf_list_zones cloudflare.cf_list_dns cloudflare.cf_create_dns slack.delete_bookmark slack.delete_canvase id_map.json memory.delete_observations redis.redis_delete aws_s3.s3_delete_object +q049,Create a Cloudflare DNS record,verb_product,cloudflare.cf_create_dns,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,356,1.0,1,808,1.0,cloudflare.cf_create_dns cloudflare.cf_list_dns salesforce.salesforce_create_record cloudflare.cf_purge_cache cloudflare.cf_list_zones salesforce.salesforce_get_record salesforce.salesforce_delete_record datadog.datadog_create_monitor salesforce.salesforce_update_record slack.create_file +q050,List Vercel deployments,verb_product,vercel.vercel_list_deployments,1.0,1.0,1.0,1.0000,1.0000,1,1,195,1.0,1,291,1.0,1,531,1.0,vercel.vercel_list_deployments kubernetes.k8s_list_deployments vercel.vercel_get_deployment vercel.vercel_list_projects vercel.vercel_create_project id_map.json github.list_packages clickup.clickup_list_tasks gitlab.list_releases jira.jira_list_projects +q051,Insert a row into Supabase,verb_product,supabase.supabase_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,245,1.0,1,352,1.0,1,636,1.0,supabase.supabase_insert mongodb.mongo_insert supabase.supabase_update supabase.supabase_list_tables supabase.supabase_select supabase.supabase_invoke_fn playwright.browser_type linear.update_cycle id_map.json linear.update_attachment +q052,Export images from Figma,paraphrase,figma.figma_get_images,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,328,1.0,1,641,1.0,figma.figma_get_images figma.figma_get_file_nodes figma.figma_get_file docker.docker_images figma.figma_get_comments figma.figma_post_comment memory.delete_observations linear.update_attachment elasticsearch.es_get_doc linear.get_attachment +q053,Send an SMS with Twilio,verb_product,twilio.twilio_send_sms,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,409,1.0,1,771,1.0,twilio.twilio_send_sms twilio.twilio_list_messages twilio.twilio_make_call discord.discord_send_message gmail.gmail_send_email slack.slack_reply_to_thread slack.slack_add_reaction filesystem.list_directory_with_sizes puppeteer.puppeteer_click slack.slack_post_message +q054,Create an Okta user,verb_product,okta.okta_create_user,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,329,1.0,1,795,1.0,okta.okta_create_user okta.okta_get_user okta.okta_deactivate_user okta.okta_list_users linear.create_user asana.asana_create_project asana.asana_create_task notion.create_users_token github.create_issue slack.slack_get_user_profile +q055,Instant query Prometheus,verb_product,prometheus.prom_query,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,375,1.0,1,650,1.0,prometheus.prom_query prometheus.prom_query_range prometheus.prom_list_metrics datadog.datadog_query_metrics postgres.query memory.search_nodes postgres.explain_query github.search_pull_requests mysql.mysql_query hubspot.hubspot_search_contacts +q056,Search Grafana dashboards,verb_product,grafana.grafana_search_dashboards,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,319,1.0,1,627,1.0,grafana.grafana_search_dashboards grafana.grafana_get_dashboard grafana.grafana_create_annotation github.search_users github.search_orgs github.search_pull_requests hubspot.hubspot_search_contacts confluence.confluence_search github.search_repositories github.search_commits +q057,Trigger a CircleCI pipeline,verb_product,circleci.circleci_trigger_pipeline,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,390,1.0,1,737,1.0,circleci.circleci_trigger_pipeline circleci.circleci_get_pipeline circleci.circleci_list_pipelines github.actions_run_trigger gitlab.gitlab_create_pipeline gitlab.update_pipeline gitlab.get_pipeline gitlab.list_pipelines hubspot.hubspot_create_deal gitlab.delete_pipeline +q058,Run terraform plan,identifier,terraform.tf_plan,1.0,1.0,1.0,1.0000,1.0000,1,1,172,1.0,1,295,1.0,1,699,1.0,terraform.tf_plan terraform.tf_apply terraform.tf_state_list github.update_check_run github.get_check_run docker.docker_run github.create_check_run github.delete_check_run hubspot.hubspot_create_deal github.update_runner +q059,Create an Asana task,verb_product,asana.asana_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,405,1.0,1,795,1.0,asana.asana_create_task asana.asana_get_task asana.asana_create_project asana.asana_update_task asana.asana_list_tasks clickup.clickup_create_task okta.okta_create_user clickup.clickup_get_task slack.create_reminder clickup.clickup_update_task +q060,Create a Trello card,verb_product,trello.trello_create_card,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,364,1.0,1,627,1.0,trello.trello_create_card trello.trello_create_board trello.trello_add_comment trello.trello_list_cards stripe.stripe_create_invoice id_map.json grafana.grafana_create_annotation vercel.vercel_create_project stripe.stripe_create_customer okta.okta_create_user +q061,Create a ClickUp task,verb_product,clickup.clickup_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,364,1.0,1,742,1.0,clickup.clickup_create_task clickup.clickup_get_task clickup.clickup_list_tasks clickup.clickup_update_task puppeteer.puppeteer_click asana.asana_create_task playwright.browser_click asana.asana_get_task asana.asana_update_task puppeteer.puppeteer_fill +q062,Create a Confluence page,verb_product,confluence.confluence_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,401,1.0,1,726,1.0,confluence.confluence_create_page confluence.confluence_get_page confluence.confluence_update_page notion.notion_create_page confluence.confluence_search notion.notion_get_page github.create_artifact notion.notion_update_page notion.create_users_token id_map.json +q063,Send a Discord message,verb_product,discord.discord_send_message,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,381,1.0,1,758,1.0,discord.discord_send_message discord.discord_add_reaction discord.discord_get_channel slack.slack_post_message twilio.twilio_send_sms discord.discord_list_channels gmail.gmail_trash_email gmail.gmail_send_email slack.slack_add_reaction slack.slack_reply_to_thread +q064,Search Elasticsearch,verb_product,elasticsearch.es_search,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,324,1.0,1,617,1.0,elasticsearch.es_search elasticsearch.es_get_doc hubspot.hubspot_search_contacts confluence.confluence_search grafana.grafana_search_dashboards github.search_users github.search_orgs github.search_pull_requests elasticsearch.es_index_doc github.search_commits +q065,Get a Redis key,verb_product,redis.redis_get,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,283,1.0,1,549,1.0,redis.redis_get redis.redis_set redis.redis_delete redis.redis_keys figma.figma_get_file jira.jira_get_issue playwright.browser_press_key slack.get_pin figma.figma_get_comments circleci.circleci_get_pipeline +q066,Find MongoDB documents,verb_product,mongodb.mongo_find,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,381,1.0,1,672,1.0,mongodb.mongo_find mongodb.mongo_delete mongodb.mongo_insert mongodb.mongo_list_collections mongodb.mongo_update id_map.json elasticsearch.es_get_doc filesystem.search_files confluence.confluence_search datadog.datadog_list_logs +q067,List SQLite tables,verb_product,sqlite.sqlite_list_tables,1.0,1.0,1.0,1.0000,1.0000,1,1,142,1.0,1,240,1.0,1,433,1.0,sqlite.sqlite_list_tables mysql.mysql_list_tables supabase.supabase_list_tables sqlite.sqlite_describe_table postgres.list_tables sqlite.sqlite_query okta.okta_list_users id_map.json postgres.list_schemas clickup.clickup_list_tasks +q068,Describe a postgres table,verb_product,postgres.describe_table,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,274,1.0,1,590,1.0,postgres.describe_table sqlite.sqlite_describe_table postgres.explain_query postgres.list_schemas postgres.list_tables postgres.query supabase.supabase_insert supabase.supabase_select mysql.mysql_list_tables supabase.supabase_update +q069,github_create_issue,identifier,github.create_repository,0.0,1.0,1.0,0.5000,0.6697,2,1,367,1.0,1,536,1.0,1,1012,1.0,github.create_repository github.create_issue github.issue_write github.create_gist gitlab.gitlab_create_issue github.create_environment github.create_package github.create_artifact bitbucket.bitbucket_create_issue github.create_runner +q070,createIssue,identifier,linear.create_user,0.0,0.0,0.0,0.0000,0.0000,39,13,3554,0.0,8,3600,0.0,4,3528,0.0,linear.create_user gmail.gmail_create_draft notion.create_data_source linear.create_cycle notion.create_comment linear.create_comment linear.create_attachment notion.create_users_token stripe.stripe_create_product okta.okta_create_user +q071,list-directory,identifier,filesystem.list_directory,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,271,1.0,1,489,1.0,filesystem.list_directory filesystem.list_directory_with_sizes okta.okta_list_users mongodb.mongo_list_collections filesystem.list_allowed_directories clickup.clickup_list_tasks terraform.tf_state_list sentry.sentry_list_projects docker.docker_images gmail.gmail_list_labels +q072,sequential thinking next thought,paraphrase,sequential_thinking.sequentialthinking,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,296,1.0,1,579,1.0,sequential_thinking.sequentialthinking id_map.json memory.create_relations memory.add_observations memory.search_nodes memory.create_entities memory.delete_relations notion.update_comment notion.notion_search memory.open_nodes +q073,What time is it in Tokyo,paraphrase,time.get_current_time,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,391,1.0,1,757,1.0,time.get_current_time time.convert_time google_calendar.gcal_list_events grafana.grafana_create_annotation playwright.browser_wait_for zendesk.zendesk_list_tickets slack.slack_create_channel filesystem.list_allowed_directories jira.jira_create_sprint jira.jira_add_worklog +q074,Convert time between timezones,verb_product,time.convert_time,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,390,1.0,1,698,1.0,time.convert_time time.get_current_time grafana.grafana_create_annotation playwright.browser_wait_for google_calendar.gcal_list_events salesforce.salesforce_convert_lead memory.create_relations git.git_diff datadog.datadog_query_metrics cloudflare.cf_list_zones +q075,Star a GitHub repo,verb_product,github.star_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,226,1.0,1,348,1.0,1,838,1.0,github.star_repository github.unstar_repository github.delete_repository github.list_starred_repositories github.actions_list github.create_or_update_file github.get_label github.create_gist github.get_discussion github.get_file_blame +q076,Fork a repository,resource,github.fork_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,386,1.0,1,883,1.0,github.fork_repository github.unstar_repository github.star_repository github.delete_repository github.create_repository github.get_repository_tree github.merge_pull_request github.list_repository_collaborators bitbucket.bitbucket_create_pr github.get_commit +q077,Get the latest GitHub release,verb_product,github.get_latest_release,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,329,1.0,1,669,1.0,github.get_latest_release github.get_release_by_tag gitlab.get_release github.get_gist github.get_teams github.get_milestone github.get_me github.list_releases github.get_repository_tree github.get_notification_details +q078,Create a gist,resource,github.create_gist,1.0,1.0,1.0,1.0000,1.0000,1,1,229,1.0,1,389,1.0,1,703,1.0,github.create_gist github.get_gist github.update_gist trello.trello_create_board gitlab.create_snippet stripe.stripe_create_invoice github.list_gists vercel.vercel_create_project github.create_milestone okta.okta_create_user +q079,Search GitHub code,verb_product,github.search_code,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,340,1.0,1,722,1.0,github.search_code github.search_orgs github.search_users github.get_code_scanning_alert github.search_pull_requests github.list_code_scanning_alerts github.search_commits github.search_repositories github.search_issues github.get_check_run +q080,List GitHub notifications,verb_product,github.list_notifications,1.0,1.0,1.0,1.0000,1.0000,1,1,174,1.0,1,307,1.0,1,671,1.0,github.list_notifications github.mark_all_notifications_read github.list_gists github.list_secret_scanning_alerts github.get_notification_details github.projects_list github.list_code_scanning_alerts github.actions_list github.list_starred_repositories github.list_dependabot_alerts +q081,Assign Copilot to an issue,verb_product,github.assign_copilot_to_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,500,1.0,1,873,1.0,github.assign_copilot_to_issue linear.linear_assign_issue jira.jira_assign_issue github.request_copilot_review github.create_issue linear.linear_create_comment jira.jira_update_issue jira.jira_transition_issue pagerduty.pagerduty_resolve_incident github.add_issue_comment +q082,Get secret scanning alerts,resource,github.list_secret_scanning_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,295,1.0,1,470,1.0,1,723,1.0,github.list_secret_scanning_alerts github.get_secret_scanning_alert github.list_code_scanning_alerts github.get_code_scanning_alert github.list_dependabot_alerts github.get_dependabot_alert id_map.json discord.discord_get_channel github.get_notification_details vercel.vercel_get_deployment +q083,Create or update a file on GitHub,verb_product,github.create_or_update_file,1.0,1.0,1.0,1.0000,0.7602,1,1,320,1.0,1,547,1.0,1,975,1.0,github.create_or_update_file github.projects_write slack.update_file github.issue_write google_drive.gdrive_update_file github.update_gist github.create_gist slack.create_file google_drive.gdrive_create_file github.update_environment +q084,List collaborators on a repo,resource,github.list_repository_collaborators,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,379,1.0,1,745,1.0,github.list_repository_collaborators github.list_labels github.list_releases github.actions_list github.list_discussions github.list_commits pagerduty.pagerduty_list_oncalls github.list_starred_repositories trello.trello_list_cards github.list_tags +q085,Reply in a Slack thread,verb_product,slack.slack_reply_to_thread,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,399,1.0,1,708,1.0,slack.slack_reply_to_thread slack.slack_get_thread_replies slack.get_reminder slack.slack_list_channels slack.slack_add_reaction slack.delete_file slack.slack_post_message slack.get_canvase slack.delete_canvase slack.update_reminder +q086,Add an emoji reaction on Slack,verb_product,slack.slack_add_reaction,1.0,1.0,1.0,1.0000,1.0000,1,1,271,1.0,1,454,1.0,1,833,1.0,slack.slack_add_reaction discord.discord_add_reaction trello.trello_add_comment slack.slack_reply_to_thread gmail.gmail_modify_email slack.slack_get_thread_replies jira.jira_add_comment slack.update_canvase jira.jira_add_worklog slack.update_reminder +q087,Search Slack messages,verb_product,slack.slack_search_messages,1.0,1.0,1.0,1.0000,1.0000,1,1,232,1.0,1,374,1.0,1,668,1.0,slack.slack_search_messages slack.slack_get_channel_history gmail.gmail_search_emails slack.slack_add_reaction twilio.twilio_list_messages github.search_pull_requests grafana.grafana_search_dashboards slack.slack_get_thread_replies github.search_users slack.get_reminder +q088,Create a Slack channel,verb_product,slack.slack_create_channel,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,339,1.0,1,751,1.0,slack.slack_create_channel slack.slack_leave_channel slack.slack_join_channel slack.slack_post_message slack.slack_set_channel_topic slack.slack_reply_to_thread slack.slack_get_thread_replies slack.slack_add_reaction slack.slack_upload_file slack.slack_get_channel_history +q089,Transition a Jira issue,verb_product,jira.jira_transition_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,222,1.0,1,342,1.0,1,760,1.0,jira.jira_transition_issue jira.jira_assign_issue jira.jira_update_issue jira.jira_get_issue jira.jira_add_comment jira.jira_create_issue jira.jira_add_worklog jira.update_sprint gitlab.gitlab_get_issue jira.jira_search_issues +q090,Search Jira with JQL,identifier,jira.jira_search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,203,1.0,1,357,1.0,1,633,1.0,jira.jira_search_issues filesystem.list_directory_with_sizes jira.get_filter grafana.grafana_search_dashboards jira.create_filter github.search_users jira.jira_get_issue hubspot.hubspot_search_contacts jira.update_filter github.search_orgs +q091,Add a worklog to Jira,verb_product,jira.jira_add_worklog,1.0,1.0,1.0,1.0000,1.0000,1,1,214,1.0,1,483,1.0,1,892,1.0,jira.jira_add_worklog jira.jira_add_comment jira.jira_list_projects github.add_comment_to_pending_review jira.jira_create_issue discord.discord_add_reaction memory.add_observations jira.create_webhook git.git_add jira.create_sprint +q092,List Linear teams,verb_product,linear.linear_list_teams,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,330,1.0,1,732,1.0,linear.linear_list_teams linear.linear_list_projects linear.linear_list_issues linear.list_cycles linear.list_users github.get_teams linear.linear_create_issue linear.list_comments linear.linear_create_project linear.list_attachments +q093,Comment on a Linear issue,verb_product,linear.linear_create_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,464,1.0,1,845,1.0,linear.linear_create_comment linear.linear_get_issue linear.linear_assign_issue linear.linear_update_issue linear.linear_create_issue github.update_issue_comment github.add_issue_comment linear.linear_search_issues linear.get_comment linear.update_comment +q094,Move a local file,paraphrase,filesystem.move_file,1.0,1.0,1.0,1.0000,1.0000,1,1,225,1.0,1,352,1.0,1,694,1.0,filesystem.move_file filesystem.read_media_file filesystem.read_text_file filesystem.read_file filesystem.write_file brave_search.brave_local_search figma.figma_get_file filesystem.edit_file slack.delete_file slack.update_file +q095,Recursively search files by pattern,paraphrase,filesystem.search_files,1.0,1.0,1.0,1.0000,1.0000,1,1,192,1.0,1,302,1.0,1,695,1.0,filesystem.search_files google_drive.gdrive_search redis.redis_keys filesystem.directory_tree linear.linear_search_issues memory.search_nodes github.search_repositories slack.list_files github.push_files filesystem.get_file_info +q096,Edit a file in place with old and new string,paraphrase,filesystem.edit_file,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,413,1.0,1,693,1.0,filesystem.edit_file filesystem.move_file filesystem.write_file slack.update_file slack.create_file filesystem.list_directory_with_sizes playwright.browser_tab_new linear.update_attachment slack.delete_file slack.get_file +q097,Read the whole knowledge graph,paraphrase,memory.read_graph,1.0,1.0,1.0,1.0000,1.0000,1,1,154,1.0,1,262,1.0,1,533,1.0,memory.read_graph memory.delete_relations memory.open_nodes memory.add_observations memory.delete_entities memory.create_entities memory.create_relations filesystem.read_media_file filesystem.read_file memory.search_nodes +q098,Search memory nodes,verb_product,memory.search_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,160,1.0,1,288,1.0,1,568,1.0,memory.search_nodes memory.open_nodes memory.read_graph memory.delete_entities figma.figma_get_file_nodes filesystem.search_files hubspot.hubspot_search_contacts memory.delete_observations memory.delete_relations github.search_orgs +q099,Show unstaged git diff,verb_product,git.git_diff_unstaged,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,313,1.0,1,523,1.0,git.git_diff_unstaged git.git_diff_staged git.git_diff git.git_show git.git_log git.git_status git.git_reset id_map.json git.git_branch git.git_commit +q100,Stage files with git add,identifier,git.git_add,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,354,1.0,1,840,1.0,git.git_add github.update_gist github.create_gist filesystem.list_directory_with_sizes git.git_diff_staged github.add_issue_comment git.git_commit github.add_comment_to_pending_review github.push_files git.git_reset +q101,Create a Shopify product,verb_product,shopify.shopify_create_product,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,377,1.0,1,702,1.0,shopify.shopify_create_product shopify.shopify_get_product shopify.shopify_update_product shopify.shopify_create_order stripe.stripe_create_product shopify.shopify_get_order shopify.shopify_list_products shopify.shopify_list_customers shopify.shopify_list_orders stripe.stripe_create_price +q102,Cancel a Stripe subscription,verb_product,stripe.stripe_cancel_subscription,1.0,1.0,1.0,1.0000,1.0000,1,1,176,1.0,1,290,1.0,1,610,1.0,stripe.stripe_cancel_subscription stripe.stripe_get_payment_intent stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_list_subscriptions stripe.stripe_create_refund stripe.stripe_get_customer stripe.stripe_list_customers stripe.stripe_create_customer stripe.stripe_create_product +q103,Create a payment intent,resource,stripe.stripe_create_payment_intent,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,282,1.0,1,636,1.0,stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent stripe.stripe_create_refund stripe.stripe_create_invoice id_map.json trello.trello_create_board stripe.stripe_create_price hubspot.hubspot_create_deal vercel.vercel_create_project stripe.stripe_create_product +q104,Acknowledge a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,405,1.0,1,649,1.0,pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls id_map.json sentry.sentry_list_events slack.get_bookmark slack.get_reminder +q105,Create a Datadog monitor,verb_product,datadog.datadog_create_monitor,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,459,1.0,1,700,1.0,datadog.datadog_create_monitor datadog.datadog_get_monitor datadog.datadog_list_monitors datadog.datadog_list_logs datadog.datadog_query_metrics id_map.json memory.add_observations trello.trello_create_board notion.list_data_sources stripe.stripe_create_invoice +q106,Resolve a Sentry issue,paraphrase,sentry.sentry_get_issue,0.0,1.0,1.0,0.5000,0.6309,2,1,274,1.0,1,412,1.0,1,759,1.0,sentry.sentry_get_issue sentry.sentry_update_issue sentry.sentry_list_issues sentry.sentry_list_events pagerduty.pagerduty_resolve_incident sentry.sentry_list_projects jira.jira_assign_issue jira.jira_get_issue gitlab.gitlab_get_issue github.assign_copilot_to_issue +q107,Scale a Kubernetes deployment,verb_product,kubernetes.k8s_scale_deployment,1.0,1.0,1.0,1.0000,1.0000,1,1,226,1.0,1,353,1.0,1,726,1.0,kubernetes.k8s_scale_deployment kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_apply vercel.vercel_get_deployment kubernetes.k8s_get_pod kubernetes.k8s_get_events kubernetes.k8s_pod_logs kubernetes.k8s_list_pods kubernetes.k8s_delete +q108,Apply a Kubernetes manifest,verb_product,kubernetes.k8s_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,195,1.0,1,334,1.0,1,727,1.0,kubernetes.k8s_apply kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_get_pod terraform.tf_apply kubernetes.k8s_get_events kubernetes.k8s_list_pods kubernetes.k8s_scale_deployment kubernetes.k8s_pod_logs kubernetes.k8s_delete +q109,Stop a docker container,verb_product,docker.docker_stop,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,345,1.0,1,664,1.0,docker.docker_stop docker.docker_rm docker.docker_logs docker.docker_run docker.docker_exec docker.docker_images docker.docker_ps docker.docker_build github.list_environments kubernetes.k8s_pod_logs +q110,Build a docker image,verb_product,docker.docker_build,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,303,1.0,1,642,1.0,docker.docker_build docker.docker_run docker.docker_images docker.docker_stop docker.docker_ps docker.docker_logs docker.docker_exec docker.docker_rm github.create_environment filesystem.read_media_file +q111,Click an element in Playwright,verb_product,playwright.browser_click,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,326,1.0,1,566,1.0,playwright.browser_click playwright.browser_type puppeteer.puppeteer_click playwright.browser_take_screenshot playwright.browser_tab_list playwright.browser_tab_new playwright.browser_tab_close playwright.browser_press_key playwright.browser_navigate playwright.browser_snapshot +q112,Type text into a browser input,paraphrase,playwright.browser_type,1.0,1.0,1.0,1.0000,0.8801,1,1,215,1.0,1,316,1.0,1,636,1.0,playwright.browser_type playwright.browser_wait_for playwright.browser_fill_form playwright.browser_tab_new playwright.browser_navigate playwright.browser_press_key playwright.browser_evaluate puppeteer.puppeteer_fill filesystem.read_text_file playwright.browser_click +q113,List GitLab projects,verb_product,gitlab.gitlab_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,320,1.0,1,641,1.0,gitlab.gitlab_list_projects gitlab.gitlab_list_issues gitlab.gitlab_list_merge_requests vercel.vercel_list_projects github.projects_list gitlab.list_pipelines gitlab.list_jobs jira.jira_list_projects gitlab.list_snippets sentry.sentry_list_projects +q114,Create a Bitbucket pull request,verb_product,bitbucket.bitbucket_create_pr,1.0,1.0,1.0,1.0000,1.0000,1,1,439,1.0,1,606,1.0,1,1091,1.0,bitbucket.bitbucket_create_pr github.create_pull_request github.pull_request_review_write bitbucket.bitbucket_list_prs bitbucket.bitbucket_create_issue github.pull_request_read github.merge_pull_request github.update_pull_request_branch github.update_pull_request github.search_pull_requests +q115,Create a Gmail draft,verb_product,gmail.gmail_create_draft,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,338,1.0,1,805,1.0,gmail.gmail_create_draft gmail.gmail_list_labels gmail.gmail_get_email gmail.gmail_modify_email gmail.gmail_trash_email gmail.gmail_send_email github.create_pull_request gmail.gmail_search_emails hubspot.hubspot_create_contact stripe.stripe_create_customer +q116,Share a Google Drive file,verb_product,google_drive.gdrive_share_file,1.0,1.0,1.0,1.0000,1.0000,1,1,271,1.0,1,417,1.0,1,702,1.0,google_drive.gdrive_share_file google_drive.gdrive_get_file google_drive.gdrive_create_file google_drive.gdrive_update_file google_drive.gdrive_search google_drive.gdrive_list_folder google_calendar.gcal_list_calendars filesystem.move_file google_calendar.gcal_delete_event filesystem.read_media_file +q117,List Google Calendar events,verb_product,google_calendar.gcal_list_events,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,427,1.0,1,702,1.0,google_calendar.gcal_list_events google_calendar.gcal_list_calendars google_calendar.gcal_create_event google_calendar.gcal_delete_event google_calendar.gcal_update_event sentry.sentry_list_events kubernetes.k8s_get_events pagerduty.pagerduty_list_oncalls gmail.gmail_list_labels google_drive.gdrive_list_folder +q118,Delete a calendar event,resource,google_calendar.gcal_delete_event,1.0,1.0,1.0,1.0000,1.0000,1,1,263,1.0,1,406,1.0,1,704,1.0,google_calendar.gcal_delete_event google_calendar.gcal_create_event linear.delete_label google_calendar.gcal_update_event slack.delete_reminder slack.delete_canvase google_calendar.gcal_list_events github.delete_milestone google_calendar.gcal_list_calendars gitlab.delete_milestone +q119,List Vercel projects,verb_product,vercel.vercel_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,151,1.0,1,253,1.0,1,550,1.0,vercel.vercel_list_projects vercel.vercel_list_deployments jira.jira_list_projects sentry.sentry_list_projects gitlab.gitlab_list_projects vercel.vercel_create_project linear.linear_list_projects vercel.vercel_get_deployment github.projects_list asana.asana_list_tasks +q120,Select rows from Supabase,verb_product,supabase.supabase_select,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,354,1.0,1,689,1.0,supabase.supabase_select supabase.supabase_insert supabase.supabase_update supabase.supabase_list_tables supabase.supabase_invoke_fn memory.delete_observations sqlite.sqlite_query elasticsearch.es_get_doc salesforce.salesforce_query datadog.datadog_list_logs +q121,Post a Figma comment,verb_product,figma.figma_post_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,308,1.0,1,688,1.0,figma.figma_post_comment figma.figma_get_comments figma.figma_get_file linear.get_comment notion.get_comment slack.slack_post_message figma.figma_get_file_nodes linear.create_comment figma.figma_get_images linear.delete_comment +q122,List Okta users,verb_product,okta.okta_list_users,1.0,1.0,1.0,1.0000,1.0000,1,1,161,1.0,1,295,1.0,1,634,1.0,okta.okta_list_users notion.notion_list_users okta.okta_create_user okta.okta_get_user notion.list_users_tokens okta.okta_deactivate_user slack.slack_list_users linear.list_users slack.slack_get_users github.list_repository_collaborators +q123,Range query Prometheus,verb_product,prometheus.prom_query_range,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,375,1.0,1,674,1.0,prometheus.prom_query_range prometheus.prom_query prometheus.prom_list_metrics datadog.datadog_query_metrics postgres.query datadog.datadog_list_monitors postgres.explain_query gmail.gmail_search_emails mysql.mysql_query memory.search_nodes +q124,Create a Grafana annotation,verb_product,grafana.grafana_create_annotation,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,326,1.0,1,692,1.0,grafana.grafana_create_annotation grafana.grafana_search_dashboards grafana.grafana_get_dashboard id_map.json slack.create_canvase trello.trello_create_board gitlab.create_snippet stripe.stripe_create_invoice github.create_artifact vercel.vercel_create_project +q125,Apply terraform,identifier,terraform.tf_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,238,1.0,1,562,1.0,terraform.tf_apply terraform.tf_plan kubernetes.k8s_apply terraform.tf_state_list id_map.json github.update_environment cloudflare.cf_create_dns cloudflare.cf_purge_cache cloudflare.cf_list_zones playwright.browser_fill_form +q126,List Asana tasks,verb_product,asana.asana_list_tasks,1.0,1.0,1.0,1.0000,1.0000,1,1,192,1.0,1,368,1.0,1,624,1.0,asana.asana_list_tasks asana.asana_get_task clickup.clickup_list_tasks asana.asana_create_task asana.asana_create_project asana.asana_update_task id_map.json okta.okta_list_users vercel.vercel_list_projects pagerduty.pagerduty_list_incidents +q127,Comment on a Trello card,verb_product,trello.trello_add_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,390,1.0,1,715,1.0,trello.trello_add_comment trello.trello_create_card trello.trello_list_cards trello.trello_create_board figma.figma_post_comment linear.get_comment notion.get_comment github.update_issue_comment linear.update_comment linear.delete_comment +q128,Update a ClickUp task,verb_product,clickup.clickup_update_task,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,399,1.0,1,709,1.0,clickup.clickup_update_task clickup.clickup_get_task clickup.clickup_list_tasks clickup.clickup_create_task asana.asana_update_task puppeteer.puppeteer_click playwright.browser_click asana.asana_get_task gitlab.update_snippet slack.update_bookmark +q129,Search Confluence,verb_product,confluence.confluence_search,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,371,1.0,1,636,1.0,confluence.confluence_search confluence.confluence_get_page confluence.confluence_create_page confluence.confluence_update_page grafana.grafana_search_dashboards github.search_orgs github.search_users hubspot.hubspot_search_contacts github.search_pull_requests github.search_commits +q130,Create a HubSpot deal,verb_product,hubspot.hubspot_create_deal,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,781,1.0,hubspot.hubspot_create_deal hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_search_contacts hubspot.hubspot_list_contacts github.create_gist slack.create_usergroup github.create_repository slack.get_usergroup github.create_branch +q131,List Zendesk tickets,verb_product,zendesk.zendesk_list_tickets,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,367,1.0,1,573,1.0,zendesk.zendesk_list_tickets zendesk.zendesk_search zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_create_ticket id_map.json google_calendar.gcal_list_calendars clickup.clickup_list_tasks jira.list_boards vercel.vercel_list_projects +q132,List Discord channels,verb_product,discord.discord_list_channels,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,300,1.0,1,594,1.0,discord.discord_list_channels slack.slack_list_channels discord.discord_get_channel discord.discord_list_members discord.discord_send_message discord.discord_add_reaction slack.list_usergroups id_map.json slack.slack_get_channel_history clickup.clickup_list_tasks +q133,Set a Redis key,verb_product,redis.redis_set,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,280,1.0,1,563,1.0,redis.redis_set redis.redis_get redis.redis_delete redis.redis_keys playwright.browser_press_key slack.slack_set_channel_topic slack.update_pin id_map.json clickup.clickup_update_task jira.jira_get_issue +q134,Insert MongoDB documents,verb_product,mongodb.mongo_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,256,1.0,1,399,1.0,1,689,1.0,mongodb.mongo_insert mongodb.mongo_update mongodb.mongo_delete mongodb.mongo_find supabase.supabase_insert mongodb.mongo_list_collections id_map.json aws_s3.s3_put_object salesforce.salesforce_create_record elasticsearch.es_index_doc +q135,Run a MySQL query,verb_product,mysql.mysql_query,1.0,1.0,1.0,1.0000,1.0000,1,1,179,1.0,1,286,1.0,1,594,1.0,mysql.mysql_query postgres.query sqlite.sqlite_query mysql.mysql_list_tables salesforce.salesforce_query postgres.explain_query supabase.supabase_select github.delete_check_run github.get_check_run fetch.fetch +q136,Index a document in Elasticsearch,verb_product,elasticsearch.es_index_doc,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,367,1.0,1,567,1.0,elasticsearch.es_index_doc elasticsearch.es_get_doc elasticsearch.es_search playwright.browser_tab_close mongodb.mongo_find id_map.json confluence.confluence_search supabase.supabase_list_tables google_drive.gdrive_search jira.get_attachment +q137,Get an S3 object,verb_product,aws_s3.s3_get_object,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,362,1.0,1,624,1.0,aws_s3.s3_get_object aws_s3.s3_delete_object aws_s3.s3_put_object aws_s3.s3_list_objects aws_s3.s3_list_buckets asana.asana_get_task okta.okta_get_user gitlab.get_snippet linear.get_attachment discord.discord_get_channel +q138,List Cloudflare zones,verb_product,cloudflare.cf_list_zones,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,285,1.0,1,525,1.0,cloudflare.cf_list_zones cloudflare.cf_list_dns cloudflare.cf_purge_cache cloudflare.cf_create_dns id_map.json aws_s3.s3_list_buckets clickup.clickup_list_tasks kubernetes.k8s_list_services vercel.vercel_list_projects kubernetes.k8s_list_deployments +q139,Create a Stripe invoice,verb_product,stripe.stripe_create_invoice,1.0,1.0,1.0,1.0000,1.0000,1,1,205,1.0,1,360,1.0,1,647,1.0,stripe.stripe_create_invoice stripe.stripe_create_product stripe.stripe_create_refund stripe.stripe_create_customer stripe.stripe_create_payment_intent stripe.stripe_list_invoices stripe.stripe_create_price stripe.stripe_get_payment_intent stripe.stripe_get_customer stripe.stripe_cancel_subscription +q140,List GitHub Actions workflows,verb_product,github.actions_list,1.0,1.0,1.0,1.0000,1.0000,1,1,268,1.0,1,453,1.0,1,778,1.0,github.actions_list github.actions_get github.actions_get_job_logs github.actions_run_trigger github.projects_list github.list_notifications github.list_gists github.list_milestones github.list_starred_repositories github.list_hooks +q141,Get GitHub Actions job logs,verb_product,github.actions_get_job_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,442,1.0,1,730,1.0,github.actions_get_job_logs github.actions_get github.actions_list github.actions_run_trigger gitlab.get_job github.get_check_run github.projects_get github.get_notification_details github.get_gist git.git_log +q142,Create a GitHub repository,verb_product,github.create_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,346,1.0,1,556,1.0,1,968,1.0,github.create_repository github.create_branch github.create_or_update_file github.create_issue github.create_gist github.create_environment git.git_create_branch github.star_repository github.create_package github.unstar_repository +q143,Checkout a git branch,verb_product,git.git_checkout,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,366,1.0,1,771,1.0,git.git_checkout git.git_branch git.git_create_branch github.create_branch github.get_check_run git.git_status gitlab.gitlab_create_merge_request git.git_show github.list_commits github.list_branches +q144,Show git log,identifier,git.git_log,1.0,1.0,1.0,1.0000,1.0000,1,1,194,1.0,1,312,1.0,1,629,1.0,git.git_log git.git_show git.git_status git.git_diff_unstaged git.git_diff git.git_branch git.git_diff_staged github.actions_get_job_logs git.git_commit git.git_checkout +q145,List postgres schemas,verb_product,postgres.list_schemas,1.0,1.0,1.0,1.0000,1.0000,1,1,169,1.0,1,262,1.0,1,455,1.0,postgres.list_schemas postgres.list_tables postgres.describe_table vercel.vercel_list_projects postgres.explain_query mysql.mysql_list_tables postgres.query id_map.json supabase.supabase_list_tables clickup.clickup_list_tasks +q146,Open nodes in the knowledge graph,verb_product,memory.open_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,163,1.0,1,262,1.0,1,564,1.0,memory.open_nodes memory.search_nodes memory.read_graph memory.create_entities memory.create_relations memory.add_observations memory.delete_relations memory.delete_entities figma.figma_get_file_nodes notion.notion_search +q147,Get file metadata from the filesystem,paraphrase,filesystem.get_file_info,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,324,1.0,1,717,1.0,filesystem.get_file_info google_drive.gdrive_get_file filesystem.read_file filesystem.read_media_file figma.figma_get_file_nodes filesystem.read_text_file github.get_file_contents gitlab.gitlab_get_file filesystem.list_directory filesystem.directory_tree +q148,Write a new local file,paraphrase,filesystem.write_file,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,384,1.0,1,856,1.0,filesystem.write_file filesystem.edit_file filesystem.read_text_file filesystem.read_media_file filesystem.read_file slack.create_file filesystem.create_directory brave_search.brave_local_search google_drive.gdrive_create_file github.issue_write +q149,Search GitHub issues,verb_product,github.search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,363,1.0,1,702,1.0,github.search_issues github.search_users github.search_orgs github.list_issues github.search_pull_requests github.search_commits github.search_repositories github.search_code linear.linear_search_issues jira.jira_search_issues +q150,Get the authenticated GitHub user,paraphrase,github.get_teams,0.0,1.0,1.0,0.5000,0.6309,2,1,179,1.0,1,285,1.0,1,625,1.0,github.get_teams github.get_me github.list_notifications slack.slack_get_user_profile github.list_gists github.get_latest_release github.get_secret_scanning_alert okta.okta_get_user github.projects_get linear.get_user diff --git a/experiments/mcp-tool-search/results/hybrid_codedb_embed.json b/experiments/mcp-tool-search/results/hybrid_codedb_embed.json new file mode 100644 index 00000000..b7390a36 --- /dev/null +++ b/experiments/mcp-tool-search/results/hybrid_codedb_embed.json @@ -0,0 +1,7013 @@ +{ + "summary": { + "n": 150, + "index_s": 2.4014929960000018, + "rss_mb": null, + "p50_ms": 0.006578000011359109, + "p95_ms": 0.014046500018594085, + "recall@1": 0.9333333333333333, + "recall@1_lo": 0.8933333333333333, + "recall@1_hi": 0.9733333333333334, + "recall@3": 0.9866666666666667, + "recall@3_lo": 0.9666666666666667, + "recall@3_hi": 1.0, + "recall@5": 0.9933333333333333, + "recall@5_lo": 0.98, + "recall@5_hi": 1.0, + "mrr@10": 0.9591111111111111, + "mrr@10_lo": 0.9315555555555555, + "mrr@10_hi": 0.9833333333333333, + "ndcg@10": 0.9658210376068275, + "ndcg@10_lo": 0.9431111280939956, + "ndcg@10_hi": 0.98539579120926 + }, + "meta": { + "name": "hybrid_codedb_embed", + "needs_model": true, + "needs_api": false, + "notes": "RRF codedb_search + embedding" + }, + "rows": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "github.issue_write", + "gitlab.gitlab_create_issue", + "github.add_issue_comment", + "github.issue_read", + "bitbucket.bitbucket_create_issue", + "github.search_issues", + "linear.linear_create_issue", + "github.create_repository", + "jira.jira_create_issue" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 380, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 584, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1088, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "kind": "verb_product", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_create_issue", + "jira.jira_create_sprint", + "jira.create_sprint", + "jira.create_webhook", + "jira.create_version", + "jira.create_filter", + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_get_ticket", + "jira.get_sprint", + "zendesk.zendesk_update_ticket" + ], + "hit1": "jira.jira_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 345, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 540, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 924, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "kind": "verb_product", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "ranking": [ + "slack.slack_post_message", + "slack.slack_add_reaction", + "slack.slack_create_channel", + "discord.discord_send_message", + "slack.slack_reply_to_thread", + "twilio.twilio_send_sms", + "slack.get_reminder", + "gmail.gmail_send_email", + "slack.slack_upload_file", + "slack.slack_leave_channel" + ], + "hit1": "slack.slack_post_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 408, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 778, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "kind": "resource", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "ranking": [ + "github.list_pull_requests", + "github.search_pull_requests", + "bitbucket.bitbucket_list_prs", + "github.create_pull_request", + "github.pull_request_read", + "gitlab.gitlab_list_merge_requests", + "playwright.browser_tab_list", + "gitlab.list_snippets", + "memory.open_nodes", + "bitbucket.bitbucket_list_repos" + ], + "hit1": "github.list_pull_requests", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 477, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 769, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "ranking": [ + "github.create_pull_request", + "github.merge_pull_request", + "github.pull_request_read", + "github.pull_request_review_write", + "github.search_pull_requests", + "github.update_pull_request", + "github.list_pull_requests", + "github.update_pull_request_branch", + "github.add_comment_to_pending_review", + "github.request_copilot_review" + ], + "hit1": "github.create_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 368, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 564, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1144, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "kind": "verb_product", + "gold": [ + "postgres.query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "mysql.mysql_query", + "sqlite.sqlite_query", + "postgres.explain_query", + "salesforce.salesforce_query", + "postgres.list_schemas", + "postgres.describe_table", + "postgres.list_tables", + "prometheus.prom_query", + "github.delete_check_run" + ], + "hit1": "postgres.query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 551, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "kind": "paraphrase", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "ranking": [ + "filesystem.read_media_file", + "filesystem.read_text_file", + "filesystem.read_file", + "github.get_file_contents", + "figma.figma_get_file_nodes", + "filesystem.get_file_info", + "gitlab.gitlab_get_file", + "github.delete_file", + "filesystem.list_directory", + "figma.figma_get_file" + ], + "hit1": "filesystem.read_media_file", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6199062332840657, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 406, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "kind": "paraphrase", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "ranking": [ + "google_drive.gdrive_list_folder", + "filesystem.list_directory", + "filesystem.list_directory_with_sizes", + "filesystem.search_files", + "filesystem.directory_tree", + "slack.list_files", + "github.push_files", + "filesystem.get_file_info", + "cloudflare.cf_purge_cache", + "filesystem.read_file" + ], + "hit1": "google_drive.gdrive_list_folder", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6862856989769305, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 191, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "ranking": [ + "memory.read_graph", + "memory.create_relations", + "memory.add_observations", + "memory.open_nodes", + "memory.create_entities", + "memory.search_nodes", + "memory.delete_entities", + "memory.delete_relations", + "postgres.list_schemas", + "sequential_thinking.sequentialthinking" + ], + "hit1": "memory.read_graph", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5271341073823443, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 164, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 568, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "kind": "verb_product", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "ranking": [ + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "grafana.grafana_search_dashboards", + "hubspot.hubspot_search_contacts", + "github.search_orgs", + "jira.jira_search_issues", + "memory.search_nodes", + "filesystem.list_directory_with_sizes", + "confluence.confluence_search", + "github.update_pull_request_branch" + ], + "hit1": "brave_search.brave_web_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 691, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "kind": "verb_product", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "ranking": [ + "fetch.fetch", + "playwright.browser_navigate", + "github.get_file_contents", + "filesystem.read_file", + "confluence.confluence_search", + "git.git_add", + "puppeteer.puppeteer_navigate", + "playwright.browser_snapshot", + "gitlab.get_snippet", + "slack.slack_set_channel_topic" + ], + "hit1": "fetch.fetch", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 369, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 659, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "kind": "identifier", + "gold": [ + "git.git_status" + ], + "alts": [], + "ranking": [ + "git.git_status", + "git.git_show", + "git.git_diff_unstaged", + "git.git_log", + "git.git_diff", + "github.list_milestones", + "git.git_diff_staged", + "github.list_code_scanning_alerts", + "gitlab.list_releases", + "github.get_notification_details" + ], + "hit1": "git.git_status", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "kind": "paraphrase", + "gold": [ + "git.git_commit" + ], + "alts": [], + "ranking": [ + "git.git_commit", + "git.git_reset", + "git.git_diff_staged", + "git.git_show", + "github.get_commit", + "github.update_milestone", + "filesystem.move_file", + "git.git_log", + "github.push_files", + "slack.update_reminder" + ], + "hit1": "git.git_commit", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_issue", + "linear.linear_get_issue", + "linear.linear_create_comment", + "linear.linear_assign_issue", + "linear.linear_update_issue", + "linear.linear_create_project", + "linear.linear_search_issues", + "bitbucket.bitbucket_create_issue", + "gitlab.gitlab_create_issue", + "linear.linear_list_issues" + ], + "hit1": "linear.linear_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 275, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 867, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "kind": "paraphrase", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_project", + "twilio.twilio_make_call", + "linear.linear_list_projects", + "clickup.clickup_get_task", + "linear.linear_create_issue", + "asana.asana_get_task", + "linear.linear_update_issue", + "clickup.clickup_update_task", + "linear.linear_list_issues", + "asana.asana_update_task" + ], + "hit1": "linear.linear_create_project", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.2, + "ndcg@10": 0.38685280723454163, + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 502, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 426, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 857, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "ranking": [ + "github.update_issue_comment", + "github.add_issue_comment", + "github.add_comment_to_pending_review", + "github.assign_copilot_to_issue", + "github.issue_write", + "github.get_discussion_comments", + "github.issue_read", + "figma.figma_post_comment", + "jira.jira_add_comment", + "github.list_repository_collaborators" + ], + "hit1": "github.update_issue_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 388, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 628, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1039, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "kind": "ambiguous", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "ranking": [ + "github.merge_pull_request", + "github.update_pull_request_branch", + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_get_merge_request", + "gitlab.gitlab_create_merge_request", + "github.create_pull_request", + "github.pull_request_read", + "github.update_pull_request", + "gitlab.gitlab_list_merge_requests", + "github.pull_request_review_write" + ], + "hit1": "github.merge_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 299, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 495, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1093, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "kind": "verb_product", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "ranking": [ + "github.actions_run_trigger", + "github.actions_get", + "github.actions_list", + "github.actions_get_job_logs", + "circleci.circleci_trigger_pipeline", + "github.update_check_run", + "github.search_users", + "github.dismiss_notification", + "github.search_orgs", + "github.get_check_run" + ], + "hit1": "github.actions_run_trigger", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 735, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "kind": "resource", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "ranking": [ + "github.list_dependabot_alerts", + "github.get_dependabot_alert", + "github.list_secret_scanning_alerts", + "github.list_code_scanning_alerts", + "github.get_code_scanning_alert", + "github.get_secret_scanning_alert", + "id_map.json", + "discord.discord_get_channel", + "vercel.vercel_get_deployment", + "pagerduty.pagerduty_list_incidents" + ], + "hit1": "github.list_dependabot_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 464, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 748, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "kind": "verb_product", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_issues", + "sentry.sentry_list_projects", + "sentry.sentry_list_events", + "sentry.sentry_get_issue", + "linear.linear_list_issues", + "sentry.sentry_update_issue", + "gitlab.gitlab_list_issues", + "github.list_issues", + "bitbucket.bitbucket_list_prs", + "pagerduty.pagerduty_list_incidents" + ], + "hit1": "sentry.sentry_list_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 393, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 809, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_customer", + "stripe.stripe_create_invoice", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_customer", + "stripe.stripe_create_product", + "stripe.stripe_list_invoices", + "stripe.stripe_create_refund", + "stripe.stripe_create_price", + "stripe.stripe_list_subscriptions", + "stripe.stripe_get_payment_intent" + ], + "hit1": "stripe.stripe_create_customer", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "kind": "paraphrase", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_refund", + "stripe.stripe_get_payment_intent", + "stripe.stripe_cancel_subscription", + "stripe.stripe_create_invoice", + "stripe.stripe_get_customer", + "stripe.stripe_create_payment_intent", + "stripe.stripe_list_invoices", + "stripe.stripe_create_product", + "stripe.stripe_list_customers", + "stripe.stripe_create_price" + ], + "hit1": "stripe.stripe_create_refund", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 197, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 644, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "kind": "verb_product", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "ranking": [ + "gmail.gmail_send_email", + "gmail.gmail_trash_email", + "gmail.gmail_get_email", + "gmail.gmail_modify_email", + "gmail.gmail_create_draft", + "gmail.gmail_list_labels", + "gmail.gmail_search_emails", + "twilio.twilio_send_sms", + "discord.discord_send_message", + "google_drive.gdrive_share_file" + ], + "hit1": "gmail.gmail_send_email", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 743, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "kind": "verb_product", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "ranking": [ + "gmail.gmail_search_emails", + "gmail.gmail_list_labels", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "gmail.gmail_create_draft", + "gmail.gmail_modify_email", + "hubspot.hubspot_search_contacts", + "gmail.gmail_send_email", + "github.search_users", + "grafana.grafana_search_dashboards" + ], + "hit1": "gmail.gmail_search_emails", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 668, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "kind": "paraphrase", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_update_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_share_file", + "slack.slack_upload_file", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "filesystem.edit_file", + "aws_s3.s3_put_object", + "filesystem.write_file" + ], + "hit1": "google_drive.gdrive_update_file", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 277, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 446, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 806, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_create_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "github.pull_request_review_write", + "sentry.sentry_list_events", + "id_map.json", + "slack.create_reminder", + "trello.trello_create_board" + ], + "hit1": "google_calendar.gcal_create_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 801, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_list_pods", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_apply", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_scale_deployment", + "docker.docker_images", + "kubernetes.k8s_delete" + ], + "hit1": "kubernetes.k8s_list_pods", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_get_pod", + "docker.docker_logs", + "github.actions_get_job_logs", + "kubernetes.k8s_get_events", + "datadog.datadog_list_logs", + "kubernetes.k8s_list_pods", + "elasticsearch.es_get_doc", + "kubernetes.k8s_list_services", + "aws_s3.s3_get_object" + ], + "hit1": "kubernetes.k8s_pod_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 408, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "ranking": [ + "docker.docker_run", + "docker.docker_exec", + "docker.docker_stop", + "docker.docker_ps", + "docker.docker_logs", + "docker.docker_rm", + "docker.docker_images", + "docker.docker_build", + "github.delete_check_run", + "terraform.tf_plan" + ], + "hit1": "docker.docker_run", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 587, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "kind": "verb_product", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "ranking": [ + "docker.docker_logs", + "kubernetes.k8s_pod_logs", + "docker.docker_stop", + "docker.docker_run", + "docker.docker_exec", + "git.git_log", + "datadog.datadog_list_logs", + "docker.docker_rm", + "docker.docker_images", + "docker.docker_ps" + ], + "hit1": "docker.docker_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 383, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 683, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "kind": "ambiguous", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "ranking": [ + "playwright.browser_navigate", + "puppeteer.puppeteer_navigate", + "playwright.browser_tab_new", + "playwright.browser_press_key", + "playwright.browser_click", + "fetch.fetch", + "playwright.browser_take_screenshot", + "playwright.browser_evaluate", + "playwright.browser_snapshot", + "playwright.browser_wait_for" + ], + "hit1": "playwright.browser_navigate", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 603, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "kind": "ambiguous", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "ranking": [ + "playwright.browser_take_screenshot", + "puppeteer.puppeteer_screenshot", + "playwright.browser_snapshot", + "playwright.browser_evaluate", + "notion.notion_get_page", + "pagerduty.pagerduty_get_incident", + "notion.notion_update_page", + "pagerduty.pagerduty_acknowledge_incident", + "confluence.confluence_get_page", + "confluence.confluence_create_page" + ], + "hit1": "playwright.browser_take_screenshot", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_create_merge_request", + "gitlab.gitlab_get_merge_request", + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_list_merge_requests", + "github.merge_pull_request", + "gitlab.gitlab_create_pipeline", + "gitlab.create_release", + "gitlab.gitlab_create_issue", + "gitlab.create_pipeline", + "github.create_pull_request" + ], + "hit1": "gitlab.gitlab_create_merge_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 983, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "kind": "verb_product", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "ranking": [ + "notion.notion_search", + "notion.notion_get_page", + "notion.notion_list_users", + "notion.notion_get_block", + "notion.notion_create_page", + "notion.notion_query_database", + "grafana.grafana_search_dashboards", + "notion.notion_update_page", + "github.search_users", + "hubspot.hubspot_search_contacts" + ], + "hit1": "notion.notion_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 640, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "kind": "verb_product", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "ranking": [ + "notion.notion_create_page", + "notion.notion_get_page", + "notion.notion_update_page", + "notion.notion_create_database", + "notion.create_data_source", + "notion.create_comment", + "notion.notion_get_comments", + "notion.create_users_token", + "confluence.confluence_create_page", + "notion.notion_search" + ], + "hit1": "notion.notion_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 419, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 834, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "kind": "verb_product", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "ranking": [ + "notion.notion_query_database", + "notion.notion_create_database", + "notion.notion_search", + "notion.notion_create_page", + "notion.notion_list_users", + "notion.notion_get_page", + "notion.notion_get_block", + "notion.get_data_source", + "postgres.query", + "notion.notion_get_comments" + ], + "hit1": "notion.notion_query_database", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 656, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_list_tickets", + "jira.jira_create_issue", + "zendesk.zendesk_search", + "id_map.json", + "jira.jira_create_sprint", + "trello.trello_create_board", + "google_calendar.gcal_create_event" + ], + "hit1": "zendesk.zendesk_create_ticket", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 426, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 770, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_search_contacts", + "hubspot.hubspot_create_deal", + "hubspot.hubspot_list_contacts", + "github.create_gist", + "slack.create_usergroup", + "github.create_repository", + "slack.get_usergroup", + "github.create_branch" + ], + "hit1": "hubspot.hubspot_create_contact", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "kind": "verb_product", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_create_lead", + "salesforce.salesforce_convert_lead", + "salesforce.salesforce_create_record", + "salesforce.salesforce_list_opportunities", + "salesforce.salesforce_get_record", + "salesforce.salesforce_delete_record", + "slack.create_pin", + "salesforce.salesforce_query", + "linear.create_user", + "salesforce.salesforce_update_record" + ], + "hit1": "salesforce.salesforce_create_lead", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "kind": "identifier", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_query", + "postgres.query", + "mysql.mysql_query", + "sqlite.sqlite_query", + "github.delete_check_run", + "prometheus.prom_query", + "github.get_check_run", + "postgres.explain_query", + "github.update_check_run", + "prometheus.prom_query_range" + ], + "hit1": "salesforce.salesforce_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 187, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 293, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 622, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "kind": "verb_product", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "ranking": [ + "shopify.shopify_list_orders", + "shopify.shopify_list_customers", + "shopify.shopify_get_order", + "shopify.shopify_list_products", + "shopify.shopify_create_order", + "shopify.shopify_get_product", + "shopify.shopify_update_product", + "shopify.shopify_create_product", + "id_map.json", + "stripe.stripe_list_subscriptions" + ], + "hit1": "shopify.shopify_list_orders", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 618, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "id_map.json", + "sentry.sentry_list_events", + "trello.trello_create_board", + "playwright.browser_press_key" + ], + "hit1": "pagerduty.pagerduty_create_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "kind": "paraphrase", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_list_oncalls", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "twilio.twilio_make_call", + "slack.slack_create_channel", + "playwright.browser_press_key", + "filesystem.list_allowed_directories" + ], + "hit1": "pagerduty.pagerduty_list_oncalls", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "kind": "verb_product", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "ranking": [ + "datadog.datadog_query_metrics", + "datadog.datadog_list_monitors", + "datadog.datadog_list_logs", + "datadog.datadog_create_monitor", + "datadog.datadog_get_monitor", + "prometheus.prom_list_metrics", + "memory.search_nodes", + "postgres.query", + "slack.slack_search_messages", + "prometheus.prom_query" + ], + "hit1": "datadog.datadog_query_metrics", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 284, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 739, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "kind": "verb_product", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "ranking": [ + "datadog.datadog_list_logs", + "datadog.datadog_list_monitors", + "datadog.datadog_get_monitor", + "datadog.datadog_create_monitor", + "datadog.datadog_query_metrics", + "github.search_users", + "docker.docker_logs", + "github.search_pull_requests", + "kubernetes.k8s_pod_logs", + "memory.search_nodes" + ], + "hit1": "datadog.datadog_list_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 778, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_list_buckets", + "aws_s3.s3_list_objects", + "aws_s3.s3_delete_object", + "aws_s3.s3_get_object", + "aws_s3.s3_put_object", + "jira.jira_list_projects", + "bitbucket.bitbucket_list_repos", + "clickup.clickup_list_tasks", + "bitbucket.bitbucket_list_prs", + "id_map.json" + ], + "hit1": "aws_s3.s3_list_buckets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 203, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 361, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 577, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "kind": "paraphrase", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_put_object", + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_list_objects", + "slack.slack_upload_file", + "twilio.twilio_send_sms", + "aws_s3.s3_list_buckets", + "github.assign_copilot_to_issue", + "gmail.gmail_send_email", + "jira.update_attachment" + ], + "hit1": "aws_s3.s3_put_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 795, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "cloudflare.cf_list_dns", + "cloudflare.cf_create_dns", + "slack.delete_bookmark", + "slack.delete_canvase", + "id_map.json", + "memory.delete_observations", + "redis.redis_delete", + "aws_s3.s3_delete_object" + ], + "hit1": "cloudflare.cf_purge_cache", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 567, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "salesforce.salesforce_create_record", + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "salesforce.salesforce_get_record", + "salesforce.salesforce_delete_record", + "datadog.datadog_create_monitor", + "salesforce.salesforce_update_record", + "slack.create_file" + ], + "hit1": "cloudflare.cf_create_dns", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 808, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_deployments", + "kubernetes.k8s_list_deployments", + "vercel.vercel_get_deployment", + "vercel.vercel_list_projects", + "vercel.vercel_create_project", + "id_map.json", + "github.list_packages", + "clickup.clickup_list_tasks", + "gitlab.list_releases", + "jira.jira_list_projects" + ], + "hit1": "vercel.vercel_list_deployments", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 531, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "ranking": [ + "supabase.supabase_insert", + "mongodb.mongo_insert", + "supabase.supabase_update", + "supabase.supabase_list_tables", + "supabase.supabase_select", + "supabase.supabase_invoke_fn", + "playwright.browser_type", + "linear.update_cycle", + "id_map.json", + "linear.update_attachment" + ], + "hit1": "supabase.supabase_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 636, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "kind": "paraphrase", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "ranking": [ + "figma.figma_get_images", + "figma.figma_get_file_nodes", + "figma.figma_get_file", + "docker.docker_images", + "figma.figma_get_comments", + "figma.figma_post_comment", + "memory.delete_observations", + "linear.update_attachment", + "elasticsearch.es_get_doc", + "linear.get_attachment" + ], + "hit1": "figma.figma_get_images", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "kind": "verb_product", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "ranking": [ + "twilio.twilio_send_sms", + "twilio.twilio_list_messages", + "twilio.twilio_make_call", + "discord.discord_send_message", + "gmail.gmail_send_email", + "slack.slack_reply_to_thread", + "slack.slack_add_reaction", + "filesystem.list_directory_with_sizes", + "puppeteer.puppeteer_click", + "slack.slack_post_message" + ], + "hit1": "twilio.twilio_send_sms", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 771, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "kind": "verb_product", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "ranking": [ + "okta.okta_create_user", + "okta.okta_get_user", + "okta.okta_deactivate_user", + "okta.okta_list_users", + "linear.create_user", + "asana.asana_create_project", + "asana.asana_create_task", + "notion.create_users_token", + "github.create_issue", + "slack.slack_get_user_profile" + ], + "hit1": "okta.okta_create_user", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 795, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query", + "prometheus.prom_query_range", + "prometheus.prom_list_metrics", + "datadog.datadog_query_metrics", + "postgres.query", + "memory.search_nodes", + "postgres.explain_query", + "github.search_pull_requests", + "mysql.mysql_query", + "hubspot.hubspot_search_contacts" + ], + "hit1": "prometheus.prom_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 650, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "kind": "verb_product", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "ranking": [ + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "grafana.grafana_create_annotation", + "github.search_users", + "github.search_orgs", + "github.search_pull_requests", + "hubspot.hubspot_search_contacts", + "confluence.confluence_search", + "github.search_repositories", + "github.search_commits" + ], + "hit1": "grafana.grafana_search_dashboards", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 627, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "kind": "verb_product", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "ranking": [ + "circleci.circleci_trigger_pipeline", + "circleci.circleci_get_pipeline", + "circleci.circleci_list_pipelines", + "github.actions_run_trigger", + "gitlab.gitlab_create_pipeline", + "gitlab.update_pipeline", + "gitlab.get_pipeline", + "gitlab.list_pipelines", + "hubspot.hubspot_create_deal", + "gitlab.delete_pipeline" + ], + "hit1": "circleci.circleci_trigger_pipeline", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "kind": "identifier", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "ranking": [ + "terraform.tf_plan", + "terraform.tf_apply", + "terraform.tf_state_list", + "github.update_check_run", + "github.get_check_run", + "docker.docker_run", + "github.create_check_run", + "github.delete_check_run", + "hubspot.hubspot_create_deal", + "github.update_runner" + ], + "hit1": "terraform.tf_plan", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "kind": "verb_product", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "ranking": [ + "asana.asana_create_task", + "asana.asana_get_task", + "asana.asana_create_project", + "asana.asana_update_task", + "asana.asana_list_tasks", + "clickup.clickup_create_task", + "okta.okta_create_user", + "clickup.clickup_get_task", + "slack.create_reminder", + "clickup.clickup_update_task" + ], + "hit1": "asana.asana_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 795, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "ranking": [ + "trello.trello_create_card", + "trello.trello_create_board", + "trello.trello_add_comment", + "trello.trello_list_cards", + "stripe.stripe_create_invoice", + "id_map.json", + "grafana.grafana_create_annotation", + "vercel.vercel_create_project", + "stripe.stripe_create_customer", + "okta.okta_create_user" + ], + "hit1": "trello.trello_create_card", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 627, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_create_task", + "clickup.clickup_get_task", + "clickup.clickup_list_tasks", + "clickup.clickup_update_task", + "puppeteer.puppeteer_click", + "asana.asana_create_task", + "playwright.browser_click", + "asana.asana_get_task", + "asana.asana_update_task", + "puppeteer.puppeteer_fill" + ], + "hit1": "clickup.clickup_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "kind": "verb_product", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "ranking": [ + "confluence.confluence_create_page", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "notion.notion_create_page", + "confluence.confluence_search", + "notion.notion_get_page", + "github.create_artifact", + "notion.notion_update_page", + "notion.create_users_token", + "id_map.json" + ], + "hit1": "confluence.confluence_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "kind": "verb_product", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "discord.discord_add_reaction", + "discord.discord_get_channel", + "slack.slack_post_message", + "twilio.twilio_send_sms", + "discord.discord_list_channels", + "gmail.gmail_trash_email", + "gmail.gmail_send_email", + "slack.slack_add_reaction", + "slack.slack_reply_to_thread" + ], + "hit1": "discord.discord_send_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_search", + "elasticsearch.es_get_doc", + "hubspot.hubspot_search_contacts", + "confluence.confluence_search", + "grafana.grafana_search_dashboards", + "github.search_users", + "github.search_orgs", + "github.search_pull_requests", + "elasticsearch.es_index_doc", + "github.search_commits" + ], + "hit1": "elasticsearch.es_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 617, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "redis.redis_set", + "redis.redis_delete", + "redis.redis_keys", + "figma.figma_get_file", + "jira.jira_get_issue", + "playwright.browser_press_key", + "slack.get_pin", + "figma.figma_get_comments", + "circleci.circleci_get_pipeline" + ], + "hit1": "redis.redis_get", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 283, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 549, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_find", + "mongodb.mongo_delete", + "mongodb.mongo_insert", + "mongodb.mongo_list_collections", + "mongodb.mongo_update", + "id_map.json", + "elasticsearch.es_get_doc", + "filesystem.search_files", + "confluence.confluence_search", + "datadog.datadog_list_logs" + ], + "hit1": "mongodb.mongo_find", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 672, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "kind": "verb_product", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "ranking": [ + "sqlite.sqlite_list_tables", + "mysql.mysql_list_tables", + "supabase.supabase_list_tables", + "sqlite.sqlite_describe_table", + "postgres.list_tables", + "sqlite.sqlite_query", + "okta.okta_list_users", + "id_map.json", + "postgres.list_schemas", + "clickup.clickup_list_tasks" + ], + "hit1": "sqlite.sqlite_list_tables", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 142, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 240, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 433, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "kind": "verb_product", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "ranking": [ + "postgres.describe_table", + "sqlite.sqlite_describe_table", + "postgres.explain_query", + "postgres.list_schemas", + "postgres.list_tables", + "postgres.query", + "supabase.supabase_insert", + "supabase.supabase_select", + "mysql.mysql_list_tables", + "supabase.supabase_update" + ], + "hit1": "postgres.describe_table", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 274, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 590, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_repository", + "github.create_issue", + "github.issue_write", + "github.create_gist", + "gitlab.gitlab_create_issue", + "github.create_environment", + "github.create_package", + "github.create_artifact", + "bitbucket.bitbucket_create_issue", + "github.create_runner" + ], + "hit1": "github.create_repository", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.66967181649423, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 367, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 536, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1012, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "ranking": [ + "linear.create_user", + "gmail.gmail_create_draft", + "notion.create_data_source", + "linear.create_cycle", + "notion.create_comment", + "linear.create_comment", + "linear.create_attachment", + "notion.create_users_token", + "stripe.stripe_create_product", + "okta.okta_create_user" + ], + "hit1": "linear.create_user", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 39, + "k3_calls": 13, + "k3_tokens": 3554, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 8, + "k5_tokens": 3600, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3528, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q071", + "query": "list-directory", + "kind": "identifier", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "ranking": [ + "filesystem.list_directory", + "filesystem.list_directory_with_sizes", + "okta.okta_list_users", + "mongodb.mongo_list_collections", + "filesystem.list_allowed_directories", + "clickup.clickup_list_tasks", + "terraform.tf_state_list", + "sentry.sentry_list_projects", + "docker.docker_images", + "gmail.gmail_list_labels" + ], + "hit1": "filesystem.list_directory", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 271, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 489, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "kind": "paraphrase", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "ranking": [ + "sequential_thinking.sequentialthinking", + "id_map.json", + "memory.create_relations", + "memory.add_observations", + "memory.search_nodes", + "memory.create_entities", + "memory.delete_relations", + "notion.update_comment", + "notion.notion_search", + "memory.open_nodes" + ], + "hit1": "sequential_thinking.sequentialthinking", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 579, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "kind": "paraphrase", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "ranking": [ + "time.get_current_time", + "time.convert_time", + "google_calendar.gcal_list_events", + "grafana.grafana_create_annotation", + "playwright.browser_wait_for", + "zendesk.zendesk_list_tickets", + "slack.slack_create_channel", + "filesystem.list_allowed_directories", + "jira.jira_create_sprint", + "jira.jira_add_worklog" + ], + "hit1": "time.get_current_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "kind": "verb_product", + "gold": [ + "time.convert_time" + ], + "alts": [], + "ranking": [ + "time.convert_time", + "time.get_current_time", + "grafana.grafana_create_annotation", + "playwright.browser_wait_for", + "google_calendar.gcal_list_events", + "salesforce.salesforce_convert_lead", + "memory.create_relations", + "git.git_diff", + "datadog.datadog_query_metrics", + "cloudflare.cf_list_zones" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "kind": "verb_product", + "gold": [ + "github.star_repository" + ], + "alts": [], + "ranking": [ + "github.star_repository", + "github.unstar_repository", + "github.delete_repository", + "github.list_starred_repositories", + "github.actions_list", + "github.create_or_update_file", + "github.get_label", + "github.create_gist", + "github.get_discussion", + "github.get_file_blame" + ], + "hit1": "github.star_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 838, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "kind": "resource", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "ranking": [ + "github.fork_repository", + "github.unstar_repository", + "github.star_repository", + "github.delete_repository", + "github.create_repository", + "github.get_repository_tree", + "github.merge_pull_request", + "github.list_repository_collaborators", + "bitbucket.bitbucket_create_pr", + "github.get_commit" + ], + "hit1": "github.fork_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 883, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "kind": "verb_product", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "ranking": [ + "github.get_latest_release", + "github.get_release_by_tag", + "gitlab.get_release", + "github.get_gist", + "github.get_teams", + "github.get_milestone", + "github.get_me", + "github.list_releases", + "github.get_repository_tree", + "github.get_notification_details" + ], + "hit1": "github.get_latest_release", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "kind": "resource", + "gold": [ + "github.create_gist" + ], + "alts": [], + "ranking": [ + "github.create_gist", + "github.get_gist", + "github.update_gist", + "trello.trello_create_board", + "gitlab.create_snippet", + "stripe.stripe_create_invoice", + "github.list_gists", + "vercel.vercel_create_project", + "github.create_milestone", + "okta.okta_create_user" + ], + "hit1": "github.create_gist", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "kind": "verb_product", + "gold": [ + "github.search_code" + ], + "alts": [], + "ranking": [ + "github.search_code", + "github.search_orgs", + "github.search_users", + "github.get_code_scanning_alert", + "github.search_pull_requests", + "github.list_code_scanning_alerts", + "github.search_commits", + "github.search_repositories", + "github.search_issues", + "github.get_check_run" + ], + "hit1": "github.search_code", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 722, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "kind": "verb_product", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "ranking": [ + "github.list_notifications", + "github.mark_all_notifications_read", + "github.list_gists", + "github.list_secret_scanning_alerts", + "github.get_notification_details", + "github.projects_list", + "github.list_code_scanning_alerts", + "github.actions_list", + "github.list_starred_repositories", + "github.list_dependabot_alerts" + ], + "hit1": "github.list_notifications", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 174, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 307, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 671, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "kind": "verb_product", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "ranking": [ + "github.assign_copilot_to_issue", + "linear.linear_assign_issue", + "jira.jira_assign_issue", + "github.request_copilot_review", + "github.create_issue", + "linear.linear_create_comment", + "jira.jira_update_issue", + "jira.jira_transition_issue", + "pagerduty.pagerduty_resolve_incident", + "github.add_issue_comment" + ], + "hit1": "github.assign_copilot_to_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 500, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 873, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "kind": "resource", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "ranking": [ + "github.list_secret_scanning_alerts", + "github.get_secret_scanning_alert", + "github.list_code_scanning_alerts", + "github.get_code_scanning_alert", + "github.list_dependabot_alerts", + "github.get_dependabot_alert", + "id_map.json", + "discord.discord_get_channel", + "github.get_notification_details", + "vercel.vercel_get_deployment" + ], + "hit1": "github.list_secret_scanning_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 723, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "ranking": [ + "github.create_or_update_file", + "github.projects_write", + "slack.update_file", + "github.issue_write", + "google_drive.gdrive_update_file", + "github.update_gist", + "github.create_gist", + "slack.create_file", + "google_drive.gdrive_create_file", + "github.update_environment" + ], + "hit1": "github.create_or_update_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 320, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 547, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 975, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "kind": "resource", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "ranking": [ + "github.list_repository_collaborators", + "github.list_labels", + "github.list_releases", + "github.actions_list", + "github.list_discussions", + "github.list_commits", + "pagerduty.pagerduty_list_oncalls", + "github.list_starred_repositories", + "trello.trello_list_cards", + "github.list_tags" + ], + "hit1": "github.list_repository_collaborators", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 745, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "kind": "verb_product", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "ranking": [ + "slack.slack_reply_to_thread", + "slack.slack_get_thread_replies", + "slack.get_reminder", + "slack.slack_list_channels", + "slack.slack_add_reaction", + "slack.delete_file", + "slack.slack_post_message", + "slack.get_canvase", + "slack.delete_canvase", + "slack.update_reminder" + ], + "hit1": "slack.slack_reply_to_thread", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 708, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "kind": "verb_product", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "ranking": [ + "slack.slack_add_reaction", + "discord.discord_add_reaction", + "trello.trello_add_comment", + "slack.slack_reply_to_thread", + "gmail.gmail_modify_email", + "slack.slack_get_thread_replies", + "jira.jira_add_comment", + "slack.update_canvase", + "jira.jira_add_worklog", + "slack.update_reminder" + ], + "hit1": "slack.slack_add_reaction", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 271, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 454, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 833, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "kind": "verb_product", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "ranking": [ + "slack.slack_search_messages", + "slack.slack_get_channel_history", + "gmail.gmail_search_emails", + "slack.slack_add_reaction", + "twilio.twilio_list_messages", + "github.search_pull_requests", + "grafana.grafana_search_dashboards", + "slack.slack_get_thread_replies", + "github.search_users", + "slack.get_reminder" + ], + "hit1": "slack.slack_search_messages", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 374, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 668, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "kind": "verb_product", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "ranking": [ + "slack.slack_create_channel", + "slack.slack_leave_channel", + "slack.slack_join_channel", + "slack.slack_post_message", + "slack.slack_set_channel_topic", + "slack.slack_reply_to_thread", + "slack.slack_get_thread_replies", + "slack.slack_add_reaction", + "slack.slack_upload_file", + "slack.slack_get_channel_history" + ], + "hit1": "slack.slack_create_channel", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "kind": "verb_product", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_transition_issue", + "jira.jira_assign_issue", + "jira.jira_update_issue", + "jira.jira_get_issue", + "jira.jira_add_comment", + "jira.jira_create_issue", + "jira.jira_add_worklog", + "jira.update_sprint", + "gitlab.gitlab_get_issue", + "jira.jira_search_issues" + ], + "hit1": "jira.jira_transition_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 222, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 760, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "kind": "identifier", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "ranking": [ + "jira.jira_search_issues", + "filesystem.list_directory_with_sizes", + "jira.get_filter", + "grafana.grafana_search_dashboards", + "jira.create_filter", + "github.search_users", + "jira.jira_get_issue", + "hubspot.hubspot_search_contacts", + "jira.update_filter", + "github.search_orgs" + ], + "hit1": "jira.jira_search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 203, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 357, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 633, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "kind": "verb_product", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "ranking": [ + "jira.jira_add_worklog", + "jira.jira_add_comment", + "jira.jira_list_projects", + "github.add_comment_to_pending_review", + "jira.jira_create_issue", + "discord.discord_add_reaction", + "memory.add_observations", + "jira.create_webhook", + "git.git_add", + "jira.create_sprint" + ], + "hit1": "jira.jira_add_worklog", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 483, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 892, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "kind": "verb_product", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "ranking": [ + "linear.linear_list_teams", + "linear.linear_list_projects", + "linear.linear_list_issues", + "linear.list_cycles", + "linear.list_users", + "github.get_teams", + "linear.linear_create_issue", + "linear.list_comments", + "linear.linear_create_project", + "linear.list_attachments" + ], + "hit1": "linear.linear_list_teams", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 732, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "ranking": [ + "linear.linear_create_comment", + "linear.linear_get_issue", + "linear.linear_assign_issue", + "linear.linear_update_issue", + "linear.linear_create_issue", + "github.update_issue_comment", + "github.add_issue_comment", + "linear.linear_search_issues", + "linear.get_comment", + "linear.update_comment" + ], + "hit1": "linear.linear_create_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 464, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 845, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "kind": "paraphrase", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "ranking": [ + "filesystem.move_file", + "filesystem.read_media_file", + "filesystem.read_text_file", + "filesystem.read_file", + "filesystem.write_file", + "brave_search.brave_local_search", + "figma.figma_get_file", + "filesystem.edit_file", + "slack.delete_file", + "slack.update_file" + ], + "hit1": "filesystem.move_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 225, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "kind": "paraphrase", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "ranking": [ + "filesystem.search_files", + "google_drive.gdrive_search", + "redis.redis_keys", + "filesystem.directory_tree", + "linear.linear_search_issues", + "memory.search_nodes", + "github.search_repositories", + "slack.list_files", + "github.push_files", + "filesystem.get_file_info" + ], + "hit1": "filesystem.search_files", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 695, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "kind": "paraphrase", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "ranking": [ + "filesystem.edit_file", + "filesystem.move_file", + "filesystem.write_file", + "slack.update_file", + "slack.create_file", + "filesystem.list_directory_with_sizes", + "playwright.browser_tab_new", + "linear.update_attachment", + "slack.delete_file", + "slack.get_file" + ], + "hit1": "filesystem.edit_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 413, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 693, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "ranking": [ + "memory.read_graph", + "memory.delete_relations", + "memory.open_nodes", + "memory.add_observations", + "memory.delete_entities", + "memory.create_entities", + "memory.create_relations", + "filesystem.read_media_file", + "filesystem.read_file", + "memory.search_nodes" + ], + "hit1": "memory.read_graph", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 154, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 533, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "kind": "verb_product", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "memory.open_nodes", + "memory.read_graph", + "memory.delete_entities", + "figma.figma_get_file_nodes", + "filesystem.search_files", + "hubspot.hubspot_search_contacts", + "memory.delete_observations", + "memory.delete_relations", + "github.search_orgs" + ], + "hit1": "memory.search_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 160, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 288, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 568, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "kind": "verb_product", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "ranking": [ + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_diff", + "git.git_show", + "git.git_log", + "git.git_status", + "git.git_reset", + "id_map.json", + "git.git_branch", + "git.git_commit" + ], + "hit1": "git.git_diff_unstaged", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 313, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 523, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "kind": "identifier", + "gold": [ + "git.git_add" + ], + "alts": [], + "ranking": [ + "git.git_add", + "github.update_gist", + "github.create_gist", + "filesystem.list_directory_with_sizes", + "git.git_diff_staged", + "github.add_issue_comment", + "git.git_commit", + "github.add_comment_to_pending_review", + "github.push_files", + "git.git_reset" + ], + "hit1": "git.git_add", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 840, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "kind": "verb_product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "ranking": [ + "shopify.shopify_create_product", + "shopify.shopify_get_product", + "shopify.shopify_update_product", + "shopify.shopify_create_order", + "stripe.stripe_create_product", + "shopify.shopify_get_order", + "shopify.shopify_list_products", + "shopify.shopify_list_customers", + "shopify.shopify_list_orders", + "stripe.stripe_create_price" + ], + "hit1": "shopify.shopify_create_product", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "kind": "verb_product", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "ranking": [ + "stripe.stripe_cancel_subscription", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_list_subscriptions", + "stripe.stripe_create_refund", + "stripe.stripe_get_customer", + "stripe.stripe_list_customers", + "stripe.stripe_create_customer", + "stripe.stripe_create_product" + ], + "hit1": "stripe.stripe_cancel_subscription", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 176, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 290, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 610, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "kind": "resource", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_refund", + "stripe.stripe_create_invoice", + "id_map.json", + "trello.trello_create_board", + "stripe.stripe_create_price", + "hubspot.hubspot_create_deal", + "vercel.vercel_create_project", + "stripe.stripe_create_product" + ], + "hit1": "stripe.stripe_create_payment_intent", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 282, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 636, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "id_map.json", + "sentry.sentry_list_events", + "slack.get_bookmark", + "slack.get_reminder" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 649, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "kind": "verb_product", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "ranking": [ + "datadog.datadog_create_monitor", + "datadog.datadog_get_monitor", + "datadog.datadog_list_monitors", + "datadog.datadog_list_logs", + "datadog.datadog_query_metrics", + "id_map.json", + "memory.add_observations", + "trello.trello_create_board", + "notion.list_data_sources", + "stripe.stripe_create_invoice" + ], + "hit1": "datadog.datadog_create_monitor", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 700, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "kind": "paraphrase", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "ranking": [ + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "sentry.sentry_list_issues", + "sentry.sentry_list_events", + "pagerduty.pagerduty_resolve_incident", + "sentry.sentry_list_projects", + "jira.jira_assign_issue", + "jira.jira_get_issue", + "gitlab.gitlab_get_issue", + "github.assign_copilot_to_issue" + ], + "hit1": "sentry.sentry_get_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 274, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_apply", + "vercel.vercel_get_deployment", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_get_events", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_delete" + ], + "hit1": "kubernetes.k8s_scale_deployment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_apply", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_pod", + "terraform.tf_apply", + "kubernetes.k8s_get_events", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_delete" + ], + "hit1": "kubernetes.k8s_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "ranking": [ + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_run", + "docker.docker_exec", + "docker.docker_images", + "docker.docker_ps", + "docker.docker_build", + "github.list_environments", + "kubernetes.k8s_pod_logs" + ], + "hit1": "docker.docker_stop", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 664, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "kind": "verb_product", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "ranking": [ + "docker.docker_build", + "docker.docker_run", + "docker.docker_images", + "docker.docker_stop", + "docker.docker_ps", + "docker.docker_logs", + "docker.docker_exec", + "docker.docker_rm", + "github.create_environment", + "filesystem.read_media_file" + ], + "hit1": "docker.docker_build", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 642, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "kind": "verb_product", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "ranking": [ + "playwright.browser_click", + "playwright.browser_type", + "puppeteer.puppeteer_click", + "playwright.browser_take_screenshot", + "playwright.browser_tab_list", + "playwright.browser_tab_new", + "playwright.browser_tab_close", + "playwright.browser_press_key", + "playwright.browser_navigate", + "playwright.browser_snapshot" + ], + "hit1": "playwright.browser_click", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 326, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 566, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "kind": "paraphrase", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "ranking": [ + "playwright.browser_type", + "playwright.browser_wait_for", + "playwright.browser_fill_form", + "playwright.browser_tab_new", + "playwright.browser_navigate", + "playwright.browser_press_key", + "playwright.browser_evaluate", + "puppeteer.puppeteer_fill", + "filesystem.read_text_file", + "playwright.browser_click" + ], + "hit1": "playwright.browser_type", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8800937667159342, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 636, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_list_projects", + "gitlab.gitlab_list_issues", + "gitlab.gitlab_list_merge_requests", + "vercel.vercel_list_projects", + "github.projects_list", + "gitlab.list_pipelines", + "gitlab.list_jobs", + "jira.jira_list_projects", + "gitlab.list_snippets", + "sentry.sentry_list_projects" + ], + "hit1": "gitlab.gitlab_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "kind": "verb_product", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "ranking": [ + "bitbucket.bitbucket_create_pr", + "github.create_pull_request", + "github.pull_request_review_write", + "bitbucket.bitbucket_list_prs", + "bitbucket.bitbucket_create_issue", + "github.pull_request_read", + "github.merge_pull_request", + "github.update_pull_request_branch", + "github.update_pull_request", + "github.search_pull_requests" + ], + "hit1": "bitbucket.bitbucket_create_pr", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 439, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 606, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1091, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "kind": "verb_product", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "ranking": [ + "gmail.gmail_create_draft", + "gmail.gmail_list_labels", + "gmail.gmail_get_email", + "gmail.gmail_modify_email", + "gmail.gmail_trash_email", + "gmail.gmail_send_email", + "github.create_pull_request", + "gmail.gmail_search_emails", + "hubspot.hubspot_create_contact", + "stripe.stripe_create_customer" + ], + "hit1": "gmail.gmail_create_draft", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 805, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "kind": "verb_product", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_share_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_update_file", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "google_calendar.gcal_list_calendars", + "filesystem.move_file", + "google_calendar.gcal_delete_event", + "filesystem.read_media_file" + ], + "hit1": "google_drive.gdrive_share_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 271, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "kind": "verb_product", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "google_calendar.gcal_create_event", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "sentry.sentry_list_events", + "kubernetes.k8s_get_events", + "pagerduty.pagerduty_list_oncalls", + "gmail.gmail_list_labels", + "google_drive.gdrive_list_folder" + ], + "hit1": "google_calendar.gcal_list_events", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_create_event", + "linear.delete_label", + "google_calendar.gcal_update_event", + "slack.delete_reminder", + "slack.delete_canvase", + "google_calendar.gcal_list_events", + "github.delete_milestone", + "google_calendar.gcal_list_calendars", + "gitlab.delete_milestone" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 263, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 406, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_projects", + "vercel.vercel_list_deployments", + "jira.jira_list_projects", + "sentry.sentry_list_projects", + "gitlab.gitlab_list_projects", + "vercel.vercel_create_project", + "linear.linear_list_projects", + "vercel.vercel_get_deployment", + "github.projects_list", + "asana.asana_list_tasks" + ], + "hit1": "vercel.vercel_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 253, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "ranking": [ + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update", + "supabase.supabase_list_tables", + "supabase.supabase_invoke_fn", + "memory.delete_observations", + "sqlite.sqlite_query", + "elasticsearch.es_get_doc", + "salesforce.salesforce_query", + "datadog.datadog_list_logs" + ], + "hit1": "supabase.supabase_select", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 689, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "kind": "verb_product", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "figma.figma_get_comments", + "figma.figma_get_file", + "linear.get_comment", + "notion.get_comment", + "slack.slack_post_message", + "figma.figma_get_file_nodes", + "linear.create_comment", + "figma.figma_get_images", + "linear.delete_comment" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 308, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 688, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "kind": "verb_product", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "ranking": [ + "okta.okta_list_users", + "notion.notion_list_users", + "okta.okta_create_user", + "okta.okta_get_user", + "notion.list_users_tokens", + "okta.okta_deactivate_user", + "slack.slack_list_users", + "linear.list_users", + "slack.slack_get_users", + "github.list_repository_collaborators" + ], + "hit1": "okta.okta_list_users", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 161, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 634, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query_range", + "prometheus.prom_query", + "prometheus.prom_list_metrics", + "datadog.datadog_query_metrics", + "postgres.query", + "datadog.datadog_list_monitors", + "postgres.explain_query", + "gmail.gmail_search_emails", + "mysql.mysql_query", + "memory.search_nodes" + ], + "hit1": "prometheus.prom_query_range", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 674, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "kind": "verb_product", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "ranking": [ + "grafana.grafana_create_annotation", + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "id_map.json", + "slack.create_canvase", + "trello.trello_create_board", + "gitlab.create_snippet", + "stripe.stripe_create_invoice", + "github.create_artifact", + "vercel.vercel_create_project" + ], + "hit1": "grafana.grafana_create_annotation", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 326, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 692, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "kind": "identifier", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "ranking": [ + "terraform.tf_apply", + "terraform.tf_plan", + "kubernetes.k8s_apply", + "terraform.tf_state_list", + "id_map.json", + "github.update_environment", + "cloudflare.cf_create_dns", + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "playwright.browser_fill_form" + ], + "hit1": "terraform.tf_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 238, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 562, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "kind": "verb_product", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "ranking": [ + "asana.asana_list_tasks", + "asana.asana_get_task", + "clickup.clickup_list_tasks", + "asana.asana_create_task", + "asana.asana_create_project", + "asana.asana_update_task", + "id_map.json", + "okta.okta_list_users", + "vercel.vercel_list_projects", + "pagerduty.pagerduty_list_incidents" + ], + "hit1": "asana.asana_list_tasks", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 624, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "ranking": [ + "trello.trello_add_comment", + "trello.trello_create_card", + "trello.trello_list_cards", + "trello.trello_create_board", + "figma.figma_post_comment", + "linear.get_comment", + "notion.get_comment", + "github.update_issue_comment", + "linear.update_comment", + "linear.delete_comment" + ], + "hit1": "trello.trello_add_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 715, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_update_task", + "clickup.clickup_get_task", + "clickup.clickup_list_tasks", + "clickup.clickup_create_task", + "asana.asana_update_task", + "puppeteer.puppeteer_click", + "playwright.browser_click", + "asana.asana_get_task", + "gitlab.update_snippet", + "slack.update_bookmark" + ], + "hit1": "clickup.clickup_update_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "kind": "verb_product", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "ranking": [ + "confluence.confluence_search", + "confluence.confluence_get_page", + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "grafana.grafana_search_dashboards", + "github.search_orgs", + "github.search_users", + "hubspot.hubspot_search_contacts", + "github.search_pull_requests", + "github.search_commits" + ], + "hit1": "confluence.confluence_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 636, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_deal", + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_search_contacts", + "hubspot.hubspot_list_contacts", + "github.create_gist", + "slack.create_usergroup", + "github.create_repository", + "slack.get_usergroup", + "github.create_branch" + ], + "hit1": "hubspot.hubspot_create_deal", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_search", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_create_ticket", + "id_map.json", + "google_calendar.gcal_list_calendars", + "clickup.clickup_list_tasks", + "jira.list_boards", + "vercel.vercel_list_projects" + ], + "hit1": "zendesk.zendesk_list_tickets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 573, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "kind": "verb_product", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "ranking": [ + "discord.discord_list_channels", + "slack.slack_list_channels", + "discord.discord_get_channel", + "discord.discord_list_members", + "discord.discord_send_message", + "discord.discord_add_reaction", + "slack.list_usergroups", + "id_map.json", + "slack.slack_get_channel_history", + "clickup.clickup_list_tasks" + ], + "hit1": "discord.discord_list_channels", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 594, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "ranking": [ + "redis.redis_set", + "redis.redis_get", + "redis.redis_delete", + "redis.redis_keys", + "playwright.browser_press_key", + "slack.slack_set_channel_topic", + "slack.update_pin", + "id_map.json", + "clickup.clickup_update_task", + "jira.jira_get_issue" + ], + "hit1": "redis.redis_set", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 563, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert", + "mongodb.mongo_update", + "mongodb.mongo_delete", + "mongodb.mongo_find", + "supabase.supabase_insert", + "mongodb.mongo_list_collections", + "id_map.json", + "aws_s3.s3_put_object", + "salesforce.salesforce_create_record", + "elasticsearch.es_index_doc" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 689, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "kind": "verb_product", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "ranking": [ + "mysql.mysql_query", + "postgres.query", + "sqlite.sqlite_query", + "mysql.mysql_list_tables", + "salesforce.salesforce_query", + "postgres.explain_query", + "supabase.supabase_select", + "github.delete_check_run", + "github.get_check_run", + "fetch.fetch" + ], + "hit1": "mysql.mysql_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 286, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 594, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "elasticsearch.es_search", + "playwright.browser_tab_close", + "mongodb.mongo_find", + "id_map.json", + "confluence.confluence_search", + "supabase.supabase_list_tables", + "google_drive.gdrive_search", + "jira.get_attachment" + ], + "hit1": "elasticsearch.es_index_doc", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 567, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_put_object", + "aws_s3.s3_list_objects", + "aws_s3.s3_list_buckets", + "asana.asana_get_task", + "okta.okta_get_user", + "gitlab.get_snippet", + "linear.get_attachment", + "discord.discord_get_channel" + ], + "hit1": "aws_s3.s3_get_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 624, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones", + "cloudflare.cf_list_dns", + "cloudflare.cf_purge_cache", + "cloudflare.cf_create_dns", + "id_map.json", + "aws_s3.s3_list_buckets", + "clickup.clickup_list_tasks", + "kubernetes.k8s_list_services", + "vercel.vercel_list_projects", + "kubernetes.k8s_list_deployments" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 285, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 525, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_create_product", + "stripe.stripe_create_refund", + "stripe.stripe_create_customer", + "stripe.stripe_create_payment_intent", + "stripe.stripe_list_invoices", + "stripe.stripe_create_price", + "stripe.stripe_get_payment_intent", + "stripe.stripe_get_customer", + "stripe.stripe_cancel_subscription" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "kind": "verb_product", + "gold": [ + "github.actions_list" + ], + "alts": [], + "ranking": [ + "github.actions_list", + "github.actions_get", + "github.actions_get_job_logs", + "github.actions_run_trigger", + "github.projects_list", + "github.list_notifications", + "github.list_gists", + "github.list_milestones", + "github.list_starred_repositories", + "github.list_hooks" + ], + "hit1": "github.actions_list", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 453, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 778, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "kind": "verb_product", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "ranking": [ + "github.actions_get_job_logs", + "github.actions_get", + "github.actions_list", + "github.actions_run_trigger", + "gitlab.get_job", + "github.get_check_run", + "github.projects_get", + "github.get_notification_details", + "github.get_gist", + "git.git_log" + ], + "hit1": "github.actions_get_job_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 442, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 730, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "kind": "verb_product", + "gold": [ + "github.create_repository" + ], + "alts": [], + "ranking": [ + "github.create_repository", + "github.create_branch", + "github.create_or_update_file", + "github.create_issue", + "github.create_gist", + "github.create_environment", + "git.git_create_branch", + "github.star_repository", + "github.create_package", + "github.unstar_repository" + ], + "hit1": "github.create_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 346, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 556, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 968, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "kind": "verb_product", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "ranking": [ + "git.git_checkout", + "git.git_branch", + "git.git_create_branch", + "github.create_branch", + "github.get_check_run", + "git.git_status", + "gitlab.gitlab_create_merge_request", + "git.git_show", + "github.list_commits", + "github.list_branches" + ], + "hit1": "git.git_checkout", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 771, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "kind": "identifier", + "gold": [ + "git.git_log" + ], + "alts": [], + "ranking": [ + "git.git_log", + "git.git_show", + "git.git_status", + "git.git_diff_unstaged", + "git.git_diff", + "git.git_branch", + "git.git_diff_staged", + "github.actions_get_job_logs", + "git.git_commit", + "git.git_checkout" + ], + "hit1": "git.git_log", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 629, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "kind": "verb_product", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "ranking": [ + "postgres.list_schemas", + "postgres.list_tables", + "postgres.describe_table", + "vercel.vercel_list_projects", + "postgres.explain_query", + "mysql.mysql_list_tables", + "postgres.query", + "id_map.json", + "supabase.supabase_list_tables", + "clickup.clickup_list_tasks" + ], + "hit1": "postgres.list_schemas", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 169, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 455, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "kind": "verb_product", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "ranking": [ + "memory.open_nodes", + "memory.search_nodes", + "memory.read_graph", + "memory.create_entities", + "memory.create_relations", + "memory.add_observations", + "memory.delete_relations", + "memory.delete_entities", + "figma.figma_get_file_nodes", + "notion.notion_search" + ], + "hit1": "memory.open_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 564, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "kind": "paraphrase", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "ranking": [ + "filesystem.get_file_info", + "google_drive.gdrive_get_file", + "filesystem.read_file", + "filesystem.read_media_file", + "figma.figma_get_file_nodes", + "filesystem.read_text_file", + "github.get_file_contents", + "gitlab.gitlab_get_file", + "filesystem.list_directory", + "filesystem.directory_tree" + ], + "hit1": "filesystem.get_file_info", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "kind": "paraphrase", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "ranking": [ + "filesystem.write_file", + "filesystem.edit_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "filesystem.read_file", + "slack.create_file", + "filesystem.create_directory", + "brave_search.brave_local_search", + "google_drive.gdrive_create_file", + "github.issue_write" + ], + "hit1": "filesystem.write_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 856, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "kind": "verb_product", + "gold": [ + "github.search_issues" + ], + "alts": [], + "ranking": [ + "github.search_issues", + "github.search_users", + "github.search_orgs", + "github.list_issues", + "github.search_pull_requests", + "github.search_commits", + "github.search_repositories", + "github.search_code", + "linear.linear_search_issues", + "jira.jira_search_issues" + ], + "hit1": "github.search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "kind": "paraphrase", + "gold": [ + "github.get_me" + ], + "alts": [], + "ranking": [ + "github.get_teams", + "github.get_me", + "github.list_notifications", + "slack.slack_get_user_profile", + "github.list_gists", + "github.get_latest_release", + "github.get_secret_scanning_alert", + "okta.okta_get_user", + "github.projects_get", + "linear.get_user" + ], + "hit1": "github.get_teams", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 285, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 625, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "cost": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.0866666666666667, + "p95_calls": 1.0, + "success_within_1": 0.9866666666666667, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 252.98666666666668, + "p95_tokens": 345.54999999999995, + "total_tokens": 37948, + "tokens_per_successful_query": 252.98666666666668, + "total_tokens_successful": 37948 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.0466666666666666, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 393.0466666666667, + "p95_tokens": 538.1999999999999, + "total_tokens": 58957, + "tokens_per_successful_query": 393.0466666666667, + "total_tokens_successful": 58957 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.02, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 729.4866666666667, + "p95_tokens": 979.3999999999999, + "total_tokens": 109423, + "tokens_per_successful_query": 729.4866666666667, + "total_tokens_successful": 109423 + } + } +} diff --git a/experiments/mcp-tool-search/results/hybrid_codedb_trigram_bm25.csv b/experiments/mcp-tool-search/results/hybrid_codedb_trigram_bm25.csv new file mode 100644 index 00000000..93ea001b --- /dev/null +++ b/experiments/mcp-tool-search/results/hybrid_codedb_trigram_bm25.csv @@ -0,0 +1,151 @@ +id,query,kind,hit1,recall@1,recall@3,recall@5,mrr@10,ndcg@10,hit_rank,k3_calls,k3_tokens,k3_success1,k5_calls,k5_tokens,k5_success1,k10_calls,k10_tokens,k10_success1,ranking +q001,Create a GitHub issue,verb_product,github.issue_write,1.0,1.0,1.0,1.0000,0.7075,1,1,264,1.0,1,495,1.0,1,884,1.0,github.issue_write git.git_show git.git_branch github.create_issue github.issue_read jira.jira_get_issue github.add_issue_comment git.git_diff_unstaged github.update_issue_comment gitlab.gitlab_get_issue +q002,Create a Jira ticket,verb_product,jira.jira_search_issues,0.0,0.0,1.0,0.2500,0.4307,4,2,471,0.0,1,383,1.0,1,774,1.0,jira.jira_search_issues jira.jira_get_issue jira.jira_list_projects jira.jira_create_issue jira.jira_add_worklog jira.jira_add_comment zendesk.zendesk_create_ticket jira.jira_create_sprint jira.jira_assign_issue zendesk.zendesk_get_ticket +q003,Send a Slack message,verb_product,slack.slack_post_message,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,366,1.0,1,734,1.0,slack.slack_post_message slack.slack_add_reaction slack.slack_search_messages slack.slack_list_users slack.slack_join_channel discord.discord_send_message slack.slack_leave_channel slack.slack_create_channel discord.discord_add_reaction slack.slack_reply_to_thread +q004,List open pull requests,resource,github.list_pull_requests,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,519,1.0,1,1035,1.0,github.list_pull_requests github.search_pull_requests gitlab.gitlab_list_merge_requests github.pull_request_review_write github.update_pull_request github.pull_request_read github.merge_pull_request github.update_pull_request_branch bitbucket.bitbucket_list_prs github.create_pull_request +q005,Open a pull request on GitHub,verb_product,github.create_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,458,1.0,1,643,1.0,1,1013,1.0,github.create_pull_request github.pull_request_review_write github.update_pull_request github.pull_request_read github.update_pull_request_branch github.merge_pull_request gitlab.gitlab_get_merge_request git.git_show gitlab.gitlab_merge_merge_request git.git_branch +q006,Run a SQL query against postgres,verb_product,postgres.query,1.0,1.0,1.0,1.0000,1.0000,1,1,167,1.0,1,272,1.0,1,544,1.0,postgres.query postgres.explain_query postgres.list_schemas mysql.mysql_query sqlite.sqlite_query postgres.list_tables postgres.describe_table salesforce.salesforce_query mysql.mysql_list_tables prometheus.prom_query +q007,Read a local file from disk,paraphrase,filesystem.read_media_file,0.0,1.0,1.0,0.3333,0.5068,3,1,213,1.0,1,418,1.0,1,813,1.0,filesystem.read_media_file brave_search.brave_local_search filesystem.read_file gitlab.gitlab_get_file github.delete_file google_drive.gdrive_get_file filesystem.read_text_file google_drive.gdrive_update_file filesystem.write_file filesystem.edit_file +q008,What files are in this folder,paraphrase,filesystem.search_files,0.0,0.0,0.0,0.1111,0.1923,9,3,690,0.0,2,753,0.0,1,731,1.0,filesystem.search_files github.push_files slack.list_files google_drive.gdrive_list_folder jira.update_sprint linear.list_users filesystem.move_file gitlab.get_pipeline filesystem.list_directory elasticsearch.es_index_doc +q009,Remember this fact in the knowledge graph,paraphrase,github.create_artifact,0.0,0.0,1.0,0.2500,0.3834,4,2,383,0.0,1,311,1.0,1,612,1.0,github.create_artifact memory.read_graph github.delete_artifact memory.create_entities github.get_artifact memory.delete_relations github.list_artifacts memory.create_relations github.update_artifact memory.add_observations +q010,Search the web with Brave,verb_product,brave_search.brave_web_search,1.0,1.0,1.0,1.0000,1.0000,1,1,245,1.0,1,317,1.0,1,653,1.0,brave_search.brave_web_search brave_search.brave_local_search filesystem.list_directory_with_sizes id_map.json jira.jira_search_issues jira.delete_webhook github.update_pull_request_branch jira.get_webhook memory.search_nodes jira.list_webhooks +q011,Fetch the contents of a URL,verb_product,fetch.fetch,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,388,1.0,1,653,1.0,fetch.fetch playwright.browser_navigate github.get_file_contents slack.slack_set_channel_topic puppeteer.puppeteer_navigate playwright.browser_snapshot filesystem.read_file git.git_add filesystem.list_directory playwright.browser_tab_new +q012,Show git status,identifier,git.git_status,1.0,1.0,1.0,1.0000,1.0000,1,1,178,1.0,1,283,1.0,1,639,1.0,git.git_status git.git_show git.git_branch git.git_diff_unstaged git.git_diff_staged git.git_diff sentry.sentry_update_issue git.git_checkout git.git_commit github.list_check_runs +q013,Commit staged changes,paraphrase,git.git_commit,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,328,1.0,1,611,1.0,git.git_commit git.git_diff_staged github.get_commit git.git_reset git.git_log git.git_show github.push_files id_map.json git.git_diff_unstaged github.search_commits +q014,Create a Linear issue,verb_product,linear.linear_create_project,0.0,0.0,1.0,0.2500,0.4307,4,2,543,0.0,1,471,1.0,1,840,1.0,linear.linear_create_project linear.linear_update_issue linear.linear_get_issue linear.linear_create_issue linear.linear_create_comment jira.jira_get_issue linear.linear_assign_issue github.issue_write gitlab.gitlab_get_issue linear.linear_list_teams +q015,Make a task in my project tracker Linear,paraphrase,linear.linear_create_project,0.0,1.0,1.0,0.3333,0.5000,3,1,290,1.0,1,451,1.0,1,733,1.0,linear.linear_create_project asana.asana_get_task linear.linear_create_issue mysql.mysql_list_tables linear.linear_update_issue mysql.mysql_query linear.linear_list_teams clickup.clickup_update_task linear.linear_list_projects vercel.vercel_create_project +q016,Post a comment on a GitHub issue,verb_product,github.update_issue_comment,0.0,0.0,1.0,0.2000,0.3869,5,2,537,0.0,1,414,1.0,1,971,1.0,github.update_issue_comment git.git_show git.git_branch figma.figma_post_comment github.add_issue_comment github.issue_read github.get_discussion_comments github.issue_write notion.create_comment github.create_issue +q017,Merge the pull request,ambiguous,gitlab.gitlab_get_merge_request,0.0,1.0,1.0,0.5000,0.6697,2,1,283,1.0,1,516,1.0,1,1093,1.0,gitlab.gitlab_get_merge_request github.merge_pull_request gitlab.gitlab_merge_merge_request github.pull_request_review_write github.pull_request_read gitlab.gitlab_create_merge_request github.update_pull_request github.update_pull_request_branch github.create_pull_request gitlab.gitlab_list_merge_requests +q018,Trigger a GitHub Actions workflow,verb_product,github.actions_run_trigger,1.0,1.0,1.0,1.0000,1.0000,1,1,296,1.0,1,449,1.0,1,733,1.0,github.actions_run_trigger git.git_show github.actions_get_job_logs github.actions_list github.actions_get git.git_branch git.git_status circleci.circleci_trigger_pipeline git.git_checkout git.git_diff_unstaged +q019,Get Dependabot alerts,resource,github.list_dependabot_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,287,1.0,1,367,1.0,1,727,1.0,github.list_dependabot_alerts github.get_dependabot_alert github.list_code_scanning_alerts github.list_secret_scanning_alerts id_map.json kubernetes.k8s_get_pod redis.redis_get github.get_discussion github.projects_get github.get_label +q020,List Sentry issues,verb_product,sentry.sentry_list_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,268,1.0,1,426,1.0,1,813,1.0,sentry.sentry_list_issues sentry.sentry_list_projects github.list_issues sentry.sentry_get_issue sentry.sentry_update_issue sentry.sentry_list_events github.search_issues linear.linear_list_issues jira.jira_search_issues gitlab.gitlab_list_issues +q021,Create a Stripe customer,verb_product,stripe.stripe_create_invoice,0.0,1.0,1.0,0.5000,0.6309,2,1,186,1.0,1,331,1.0,1,658,1.0,stripe.stripe_create_invoice stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_list_customers stripe.stripe_create_price stripe.stripe_create_product stripe.stripe_list_invoices stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent stripe.stripe_list_subscriptions +q022,Refund a Stripe payment,paraphrase,stripe.stripe_create_refund,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,347,1.0,1,613,1.0,stripe.stripe_create_refund stripe.stripe_get_payment_intent stripe.stripe_create_payment_intent stripe.stripe_list_customers stripe.stripe_create_invoice stripe.stripe_create_price stripe.stripe_create_product stripe.stripe_create_customer stripe.stripe_get_customer id_map.json +q023,Send an email with Gmail,verb_product,gmail.gmail_send_email,1.0,1.0,1.0,1.0000,1.0000,1,1,262,1.0,1,377,1.0,1,668,1.0,gmail.gmail_send_email gmail.gmail_modify_email gmail.gmail_get_email gmail.gmail_trash_email filesystem.list_directory_with_sizes gmail.gmail_list_labels puppeteer.puppeteer_click gmail.gmail_search_emails gmail.gmail_create_draft asana.asana_get_task +q024,Search my Gmail inbox,verb_product,gmail.gmail_send_email,0.0,0.0,0.0,0.1429,0.3333,7,3,642,0.0,2,711,0.0,1,691,1.0,gmail.gmail_send_email gmail.gmail_list_labels gmail.gmail_modify_email gmail.gmail_trash_email gmail.gmail_get_email mysql.mysql_list_tables gmail.gmail_search_emails gmail.gmail_create_draft mysql.mysql_query brave_search.brave_web_search +q025,Upload a file to Google Drive,paraphrase,google_drive.gdrive_get_file,0.0,0.0,1.0,0.2000,0.3869,5,2,526,0.0,1,415,1.0,1,907,1.0,google_drive.gdrive_get_file google_drive.gdrive_update_file slack.slack_upload_file google_drive.gdrive_search google_drive.gdrive_create_file slack.slack_reply_to_thread gitlab.gitlab_get_file google_drive.gdrive_share_file github.add_comment_to_pending_review google_calendar.gcal_delete_event +q026,Create a calendar event,resource,google_calendar.gcal_delete_event,0.0,1.0,1.0,0.5000,0.6309,2,1,300,1.0,1,427,1.0,1,805,1.0,google_calendar.gcal_delete_event google_calendar.gcal_create_event google_calendar.gcal_update_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars id_map.json github.create_gist kubernetes.k8s_get_events github.create_milestone github.pull_request_review_write +q027,List Kubernetes pods,verb_product,kubernetes.k8s_list_pods,1.0,1.0,1.0,1.0000,1.0000,1,1,230,1.0,1,370,1.0,1,665,1.0,kubernetes.k8s_list_pods kubernetes.k8s_list_deployments kubernetes.k8s_scale_deployment kubernetes.k8s_list_services kubernetes.k8s_delete kubernetes.k8s_pod_logs kubernetes.k8s_get_events kubernetes.k8s_get_pod kubernetes.k8s_apply id_map.json +q028,Get logs from a k8s pod,verb_product,kubernetes.k8s_pod_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,455,1.0,1,759,1.0,kubernetes.k8s_pod_logs kubernetes.k8s_get_pod github.actions_get_job_logs docker.docker_logs datadog.datadog_list_logs kubernetes.k8s_list_pods git.git_log id_map.json aws_s3.s3_get_object github.delete_file +q029,Run a docker container,verb_product,docker.docker_rm,0.0,1.0,1.0,0.5000,0.6309,2,1,219,1.0,1,348,1.0,1,688,1.0,docker.docker_rm docker.docker_run docker.docker_stop docker.docker_exec docker.docker_build docker.docker_logs docker.docker_ps docker.docker_images kubernetes.k8s_pod_logs github.create_check_run +q030,Show docker container logs,verb_product,docker.docker_rm,0.0,1.0,1.0,0.5000,0.6309,2,1,188,1.0,1,345,1.0,1,747,1.0,docker.docker_rm docker.docker_logs docker.docker_stop docker.docker_run docker.docker_exec kubernetes.k8s_pod_logs docker.docker_build docker.docker_ps datadog.datadog_list_logs github.actions_get_job_logs +q031,Navigate the browser to a URL,ambiguous,playwright.browser_navigate,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,265,1.0,1,742,1.0,playwright.browser_navigate puppeteer.puppeteer_navigate playwright.browser_tab_new playwright.browser_tab_list playwright.browser_tab_close playwright.browser_wait_for github.add_comment_to_pending_review slack.slack_reply_to_thread playwright.browser_click github.assign_copilot_to_issue +q032,Take a screenshot of the page,ambiguous,playwright.browser_take_screenshot,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,338,1.0,1,609,1.0,playwright.browser_take_screenshot puppeteer.puppeteer_screenshot playwright.browser_snapshot slack.slack_set_channel_topic notion.notion_get_page notion.notion_update_page id_map.json confluence.confluence_get_page filesystem.list_directory confluence.confluence_create_page +q033,Create a GitLab merge request,verb_product,gitlab.gitlab_get_merge_request,0.0,1.0,1.0,0.3333,0.5000,3,1,297,1.0,1,475,1.0,1,841,1.0,gitlab.gitlab_get_merge_request gitlab.gitlab_merge_merge_request gitlab.gitlab_create_merge_request gitlab.gitlab_list_merge_requests github.merge_pull_request git.git_show git.git_branch git.git_checkout git.git_diff_unstaged github.pull_request_review_write +q034,Search Notion pages,verb_product,notion.notion_search,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,386,1.0,1,686,1.0,notion.notion_search notion.notion_create_database notion.notion_query_database notion.notion_append_block notion.notion_get_comments notion.notion_get_page notion.notion_get_block notion.notion_update_page notion.notion_list_users notion.notion_create_page +q035,Create a Notion page,verb_product,notion.notion_update_page,0.0,0.0,0.0,0.1250,0.3155,8,3,695,0.0,2,727,0.0,1,706,1.0,notion.notion_update_page notion.notion_get_page notion.notion_search notion.notion_get_comments notion.notion_append_block notion.notion_create_database notion.notion_query_database notion.notion_create_page confluence.confluence_get_page notion.notion_get_block +q036,Query a Notion database,verb_product,notion.notion_query_database,1.0,1.0,1.0,1.0000,1.0000,1,1,261,1.0,1,387,1.0,1,730,1.0,notion.notion_query_database notion.notion_create_database notion.notion_search notion.notion_append_block notion.notion_get_comments notion.notion_create_page notion.notion_get_page mongodb.mongo_delete notion.notion_get_block mongodb.mongo_find +q037,Create a Zendesk ticket,verb_product,zendesk.zendesk_get_ticket,0.0,1.0,1.0,0.5000,0.6309,2,1,257,1.0,1,368,1.0,1,761,1.0,zendesk.zendesk_get_ticket zendesk.zendesk_create_ticket zendesk.zendesk_update_ticket zendesk.zendesk_search zendesk.zendesk_list_tickets jira.jira_create_issue id_map.json github.create_gist github.create_milestone github.create_hook +q038,Create a HubSpot contact,verb_product,hubspot.hubspot_create_contact,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,364,1.0,1,779,1.0,hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_list_contacts hubspot.hubspot_create_deal hubspot.hubspot_search_contacts id_map.json github.create_gist github.delete_file github.create_repository github.pull_request_review_write +q039,Create a Salesforce lead,verb_product,salesforce.salesforce_create_lead,1.0,1.0,1.0,1.0000,1.0000,1,1,273,1.0,1,423,1.0,1,682,1.0,salesforce.salesforce_create_lead salesforce.salesforce_convert_lead salesforce.salesforce_update_record salesforce.salesforce_get_record salesforce.salesforce_delete_record salesforce.salesforce_list_opportunities salesforce.salesforce_create_record salesforce.salesforce_query id_map.json github.create_gist +q040,Run a SOQL query,identifier,salesforce.salesforce_query,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,291,1.0,1,724,1.0,salesforce.salesforce_query postgres.query sqlite.sqlite_query mysql.mysql_query prometheus.prom_query docker.docker_run postgres.explain_query prometheus.prom_query_range datadog.datadog_query_metrics notion.notion_query_database +q041,List Shopify orders,verb_product,shopify.shopify_list_orders,1.0,1.0,1.0,1.0000,1.0000,1,1,230,1.0,1,376,1.0,1,606,1.0,shopify.shopify_list_orders shopify.shopify_list_products shopify.shopify_update_product shopify.shopify_create_product shopify.shopify_get_product shopify.shopify_list_customers id_map.json shopify.shopify_get_order shopify.shopify_create_order clickup.clickup_list_tasks +q042,Create a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,0.0,1.0,1.0,0.3333,0.5000,3,1,259,1.0,1,403,1.0,1,838,1.0,pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls github.create_gist github.create_milestone github.create_hook github.create_environment +q043,Who is on call in PagerDuty,paraphrase,pagerduty.pagerduty_list_oncalls,1.0,1.0,1.0,1.0000,1.0000,1,1,213,1.0,1,346,1.0,1,749,1.0,pagerduty.pagerduty_list_oncalls filesystem.list_allowed_directories twilio.twilio_make_call notion.notion_get_comments pagerduty.pagerduty_list_incidents pagerduty.pagerduty_get_incident github.get_discussion_comments pagerduty.pagerduty_create_incident slack.slack_create_channel pagerduty.pagerduty_acknowledge_incident +q044,Query Datadog metrics,verb_product,datadog.datadog_query_metrics,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,449,1.0,1,668,1.0,datadog.datadog_query_metrics datadog.datadog_create_monitor datadog.datadog_list_monitors datadog.datadog_list_logs prometheus.prom_list_metrics datadog.datadog_get_monitor id_map.json prometheus.prom_query postgres.query postgres.explain_query +q045,Search Datadog logs,verb_product,datadog.datadog_list_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,284,1.0,1,493,1.0,1,863,1.0,datadog.datadog_list_logs datadog.datadog_query_metrics datadog.datadog_list_monitors datadog.datadog_create_monitor github.actions_get_job_logs datadog.datadog_get_monitor kubernetes.k8s_pod_logs docker.docker_logs git.git_log brave_search.brave_web_search +q046,List S3 buckets,verb_product,aws_s3.s3_list_buckets,1.0,1.0,1.0,1.0000,1.0000,1,1,133,1.0,1,262,1.0,1,590,1.0,aws_s3.s3_list_buckets id_map.json aws_s3.s3_list_objects clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments kubernetes.k8s_list_services github.projects_list github.list_milestones github.list_hooks +q047,Upload an object to S3,paraphrase,aws_s3.s3_put_object,1.0,1.0,1.0,1.0000,1.0000,1,1,262,1.0,1,487,1.0,1,857,1.0,aws_s3.s3_put_object slack.slack_upload_file aws_s3.s3_delete_object aws_s3.s3_get_object github.add_comment_to_pending_review slack.slack_reply_to_thread github.assign_copilot_to_issue slack.slack_add_reaction id_map.json twilio.twilio_send_sms +q048,Purge Cloudflare cache,verb_product,cloudflare.cf_purge_cache,1.0,1.0,1.0,1.0000,1.0000,1,1,231,1.0,1,286,1.0,1,643,1.0,cloudflare.cf_purge_cache cloudflare.cf_list_zones cloudflare.cf_create_dns cloudflare.cf_list_dns id_map.json +q049,Create a Cloudflare DNS record,verb_product,cloudflare.cf_create_dns,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,363,1.0,1,660,1.0,cloudflare.cf_create_dns cloudflare.cf_list_dns cloudflare.cf_list_zones cloudflare.cf_purge_cache salesforce.salesforce_get_record salesforce.salesforce_update_record salesforce.salesforce_create_record salesforce.salesforce_delete_record git.git_commit id_map.json +q050,List Vercel deployments,verb_product,vercel.vercel_list_deployments,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,291,1.0,1,513,1.0,vercel.vercel_list_deployments vercel.vercel_list_projects vercel.vercel_get_deployment kubernetes.k8s_list_deployments vercel.vercel_create_project id_map.json clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_services aws_s3.s3_list_buckets +q051,Insert a row into Supabase,verb_product,supabase.supabase_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,389,1.0,1,620,1.0,supabase.supabase_insert supabase.supabase_select supabase.supabase_update mongodb.mongo_insert supabase.supabase_invoke_fn supabase.supabase_list_tables playwright.browser_type id_map.json playwright.browser_click playwright.browser_evaluate +q052,Export images from Figma,paraphrase,figma.figma_get_images,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,325,1.0,1,730,1.0,figma.figma_get_images figma.figma_get_file docker.docker_images figma.figma_post_comment figma.figma_get_comments figma.figma_get_file_nodes github.delete_file elasticsearch.es_get_doc mongodb.mongo_delete aws_s3.s3_get_object +q053,Send an SMS with Twilio,verb_product,twilio.twilio_send_sms,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,362,1.0,1,665,1.0,twilio.twilio_send_sms twilio.twilio_list_messages twilio.twilio_make_call filesystem.list_directory_with_sizes puppeteer.puppeteer_click id_map.json gmail.gmail_send_email asana.asana_get_task puppeteer.puppeteer_fill asana.asana_update_task +q054,Create an Okta user,verb_product,okta.okta_deactivate_user,0.0,1.0,1.0,0.3333,0.5000,3,1,183,1.0,1,285,1.0,1,657,1.0,okta.okta_deactivate_user okta.okta_get_user okta.okta_create_user okta.okta_list_users linear.get_user slack.slack_get_user_profile asana.asana_create_task linear.create_user asana.asana_get_task linear.update_user +q055,Instant query Prometheus,verb_product,prometheus.prom_query,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,274,1.0,1,595,1.0,prometheus.prom_query prometheus.prom_list_metrics prometheus.prom_query_range id_map.json postgres.query postgres.explain_query salesforce.salesforce_query sqlite.sqlite_query mysql.mysql_query datadog.datadog_query_metrics +q056,Search Grafana dashboards,verb_product,grafana.grafana_search_dashboards,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,306,1.0,1,624,1.0,grafana.grafana_search_dashboards grafana.grafana_create_annotation grafana.grafana_get_dashboard id_map.json brave_search.brave_web_search brave_search.brave_local_search confluence.confluence_search github.search_issues github.search_users github.search_orgs +q057,Trigger a CircleCI pipeline,verb_product,circleci.circleci_trigger_pipeline,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,390,1.0,1,664,1.0,circleci.circleci_trigger_pipeline circleci.circleci_get_pipeline circleci.circleci_list_pipelines gitlab.gitlab_create_pipeline github.actions_run_trigger gitlab.create_pipeline gitlab.get_pipeline id_map.json gitlab.update_pipeline gitlab.delete_pipeline +q058,Run terraform plan,identifier,terraform.tf_plan,1.0,1.0,1.0,1.0000,1.0000,1,1,172,1.0,1,227,1.0,1,598,1.0,terraform.tf_plan terraform.tf_apply terraform.tf_state_list postgres.explain_query id_map.json docker.docker_run github.create_check_run github.get_check_run github.update_check_run github.delete_check_run +q059,Create an Asana task,verb_product,asana.asana_get_task,0.0,1.0,1.0,0.3333,0.5000,3,1,267,1.0,1,405,1.0,1,783,1.0,asana.asana_get_task asana.asana_update_task asana.asana_create_task asana.asana_create_project asana.asana_list_tasks clickup.clickup_get_task clickup.clickup_update_task clickup.clickup_create_task slack.slack_add_reaction okta.okta_create_user +q060,Create a Trello card,verb_product,trello.trello_create_card,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,316,1.0,1,784,1.0,trello.trello_create_card trello.trello_list_cards trello.trello_add_comment trello.trello_create_board id_map.json github.create_gist github.create_milestone github.create_hook github.create_environment github.create_package +q061,Create a ClickUp task,verb_product,clickup.clickup_update_task,0.0,1.0,1.0,0.3333,0.5000,3,1,257,1.0,1,362,1.0,1,751,1.0,clickup.clickup_update_task clickup.clickup_get_task clickup.clickup_create_task clickup.clickup_list_tasks asana.asana_get_task puppeteer.puppeteer_click playwright.browser_click asana.asana_update_task asana.asana_create_task github.create_gist +q062,Create a Confluence page,verb_product,confluence.confluence_get_page,0.0,1.0,1.0,0.5000,0.6309,2,1,260,1.0,1,367,1.0,1,703,1.0,confluence.confluence_get_page confluence.confluence_create_page confluence.confluence_update_page confluence.confluence_search notion.notion_get_page notion.notion_update_page notion.notion_create_page playwright.browser_take_screenshot notion.notion_get_comments playwright.browser_press_key +q063,Send a Discord message,verb_product,discord.discord_add_reaction,0.0,1.0,1.0,0.5000,0.6309,2,1,224,1.0,1,343,1.0,1,776,1.0,discord.discord_add_reaction discord.discord_send_message discord.discord_get_channel slack.slack_post_message discord.discord_list_channels discord.discord_list_members slack.slack_add_reaction gmail.gmail_send_email github.delete_file twilio.twilio_send_sms +q064,Search Elasticsearch,verb_product,elasticsearch.es_get_doc,0.0,1.0,1.0,0.5000,0.6309,2,1,239,1.0,1,328,1.0,1,646,1.0,elasticsearch.es_get_doc elasticsearch.es_search elasticsearch.es_index_doc id_map.json brave_search.brave_web_search brave_search.brave_local_search confluence.confluence_search github.search_issues github.search_users github.search_orgs +q065,Get a Redis key,verb_product,redis.redis_get,1.0,1.0,1.0,1.0000,1.0000,1,1,185,1.0,1,280,1.0,1,586,1.0,redis.redis_get redis.redis_set redis.redis_keys redis.redis_delete playwright.browser_press_key jira.jira_get_issue figma.figma_get_comments figma.figma_post_comment aws_s3.s3_get_object figma.figma_get_file +q066,Find MongoDB documents,verb_product,mongodb.mongo_insert,0.0,1.0,1.0,0.5000,0.6309,2,1,244,1.0,1,381,1.0,1,738,1.0,mongodb.mongo_insert mongodb.mongo_find mongodb.mongo_delete mongodb.mongo_update mongodb.mongo_list_collections +q067,List SQLite tables,verb_product,sqlite.sqlite_list_tables,1.0,1.0,1.0,1.0000,1.0000,1,1,153,1.0,1,252,1.0,1,458,1.0,sqlite.sqlite_list_tables sqlite.sqlite_describe_table postgres.list_tables mysql.mysql_list_tables sqlite.sqlite_query supabase.supabase_list_tables id_map.json postgres.list_schemas clickup.clickup_list_tasks trello.trello_list_cards +q068,Describe a postgres table,verb_product,postgres.describe_table,1.0,1.0,1.0,1.0000,1.0000,1,1,170,1.0,1,272,1.0,1,543,1.0,postgres.describe_table sqlite.sqlite_describe_table postgres.list_schemas postgres.list_tables postgres.query supabase.supabase_insert postgres.explain_query supabase.supabase_update supabase.supabase_select id_map.json +q069,github_create_issue,identifier,github.issue_write,1.0,1.0,1.0,1.0000,0.7075,1,1,303,1.0,1,487,1.0,1,1053,1.0,github.issue_write sentry.sentry_get_issue github.list_check_runs github.create_issue jira.jira_get_issue github.issue_read github.delete_file github.add_issue_comment github.pull_request_review_write github.update_issue_comment +q070,createIssue,identifier,github.issue_write,0.0,0.0,1.0,0.2000,0.2471,5,2,598,0.0,1,481,1.0,1,917,1.0,github.issue_write jira.jira_get_issue sentry.sentry_get_issue gitlab.gitlab_get_issue github.create_issue github.issue_read bitbucket.bitbucket_create_issue sentry.sentry_update_issue gitlab.gitlab_create_issue jira.jira_assign_issue +q071,list-directory,identifier,filesystem.create_directory,0.0,1.0,1.0,0.5000,0.6309,2,1,182,1.0,1,296,1.0,1,644,1.0,filesystem.create_directory filesystem.list_directory filesystem.directory_tree filesystem.list_directory_with_sizes filesystem.get_file_info github.get_file_contents filesystem.list_allowed_directories filesystem.read_file filesystem.read_text_file filesystem.read_media_file +q072,sequential thinking next thought,paraphrase,sequential_thinking.sequentialthinking,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,291,1.0,1,713,1.0,sequential_thinking.sequentialthinking id_map.json +q073,What time is it in Tokyo,paraphrase,time.convert_time,1.0,1.0,1.0,1.0000,0.8597,1,1,190,1.0,1,329,1.0,1,748,1.0,time.convert_time time.get_current_time filesystem.list_allowed_directories playwright.browser_wait_for slack.slack_create_channel jira.jira_add_worklog github.create_hook gitlab.list_pipelines github.update_hook google_calendar.gcal_list_events +q074,Convert time between timezones,verb_product,time.convert_time,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,335,1.0,1,650,1.0,time.convert_time salesforce.salesforce_convert_lead time.get_current_time playwright.browser_wait_for git.git_diff memory.create_relations google_calendar.gcal_list_events jira.jira_add_worklog grafana.grafana_create_annotation id_map.json +q075,Star a GitHub repo,verb_product,github.star_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,195,1.0,1,301,1.0,1,755,1.0,github.star_repository git.git_show git.git_branch git.git_status git.git_diff_unstaged github.unstar_repository git.git_checkout github.delete_file github.pull_request_review_write git.git_diff +q076,Fork a repository,resource,github.fork_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,425,1.0,1,720,1.0,github.fork_repository github.star_repository github.unstar_repository github.create_repository github.get_repository_tree github.delete_repository id_map.json github.list_repository_collaborators github.list_dependabot_alerts github.list_branches +q077,Get the latest GitHub release,verb_product,github.get_latest_release,1.0,1.0,1.0,1.0000,1.0000,1,1,206,1.0,1,344,1.0,1,640,1.0,github.get_latest_release git.git_show git.git_status git.git_branch github.get_release_by_tag git.git_commit git.git_diff_unstaged git.git_add git.git_diff_staged git.git_checkout +q078,Create a gist,resource,github.create_gist,1.0,1.0,1.0,1.0000,1.0000,1,1,229,1.0,1,385,1.0,1,770,1.0,github.create_gist github.get_gist github.update_gist github.list_gists github.create_milestone id_map.json github.create_hook github.create_environment github.create_package github.create_artifact +q079,Search GitHub code,verb_product,github.get_code_scanning_alert,0.0,1.0,1.0,0.3333,0.5000,3,1,246,1.0,1,390,1.0,1,673,1.0,github.get_code_scanning_alert git.git_show github.search_code github.list_code_scanning_alerts git.git_branch git.git_status git.git_checkout git.git_diff_unstaged git.git_diff git.git_diff_staged +q080,List GitHub notifications,verb_product,github.list_notifications,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,282,1.0,1,577,1.0,github.list_notifications git.git_show github.mark_all_notifications_read git.git_branch git.git_status git.git_checkout git.git_diff_unstaged git.git_diff git.git_diff_staged git.git_commit +q081,Assign Copilot to an issue,verb_product,github.assign_copilot_to_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,457,1.0,1,983,1.0,github.assign_copilot_to_issue jira.jira_assign_issue github.request_copilot_review linear.linear_assign_issue slack.slack_reply_to_thread github.add_comment_to_pending_review jira.jira_update_issue asana.asana_create_task jira.jira_transition_issue github.create_issue +q082,Get secret scanning alerts,resource,github.list_secret_scanning_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,295,1.0,1,470,1.0,1,747,1.0,github.list_secret_scanning_alerts github.get_secret_scanning_alert github.list_code_scanning_alerts github.list_dependabot_alerts github.get_code_scanning_alert id_map.json kubernetes.k8s_get_pod redis.redis_get github.get_discussion github.projects_get +q083,Create or update a file on GitHub,verb_product,github.create_or_update_file,1.0,1.0,1.0,1.0000,0.7602,1,1,276,1.0,1,446,1.0,1,864,1.0,github.create_or_update_file git.git_show git.git_branch google_drive.gdrive_get_file github.delete_file git.git_diff google_drive.gdrive_update_file git.git_add github.get_file_contents gitlab.gitlab_get_file +q084,List collaborators on a repo,resource,github.list_repository_collaborators,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,412,1.0,1,837,1.0,github.list_repository_collaborators github.get_discussion_comments pagerduty.pagerduty_list_oncalls notion.notion_get_comments github.list_commits github.star_repository trello.trello_list_cards github.unstar_repository github.update_issue_comment github.get_repository_tree +q085,Reply in a Slack thread,verb_product,slack.slack_reply_to_thread,1.0,1.0,1.0,1.0000,1.0000,1,1,256,1.0,1,391,1.0,1,705,1.0,slack.slack_reply_to_thread slack.slack_get_thread_replies slack.slack_search_messages slack.slack_list_users slack.slack_post_message slack.slack_list_channels slack.slack_add_reaction slack.slack_join_channel slack.slack_leave_channel github.get_notification_details +q086,Add an emoji reaction on Slack,verb_product,slack.slack_add_reaction,1.0,1.0,1.0,1.0000,1.0000,1,1,252,1.0,1,356,1.0,1,741,1.0,slack.slack_add_reaction discord.discord_add_reaction slack.slack_join_channel asana.asana_get_task slack.slack_leave_channel github.get_discussion_comments slack.slack_create_channel asana.asana_update_task slack.slack_search_messages pagerduty.pagerduty_list_oncalls +q087,Search Slack messages,verb_product,slack.slack_search_messages,1.0,1.0,1.0,1.0000,1.0000,1,1,226,1.0,1,386,1.0,1,703,1.0,slack.slack_search_messages slack.slack_get_channel_history slack.slack_list_users slack.slack_post_message slack.slack_add_reaction twilio.twilio_list_messages slack.slack_join_channel slack.slack_leave_channel slack.slack_reply_to_thread slack.slack_create_channel +q088,Create a Slack channel,verb_product,slack.slack_set_channel_topic,0.0,0.0,0.0,0.1000,0.2891,10,4,837,0.0,2,705,0.0,1,685,1.0,slack.slack_set_channel_topic slack.slack_search_messages slack.slack_leave_channel slack.slack_get_channel_history slack.slack_join_channel slack.slack_add_reaction slack.slack_list_users slack.slack_post_message discord.discord_get_channel slack.slack_create_channel +q089,Transition a Jira issue,verb_product,jira.jira_transition_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,213,1.0,1,337,1.0,1,789,1.0,jira.jira_transition_issue jira.jira_get_issue jira.jira_search_issues jira.jira_list_projects jira.jira_add_worklog jira.jira_add_comment jira.jira_assign_issue github.issue_write jira.jira_update_issue jira.jira_create_issue +q090,Search Jira with JQL,identifier,jira.jira_search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,179,1.0,1,337,1.0,1,734,1.0,jira.jira_search_issues jira.jira_list_projects jira.jira_get_issue jira.jira_add_worklog jira.jira_add_comment filesystem.list_directory_with_sizes jira.jira_assign_issue jira.jira_create_sprint jira.jira_update_issue github.update_pull_request_branch +q091,Add a worklog to Jira,verb_product,jira.jira_add_worklog,1.0,1.0,1.0,1.0000,1.0000,1,1,252,1.0,1,338,1.0,1,821,1.0,jira.jira_add_worklog jira.jira_add_comment jira.jira_search_issues jira.jira_list_projects jira.jira_get_issue github.add_comment_to_pending_review jira.jira_transition_issue slack.slack_reply_to_thread jira.jira_assign_issue github.assign_copilot_to_issue +q092,List Linear teams,verb_product,linear.linear_list_teams,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,437,1.0,1,728,1.0,linear.linear_list_teams linear.linear_create_project linear.linear_update_issue linear.linear_create_issue linear.linear_get_issue linear.linear_create_comment github.get_teams linear.linear_list_projects linear.linear_list_issues linear.linear_search_issues +q093,Comment on a Linear issue,verb_product,linear.linear_update_issue,0.0,0.0,1.0,0.2000,0.3869,5,2,552,0.0,1,426,1.0,1,871,1.0,linear.linear_update_issue linear.linear_get_issue linear.create_comment figma.figma_post_comment linear.linear_create_comment github.update_issue_comment linear.linear_create_project linear.linear_create_issue jira.jira_get_issue linear.linear_assign_issue +q094,Move a local file,paraphrase,filesystem.move_file,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,324,1.0,1,744,1.0,filesystem.move_file brave_search.brave_local_search figma.figma_get_file filesystem.read_file google_drive.gdrive_get_file google_drive.gdrive_update_file filesystem.read_text_file gitlab.gitlab_get_file gmail.gmail_modify_email filesystem.read_media_file +q095,Recursively search files by pattern,paraphrase,filesystem.search_files,1.0,1.0,1.0,1.0000,1.0000,1,1,280,1.0,1,419,1.0,1,721,1.0,filesystem.search_files github.push_files slack.list_files github.get_release_by_tag redis.redis_keys filesystem.list_directory filesystem.move_file salesforce.salesforce_get_record jira.jira_get_issue filesystem.list_directory_with_sizes +q096,Edit a file in place with old and new string,paraphrase,filesystem.edit_file,1.0,1.0,1.0,1.0000,1.0000,1,1,228,1.0,1,369,1.0,1,772,1.0,filesystem.edit_file figma.figma_get_file google_drive.gdrive_get_file playwright.browser_tab_new google_drive.gdrive_update_file filesystem.list_directory_with_sizes gitlab.gitlab_get_file filesystem.write_file slack.create_file google_drive.gdrive_share_file +q097,Read the whole knowledge graph,paraphrase,memory.read_graph,1.0,1.0,1.0,1.0000,1.0000,1,1,155,1.0,1,255,1.0,1,523,1.0,memory.read_graph memory.create_relations memory.delete_relations memory.delete_entities memory.create_entities memory.add_observations memory.search_nodes memory.open_nodes playwright.browser_navigate memory.delete_observations +q098,Search memory nodes,verb_product,memory.search_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,202,1.0,1,305,1.0,1,512,1.0,memory.search_nodes figma.figma_get_file_nodes memory.open_nodes memory.delete_entities memory.create_relations memory.add_observations memory.delete_observations memory.create_entities id_map.json memory.delete_relations +q099,Show unstaged git diff,verb_product,git.git_diff_unstaged,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,297,1.0,1,540,1.0,git.git_diff_unstaged git.git_diff_staged git.git_diff git.git_show git.git_branch git.git_status id_map.json git.git_checkout git.git_commit git.git_add +q100,Stage files with git add,identifier,filesystem.list_directory_with_sizes,0.0,0.0,0.0,0.1111,0.3010,9,3,634,0.0,2,675,0.0,1,654,1.0,filesystem.list_directory_with_sizes filesystem.search_files git.git_show git.git_branch github.push_files git.git_status git.git_diff_unstaged git.git_diff_staged git.git_add git.git_checkout +q101,Create a Shopify product,verb_product,shopify.shopify_create_product,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,377,1.0,1,702,1.0,shopify.shopify_create_product shopify.shopify_update_product shopify.shopify_get_product shopify.shopify_list_products shopify.shopify_list_orders stripe.stripe_create_product stripe.stripe_create_price shopify.shopify_create_order shopify.shopify_get_order shopify.shopify_list_customers +q102,Cancel a Stripe subscription,verb_product,stripe.stripe_cancel_subscription,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,328,1.0,1,621,1.0,stripe.stripe_cancel_subscription stripe.stripe_list_customers stripe.stripe_list_subscriptions stripe.stripe_create_invoice stripe.stripe_create_price stripe.stripe_create_product stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_get_payment_intent stripe.stripe_list_invoices +q103,Create a payment intent,resource,stripe.stripe_create_payment_intent,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,312,1.0,1,802,1.0,stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent stripe.stripe_create_refund id_map.json github.create_gist github.create_milestone github.create_hook github.create_environment github.create_package github.create_artifact +q104,Acknowledge a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,405,1.0,1,626,1.0,pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls id_map.json +q105,Create a Datadog monitor,verb_product,datadog.datadog_create_monitor,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,459,1.0,1,832,1.0,datadog.datadog_create_monitor datadog.datadog_get_monitor datadog.datadog_list_monitors datadog.datadog_query_metrics datadog.datadog_list_logs id_map.json github.create_gist github.create_milestone github.create_hook github.create_environment +q106,Resolve a Sentry issue,paraphrase,sentry.sentry_update_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,230,1.0,1,391,1.0,1,823,1.0,sentry.sentry_update_issue sentry.sentry_get_issue sentry.sentry_list_projects pagerduty.pagerduty_resolve_incident sentry.sentry_list_issues sentry.sentry_list_events jira.jira_get_issue github.issue_write github.issue_read gitlab.gitlab_get_issue +q107,Scale a Kubernetes deployment,verb_product,kubernetes.k8s_scale_deployment,1.0,1.0,1.0,1.0000,1.0000,1,1,262,1.0,1,374,1.0,1,675,1.0,kubernetes.k8s_scale_deployment kubernetes.k8s_list_deployments kubernetes.k8s_delete kubernetes.k8s_list_services vercel.vercel_get_deployment kubernetes.k8s_pod_logs kubernetes.k8s_get_pod id_map.json kubernetes.k8s_apply vercel.vercel_list_deployments +q108,Apply a Kubernetes manifest,verb_product,kubernetes.k8s_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,273,1.0,1,390,1.0,1,670,1.0,kubernetes.k8s_apply kubernetes.k8s_scale_deployment kubernetes.k8s_delete kubernetes.k8s_list_deployments terraform.tf_apply kubernetes.k8s_list_services kubernetes.k8s_pod_logs kubernetes.k8s_get_pod kubernetes.k8s_get_events id_map.json +q109,Stop a docker container,verb_product,docker.docker_stop,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,345,1.0,1,587,1.0,docker.docker_stop docker.docker_rm docker.docker_run docker.docker_logs docker.docker_exec docker.docker_build docker.docker_ps kubernetes.k8s_pod_logs docker.docker_images id_map.json +q110,Build a docker image,verb_product,docker.docker_build,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,330,1.0,1,571,1.0,docker.docker_build docker.docker_run docker.docker_rm docker.docker_stop docker.docker_ps docker.docker_images filesystem.read_media_file figma.figma_get_images docker.docker_logs id_map.json +q111,Click an element in Playwright,verb_product,playwright.browser_click,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,381,1.0,1,667,1.0,playwright.browser_click puppeteer.puppeteer_click playwright.browser_type playwright.browser_take_screenshot clickup.clickup_create_task clickup.clickup_list_tasks clickup.clickup_update_task playwright.browser_snapshot playwright.browser_navigate asana.asana_get_task +q112,Type text into a browser input,paraphrase,playwright.browser_type,1.0,1.0,1.0,1.0000,0.7602,1,1,217,1.0,1,316,1.0,1,614,1.0,playwright.browser_type playwright.browser_wait_for playwright.browser_navigate playwright.browser_tab_new playwright.browser_tab_close playwright.browser_tab_list filesystem.read_text_file playwright.browser_snapshot mongodb.mongo_insert playwright.browser_click +q113,List GitLab projects,verb_product,gitlab.gitlab_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,178,1.0,1,296,1.0,1,565,1.0,gitlab.gitlab_list_projects git.git_show git.git_branch git.git_status github.projects_list vercel.vercel_list_projects git.git_checkout sentry.sentry_list_projects git.git_diff git.git_diff_unstaged +q114,Create a Bitbucket pull request,verb_product,bitbucket.bitbucket_list_prs,0.0,1.0,1.0,0.3333,0.5000,3,1,313,1.0,1,509,1.0,1,1020,1.0,bitbucket.bitbucket_list_prs bitbucket.bitbucket_create_issue bitbucket.bitbucket_create_pr github.pull_request_review_write bitbucket.bitbucket_list_repos github.update_pull_request github.pull_request_read github.update_pull_request_branch gitlab.gitlab_get_merge_request github.merge_pull_request +q115,Create a Gmail draft,verb_product,gmail.gmail_create_draft,1.0,1.0,1.0,1.0000,1.0000,1,1,227,1.0,1,377,1.0,1,730,1.0,gmail.gmail_create_draft gmail.gmail_send_email gmail.gmail_list_labels gmail.gmail_modify_email gmail.gmail_trash_email gmail.gmail_get_email github.create_pull_request gmail.gmail_search_emails id_map.json github.create_gist +q116,Share a Google Drive file,verb_product,google_drive.gdrive_get_file,0.0,1.0,1.0,0.5000,0.6309,2,1,218,1.0,1,417,1.0,1,778,1.0,google_drive.gdrive_get_file google_drive.gdrive_share_file google_drive.gdrive_search google_drive.gdrive_update_file google_drive.gdrive_create_file gitlab.gitlab_get_file google_drive.gdrive_list_folder figma.figma_get_file google_calendar.gcal_delete_event google_calendar.gcal_update_event +q117,List Google Calendar events,verb_product,google_calendar.gcal_delete_event,0.0,1.0,1.0,0.5000,0.6309,2,1,269,1.0,1,441,1.0,1,770,1.0,google_calendar.gcal_delete_event google_calendar.gcal_list_events google_calendar.gcal_update_event google_calendar.gcal_create_event kubernetes.k8s_get_events google_calendar.gcal_list_calendars sentry.sentry_list_events google_drive.gdrive_search google_drive.gdrive_create_file google_drive.gdrive_list_folder +q118,Delete a calendar event,resource,google_calendar.gcal_delete_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,427,1.0,1,754,1.0,google_calendar.gcal_delete_event google_calendar.gcal_create_event google_calendar.gcal_update_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars id_map.json kubernetes.k8s_delete kubernetes.k8s_get_events redis.redis_delete github.pull_request_review_write +q119,List Vercel projects,verb_product,vercel.vercel_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,286,1.0,1,550,1.0,vercel.vercel_list_projects vercel.vercel_get_deployment vercel.vercel_list_deployments vercel.vercel_create_project sentry.sentry_list_projects jira.jira_list_projects github.projects_list gitlab.gitlab_list_projects linear.linear_list_projects github.projects_get +q120,Select rows from Supabase,verb_product,supabase.supabase_select,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,354,1.0,1,737,1.0,supabase.supabase_select supabase.supabase_insert supabase.supabase_update supabase.supabase_list_tables supabase.supabase_invoke_fn github.delete_file elasticsearch.es_get_doc mongodb.mongo_delete aws_s3.s3_get_object puppeteer.puppeteer_click +q121,Post a Figma comment,verb_product,figma.figma_post_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,368,1.0,1,781,1.0,figma.figma_post_comment figma.figma_get_comments figma.figma_get_file figma.figma_get_images figma.figma_get_file_nodes linear.create_comment notion.create_comment github.update_issue_comment linear.get_comment linear.update_comment +q122,List Okta users,verb_product,okta.okta_create_user,0.0,1.0,1.0,0.5000,0.6309,2,1,179,1.0,1,298,1.0,1,595,1.0,okta.okta_create_user okta.okta_list_users okta.okta_deactivate_user slack.slack_list_users okta.okta_get_user linear.list_users notion.list_users_tokens github.search_users notion.delete_users_token notion.notion_list_users +q123,Range query Prometheus,verb_product,prometheus.prom_query_range,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,274,1.0,1,595,1.0,prometheus.prom_query_range prometheus.prom_list_metrics prometheus.prom_query id_map.json postgres.query postgres.explain_query salesforce.salesforce_query sqlite.sqlite_query mysql.mysql_query datadog.datadog_query_metrics +q124,Create a Grafana annotation,verb_product,grafana.grafana_create_annotation,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,296,1.0,1,786,1.0,grafana.grafana_create_annotation grafana.grafana_get_dashboard grafana.grafana_search_dashboards id_map.json github.create_gist github.create_milestone github.create_hook github.create_environment github.create_package github.create_artifact +q125,Apply terraform,identifier,terraform.tf_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,238,1.0,1,595,1.0,terraform.tf_apply terraform.tf_plan kubernetes.k8s_apply terraform.tf_state_list id_map.json +q126,List Asana tasks,verb_product,asana.asana_list_tasks,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,386,1.0,1,641,1.0,asana.asana_list_tasks asana.asana_get_task asana.asana_create_task asana.asana_update_task clickup.clickup_list_tasks asana.asana_create_project id_map.json trello.trello_list_cards kubernetes.k8s_list_deployments kubernetes.k8s_list_services +q127,Comment on a Trello card,verb_product,trello.trello_add_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,390,1.0,1,892,1.0,trello.trello_add_comment trello.trello_list_cards trello.trello_create_card figma.figma_post_comment trello.trello_create_board github.update_issue_comment linear.create_comment notion.create_comment linear.get_comment github.add_comment_to_pending_review +q128,Update a ClickUp task,verb_product,clickup.clickup_update_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,362,1.0,1,761,1.0,clickup.clickup_update_task clickup.clickup_create_task clickup.clickup_get_task clickup.clickup_list_tasks asana.asana_get_task puppeteer.puppeteer_click asana.asana_update_task playwright.browser_click asana.asana_create_task mongodb.mongo_update +q129,Search Confluence,verb_product,confluence.confluence_get_page,0.0,1.0,1.0,0.5000,0.6309,2,1,221,1.0,1,401,1.0,1,731,1.0,confluence.confluence_get_page confluence.confluence_search confluence.confluence_create_page confluence.confluence_update_page brave_search.brave_web_search brave_search.brave_local_search github.search_issues github.search_users github.search_orgs elasticsearch.es_search +q130,Create a HubSpot deal,verb_product,hubspot.hubspot_create_deal,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,872,1.0,hubspot.hubspot_create_deal hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_list_contacts hubspot.hubspot_search_contacts github.delete_file github.create_gist github.pull_request_review_write github.create_repository github.get_discussion_comments +q131,List Zendesk tickets,verb_product,zendesk.zendesk_list_tickets,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,367,1.0,1,604,1.0,zendesk.zendesk_list_tickets zendesk.zendesk_search zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_create_ticket id_map.json clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments kubernetes.k8s_list_services +q132,List Discord channels,verb_product,discord.discord_list_channels,1.0,1.0,1.0,1.0000,1.0000,1,1,205,1.0,1,321,1.0,1,573,1.0,discord.discord_list_channels discord.discord_get_channel discord.discord_add_reaction discord.discord_list_members discord.discord_send_message slack.slack_list_channels id_map.json clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments +q133,Set a Redis key,verb_product,redis.redis_get,0.0,1.0,1.0,0.5000,0.6309,2,1,185,1.0,1,280,1.0,1,605,1.0,redis.redis_get redis.redis_set redis.redis_keys redis.redis_delete playwright.browser_press_key jira.jira_get_issue figma.figma_post_comment slack.slack_set_channel_topic figma.figma_get_comments aws_s3.s3_get_object +q134,Insert MongoDB documents,verb_product,mongodb.mongo_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,256,1.0,1,381,1.0,1,607,1.0,mongodb.mongo_insert mongodb.mongo_delete mongodb.mongo_update mongodb.mongo_find mongodb.mongo_list_collections supabase.supabase_insert id_map.json +q135,Run a MySQL query,verb_product,mysql.mysql_query,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,266,1.0,1,543,1.0,mysql.mysql_query postgres.query postgres.explain_query mysql.mysql_list_tables postgres.list_schemas sqlite.sqlite_query postgres.list_tables postgres.describe_table salesforce.salesforce_query prometheus.prom_query +q136,Index a document in Elasticsearch,verb_product,elasticsearch.es_index_doc,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,367,1.0,1,669,1.0,elasticsearch.es_index_doc elasticsearch.es_get_doc elasticsearch.es_search mongodb.mongo_find playwright.browser_tab_close id_map.json github.create_milestone mongodb.mongo_insert github.get_milestone jira.update_sprint +q137,Get an S3 object,verb_product,aws_s3.s3_delete_object,0.0,1.0,1.0,0.5000,0.6309,2,1,249,1.0,1,373,1.0,1,687,1.0,aws_s3.s3_delete_object aws_s3.s3_get_object aws_s3.s3_put_object asana.asana_get_task aws_s3.s3_list_objects asana.asana_create_task id_map.json okta.okta_get_user asana.asana_update_task elasticsearch.es_search +q138,List Cloudflare zones,verb_product,cloudflare.cf_list_zones,1.0,1.0,1.0,1.0000,1.0000,1,1,230,1.0,1,363,1.0,1,600,1.0,cloudflare.cf_list_zones cloudflare.cf_purge_cache cloudflare.cf_create_dns cloudflare.cf_list_dns time.convert_time clickup.clickup_list_tasks id_map.json trello.trello_list_cards kubernetes.k8s_list_deployments kubernetes.k8s_list_services +q139,Create a Stripe invoice,verb_product,stripe.stripe_create_invoice,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,330,1.0,1,575,1.0,stripe.stripe_create_invoice stripe.stripe_create_product stripe.stripe_create_price stripe.stripe_list_invoices stripe.stripe_list_customers stripe.stripe_create_customer stripe.stripe_get_payment_intent stripe.stripe_create_refund stripe.stripe_get_customer id_map.json +q140,List GitHub Actions workflows,verb_product,github.actions_list,1.0,1.0,1.0,1.0000,1.0000,1,1,268,1.0,1,408,1.0,1,726,1.0,github.actions_list github.actions_run_trigger git.git_show git.git_branch github.actions_get_job_logs github.actions_get git.git_status git.git_diff_unstaged git.git_checkout git.git_diff +q141,Get GitHub Actions job logs,verb_product,github.actions_get_job_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,296,1.0,1,449,1.0,1,727,1.0,github.actions_get_job_logs github.actions_run_trigger git.git_show github.actions_list github.actions_get git.git_branch gitlab.delete_job git.git_log git.git_diff_unstaged git.git_checkout +q142,Create a GitHub repository,verb_product,github.star_repository,0.0,1.0,1.0,0.5000,0.6309,2,1,239,1.0,1,427,1.0,1,707,1.0,github.star_repository github.create_repository github.unstar_repository github.fork_repository github.get_repository_tree git.git_status git.git_show github.delete_repository git.git_branch git.git_diff_unstaged +q143,Checkout a git branch,verb_product,git.git_checkout,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,373,1.0,1,702,1.0,git.git_checkout git.git_branch git.git_create_branch git.git_show github.create_branch git.git_status git.git_diff_unstaged git.git_diff github.update_pull_request_branch git.git_commit +q144,Show git log,identifier,git.git_show,0.0,1.0,1.0,0.5000,0.6309,2,1,188,1.0,1,294,1.0,1,634,1.0,git.git_show git.git_log git.git_branch git.git_status git.git_diff_unstaged git.git_diff_staged git.git_diff git.git_checkout datadog.datadog_list_logs git.git_commit +q145,List postgres schemas,verb_product,postgres.list_schemas,1.0,1.0,1.0,1.0000,1.0000,1,1,169,1.0,1,277,1.0,1,514,1.0,postgres.list_schemas postgres.list_tables postgres.describe_table postgres.query postgres.explain_query id_map.json clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments kubernetes.k8s_list_services +q146,Open nodes in the knowledge graph,verb_product,memory.open_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,308,1.0,1,507,1.0,memory.open_nodes memory.search_nodes memory.create_relations memory.delete_entities figma.figma_get_file_nodes memory.read_graph memory.create_entities memory.delete_relations id_map.json jira.update_sprint +q147,Get file metadata from the filesystem,paraphrase,filesystem.read_media_file,0.0,1.0,1.0,0.3333,0.5000,3,1,191,1.0,1,339,1.0,1,692,1.0,filesystem.read_media_file google_drive.gdrive_get_file filesystem.get_file_info gitlab.gitlab_get_file filesystem.read_file memory.delete_entities filesystem.read_text_file google_drive.gdrive_update_file figma.figma_get_file filesystem.write_file +q148,Write a new local file,paraphrase,playwright.browser_tab_new,0.0,0.0,0.0,0.1429,0.3333,7,3,667,0.0,2,737,0.0,1,716,1.0,playwright.browser_tab_new brave_search.brave_local_search filesystem.edit_file filesystem.read_file figma.figma_get_file filesystem.read_media_file filesystem.write_file google_drive.gdrive_get_file google_drive.gdrive_update_file gitlab.gitlab_get_file +q149,Search GitHub issues,verb_product,git.git_show,0.0,0.0,1.0,0.2500,0.4307,4,2,459,0.0,1,367,1.0,1,714,1.0,git.git_show github.list_issues git.git_branch github.search_issues git.git_status jira.jira_search_issues git.git_checkout git.git_diff_unstaged sentry.sentry_list_issues git.git_diff +q150,Get the authenticated GitHub user,paraphrase,git.git_show,0.0,0.0,0.0,0.0000,0.0000,11,4,751,0.0,3,994,0.0,2,1377,0.0,git.git_show git.git_status git.git_branch git.git_commit github.list_notifications git.git_add git.git_diff_unstaged git.git_checkout git.git_diff_staged playwright.browser_navigate diff --git a/experiments/mcp-tool-search/results/hybrid_codedb_trigram_bm25.json b/experiments/mcp-tool-search/results/hybrid_codedb_trigram_bm25.json new file mode 100644 index 00000000..aed21195 --- /dev/null +++ b/experiments/mcp-tool-search/results/hybrid_codedb_trigram_bm25.json @@ -0,0 +1,6984 @@ +{ + "summary": { + "n": 150, + "index_s": 0.4446164010000757, + "rss_mb": null, + "p50_ms": 0.0062194999941311835, + "p95_ms": 0.00990849998174781, + "recall@1": 0.7066666666666667, + "recall@1_lo": 0.6333333333333333, + "recall@1_hi": 0.78, + "recall@3": 0.9, + "recall@3_lo": 0.8533333333333334, + "recall@3_hi": 0.9466666666666667, + "recall@5": 0.9533333333333334, + "recall@5_lo": 0.9133333333333333, + "recall@5_hi": 0.9866666666666667, + "mrr@10": 0.8091084656084656, + "mrr@10_lo": 0.7606640211640212, + "mrr@10_hi": 0.8567671957671958, + "ndcg@10": 0.8445062459089498, + "ndcg@10_lo": 0.8062462473957297, + "ndcg@10_hi": 0.882210848512806 + }, + "meta": { + "name": "hybrid_codedb_trigram_bm25", + "needs_model": false, + "needs_api": false, + "notes": "RRF codedb trigram tokens + weighted BM25" + }, + "rows": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.issue_write", + "git.git_show", + "git.git_branch", + "github.create_issue", + "github.issue_read", + "jira.jira_get_issue", + "github.add_issue_comment", + "git.git_diff_unstaged", + "github.update_issue_comment", + "gitlab.gitlab_get_issue" + ], + "hit1": "github.issue_write", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7074887171046738, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 264, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 495, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 884, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "kind": "verb_product", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_search_issues", + "jira.jira_get_issue", + "jira.jira_list_projects", + "jira.jira_create_issue", + "jira.jira_add_worklog", + "jira.jira_add_comment", + "zendesk.zendesk_create_ticket", + "jira.jira_create_sprint", + "jira.jira_assign_issue", + "zendesk.zendesk_get_ticket" + ], + "hit1": "jira.jira_search_issues", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 471, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 383, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "kind": "verb_product", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "ranking": [ + "slack.slack_post_message", + "slack.slack_add_reaction", + "slack.slack_search_messages", + "slack.slack_list_users", + "slack.slack_join_channel", + "discord.discord_send_message", + "slack.slack_leave_channel", + "slack.slack_create_channel", + "discord.discord_add_reaction", + "slack.slack_reply_to_thread" + ], + "hit1": "slack.slack_post_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 734, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "kind": "resource", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "ranking": [ + "github.list_pull_requests", + "github.search_pull_requests", + "gitlab.gitlab_list_merge_requests", + "github.pull_request_review_write", + "github.update_pull_request", + "github.pull_request_read", + "github.merge_pull_request", + "github.update_pull_request_branch", + "bitbucket.bitbucket_list_prs", + "github.create_pull_request" + ], + "hit1": "github.list_pull_requests", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 519, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1035, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "ranking": [ + "github.create_pull_request", + "github.pull_request_review_write", + "github.update_pull_request", + "github.pull_request_read", + "github.update_pull_request_branch", + "github.merge_pull_request", + "gitlab.gitlab_get_merge_request", + "git.git_show", + "gitlab.gitlab_merge_merge_request", + "git.git_branch" + ], + "hit1": "github.create_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 458, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 643, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1013, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "kind": "verb_product", + "gold": [ + "postgres.query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "postgres.explain_query", + "postgres.list_schemas", + "mysql.mysql_query", + "sqlite.sqlite_query", + "postgres.list_tables", + "postgres.describe_table", + "salesforce.salesforce_query", + "mysql.mysql_list_tables", + "prometheus.prom_query" + ], + "hit1": "postgres.query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 167, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 272, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 544, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "kind": "paraphrase", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "ranking": [ + "filesystem.read_media_file", + "brave_search.brave_local_search", + "filesystem.read_file", + "gitlab.gitlab_get_file", + "github.delete_file", + "google_drive.gdrive_get_file", + "filesystem.read_text_file", + "google_drive.gdrive_update_file", + "filesystem.write_file", + "filesystem.edit_file" + ], + "hit1": "filesystem.read_media_file", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5067916889545789, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 813, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "kind": "paraphrase", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "ranking": [ + "filesystem.search_files", + "github.push_files", + "slack.list_files", + "google_drive.gdrive_list_folder", + "jira.update_sprint", + "linear.list_users", + "filesystem.move_file", + "gitlab.get_pipeline", + "filesystem.list_directory", + "elasticsearch.es_index_doc" + ], + "hit1": "filesystem.search_files", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.1111111111111111, + "ndcg@10": 0.1922943146971571, + "hit_rank": 9, + "k3_calls": 3, + "k3_tokens": 690, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 753, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 731, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "ranking": [ + "github.create_artifact", + "memory.read_graph", + "github.delete_artifact", + "memory.create_entities", + "github.get_artifact", + "memory.delete_relations", + "github.list_artifacts", + "memory.create_relations", + "github.update_artifact", + "memory.add_observations" + ], + "hit1": "github.create_artifact", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.3834409525148764, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 383, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 311, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 612, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "kind": "verb_product", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "ranking": [ + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "filesystem.list_directory_with_sizes", + "id_map.json", + "jira.jira_search_issues", + "jira.delete_webhook", + "github.update_pull_request_branch", + "jira.get_webhook", + "memory.search_nodes", + "jira.list_webhooks" + ], + "hit1": "brave_search.brave_web_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 653, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "kind": "verb_product", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "ranking": [ + "fetch.fetch", + "playwright.browser_navigate", + "github.get_file_contents", + "slack.slack_set_channel_topic", + "puppeteer.puppeteer_navigate", + "playwright.browser_snapshot", + "filesystem.read_file", + "git.git_add", + "filesystem.list_directory", + "playwright.browser_tab_new" + ], + "hit1": "fetch.fetch", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 653, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "kind": "identifier", + "gold": [ + "git.git_status" + ], + "alts": [], + "ranking": [ + "git.git_status", + "git.git_show", + "git.git_branch", + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_diff", + "sentry.sentry_update_issue", + "git.git_checkout", + "git.git_commit", + "github.list_check_runs" + ], + "hit1": "git.git_status", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 283, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 639, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "kind": "paraphrase", + "gold": [ + "git.git_commit" + ], + "alts": [], + "ranking": [ + "git.git_commit", + "git.git_diff_staged", + "github.get_commit", + "git.git_reset", + "git.git_log", + "git.git_show", + "github.push_files", + "id_map.json", + "git.git_diff_unstaged", + "github.search_commits" + ], + "hit1": "git.git_commit", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 611, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_project", + "linear.linear_update_issue", + "linear.linear_get_issue", + "linear.linear_create_issue", + "linear.linear_create_comment", + "jira.jira_get_issue", + "linear.linear_assign_issue", + "github.issue_write", + "gitlab.gitlab_get_issue", + "linear.linear_list_teams" + ], + "hit1": "linear.linear_create_project", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 543, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 471, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 840, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "kind": "paraphrase", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_project", + "asana.asana_get_task", + "linear.linear_create_issue", + "mysql.mysql_list_tables", + "linear.linear_update_issue", + "mysql.mysql_query", + "linear.linear_list_teams", + "clickup.clickup_update_task", + "linear.linear_list_projects", + "vercel.vercel_create_project" + ], + "hit1": "linear.linear_create_project", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 290, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 451, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "ranking": [ + "github.update_issue_comment", + "git.git_show", + "git.git_branch", + "figma.figma_post_comment", + "github.add_issue_comment", + "github.issue_read", + "github.get_discussion_comments", + "github.issue_write", + "notion.create_comment", + "github.create_issue" + ], + "hit1": "github.update_issue_comment", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.2, + "ndcg@10": 0.38685280723454163, + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 537, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 414, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 971, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "kind": "ambiguous", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "ranking": [ + "gitlab.gitlab_get_merge_request", + "github.merge_pull_request", + "gitlab.gitlab_merge_merge_request", + "github.pull_request_review_write", + "github.pull_request_read", + "gitlab.gitlab_create_merge_request", + "github.update_pull_request", + "github.update_pull_request_branch", + "github.create_pull_request", + "gitlab.gitlab_list_merge_requests" + ], + "hit1": "gitlab.gitlab_get_merge_request", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.66967181649423, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 516, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1093, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "kind": "verb_product", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "ranking": [ + "github.actions_run_trigger", + "git.git_show", + "github.actions_get_job_logs", + "github.actions_list", + "github.actions_get", + "git.git_branch", + "git.git_status", + "circleci.circleci_trigger_pipeline", + "git.git_checkout", + "git.git_diff_unstaged" + ], + "hit1": "github.actions_run_trigger", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 296, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 449, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "kind": "resource", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "ranking": [ + "github.list_dependabot_alerts", + "github.get_dependabot_alert", + "github.list_code_scanning_alerts", + "github.list_secret_scanning_alerts", + "id_map.json", + "kubernetes.k8s_get_pod", + "redis.redis_get", + "github.get_discussion", + "github.projects_get", + "github.get_label" + ], + "hit1": "github.list_dependabot_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 287, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "kind": "verb_product", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_issues", + "sentry.sentry_list_projects", + "github.list_issues", + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "sentry.sentry_list_events", + "github.search_issues", + "linear.linear_list_issues", + "jira.jira_search_issues", + "gitlab.gitlab_list_issues" + ], + "hit1": "sentry.sentry_list_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 426, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 813, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_list_customers", + "stripe.stripe_create_price", + "stripe.stripe_create_product", + "stripe.stripe_list_invoices", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent", + "stripe.stripe_list_subscriptions" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "kind": "paraphrase", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_refund", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_payment_intent", + "stripe.stripe_list_customers", + "stripe.stripe_create_invoice", + "stripe.stripe_create_price", + "stripe.stripe_create_product", + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "id_map.json" + ], + "hit1": "stripe.stripe_create_refund", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 347, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 613, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "kind": "verb_product", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "ranking": [ + "gmail.gmail_send_email", + "gmail.gmail_modify_email", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "filesystem.list_directory_with_sizes", + "gmail.gmail_list_labels", + "puppeteer.puppeteer_click", + "gmail.gmail_search_emails", + "gmail.gmail_create_draft", + "asana.asana_get_task" + ], + "hit1": "gmail.gmail_send_email", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 668, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "kind": "verb_product", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "ranking": [ + "gmail.gmail_send_email", + "gmail.gmail_list_labels", + "gmail.gmail_modify_email", + "gmail.gmail_trash_email", + "gmail.gmail_get_email", + "mysql.mysql_list_tables", + "gmail.gmail_search_emails", + "gmail.gmail_create_draft", + "mysql.mysql_query", + "brave_search.brave_web_search" + ], + "hit1": "gmail.gmail_send_email", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.14285714285714285, + "ndcg@10": 0.3333333333333333, + "hit_rank": 7, + "k3_calls": 3, + "k3_tokens": 642, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 711, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 691, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "kind": "paraphrase", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_get_file", + "google_drive.gdrive_update_file", + "slack.slack_upload_file", + "google_drive.gdrive_search", + "google_drive.gdrive_create_file", + "slack.slack_reply_to_thread", + "gitlab.gitlab_get_file", + "google_drive.gdrive_share_file", + "github.add_comment_to_pending_review", + "google_calendar.gcal_delete_event" + ], + "hit1": "google_drive.gdrive_get_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.2, + "ndcg@10": 0.38685280723454163, + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 526, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 415, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 907, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "id_map.json", + "github.create_gist", + "kubernetes.k8s_get_events", + "github.create_milestone", + "github.pull_request_review_write" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 805, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_list_pods", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_list_services", + "kubernetes.k8s_delete", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_get_events", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_apply", + "id_map.json" + ], + "hit1": "kubernetes.k8s_list_pods", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_get_pod", + "github.actions_get_job_logs", + "docker.docker_logs", + "datadog.datadog_list_logs", + "kubernetes.k8s_list_pods", + "git.git_log", + "id_map.json", + "aws_s3.s3_get_object", + "github.delete_file" + ], + "hit1": "kubernetes.k8s_pod_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "ranking": [ + "docker.docker_rm", + "docker.docker_run", + "docker.docker_stop", + "docker.docker_exec", + "docker.docker_build", + "docker.docker_logs", + "docker.docker_ps", + "docker.docker_images", + "kubernetes.k8s_pod_logs", + "github.create_check_run" + ], + "hit1": "docker.docker_rm", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 688, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "kind": "verb_product", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "ranking": [ + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_stop", + "docker.docker_run", + "docker.docker_exec", + "kubernetes.k8s_pod_logs", + "docker.docker_build", + "docker.docker_ps", + "datadog.datadog_list_logs", + "github.actions_get_job_logs" + ], + "hit1": "docker.docker_rm", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 747, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "kind": "ambiguous", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "ranking": [ + "playwright.browser_navigate", + "puppeteer.puppeteer_navigate", + "playwright.browser_tab_new", + "playwright.browser_tab_list", + "playwright.browser_tab_close", + "playwright.browser_wait_for", + "github.add_comment_to_pending_review", + "slack.slack_reply_to_thread", + "playwright.browser_click", + "github.assign_copilot_to_issue" + ], + "hit1": "playwright.browser_navigate", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 265, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "kind": "ambiguous", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "ranking": [ + "playwright.browser_take_screenshot", + "puppeteer.puppeteer_screenshot", + "playwright.browser_snapshot", + "slack.slack_set_channel_topic", + "notion.notion_get_page", + "notion.notion_update_page", + "id_map.json", + "confluence.confluence_get_page", + "filesystem.list_directory", + "confluence.confluence_create_page" + ], + "hit1": "playwright.browser_take_screenshot", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 609, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_get_merge_request", + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_create_merge_request", + "gitlab.gitlab_list_merge_requests", + "github.merge_pull_request", + "git.git_show", + "git.git_branch", + "git.git_checkout", + "git.git_diff_unstaged", + "github.pull_request_review_write" + ], + "hit1": "gitlab.gitlab_get_merge_request", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 841, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "kind": "verb_product", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "ranking": [ + "notion.notion_search", + "notion.notion_create_database", + "notion.notion_query_database", + "notion.notion_append_block", + "notion.notion_get_comments", + "notion.notion_get_page", + "notion.notion_get_block", + "notion.notion_update_page", + "notion.notion_list_users", + "notion.notion_create_page" + ], + "hit1": "notion.notion_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "kind": "verb_product", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "ranking": [ + "notion.notion_update_page", + "notion.notion_get_page", + "notion.notion_search", + "notion.notion_get_comments", + "notion.notion_append_block", + "notion.notion_create_database", + "notion.notion_query_database", + "notion.notion_create_page", + "confluence.confluence_get_page", + "notion.notion_get_block" + ], + "hit1": "notion.notion_update_page", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.125, + "ndcg@10": 0.31546487678572877, + "hit_rank": 8, + "k3_calls": 3, + "k3_tokens": 695, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 727, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "kind": "verb_product", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "ranking": [ + "notion.notion_query_database", + "notion.notion_create_database", + "notion.notion_search", + "notion.notion_append_block", + "notion.notion_get_comments", + "notion.notion_create_page", + "notion.notion_get_page", + "mongodb.mongo_delete", + "notion.notion_get_block", + "mongodb.mongo_find" + ], + "hit1": "notion.notion_query_database", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 387, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 730, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_search", + "zendesk.zendesk_list_tickets", + "jira.jira_create_issue", + "id_map.json", + "github.create_gist", + "github.create_milestone", + "github.create_hook" + ], + "hit1": "zendesk.zendesk_get_ticket", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 761, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_create_deal", + "hubspot.hubspot_search_contacts", + "id_map.json", + "github.create_gist", + "github.delete_file", + "github.create_repository", + "github.pull_request_review_write" + ], + "hit1": "hubspot.hubspot_create_contact", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 779, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "kind": "verb_product", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_create_lead", + "salesforce.salesforce_convert_lead", + "salesforce.salesforce_update_record", + "salesforce.salesforce_get_record", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_list_opportunities", + "salesforce.salesforce_create_record", + "salesforce.salesforce_query", + "id_map.json", + "github.create_gist" + ], + "hit1": "salesforce.salesforce_create_lead", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 273, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 423, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "kind": "identifier", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_query", + "postgres.query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "prometheus.prom_query", + "docker.docker_run", + "postgres.explain_query", + "prometheus.prom_query_range", + "datadog.datadog_query_metrics", + "notion.notion_query_database" + ], + "hit1": "salesforce.salesforce_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "kind": "verb_product", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "ranking": [ + "shopify.shopify_list_orders", + "shopify.shopify_list_products", + "shopify.shopify_update_product", + "shopify.shopify_create_product", + "shopify.shopify_get_product", + "shopify.shopify_list_customers", + "id_map.json", + "shopify.shopify_get_order", + "shopify.shopify_create_order", + "clickup.clickup_list_tasks" + ], + "hit1": "shopify.shopify_list_orders", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 376, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 838, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "kind": "paraphrase", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_list_oncalls", + "filesystem.list_allowed_directories", + "twilio.twilio_make_call", + "notion.notion_get_comments", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_get_incident", + "github.get_discussion_comments", + "pagerduty.pagerduty_create_incident", + "slack.slack_create_channel", + "pagerduty.pagerduty_acknowledge_incident" + ], + "hit1": "pagerduty.pagerduty_list_oncalls", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 346, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 749, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "kind": "verb_product", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "ranking": [ + "datadog.datadog_query_metrics", + "datadog.datadog_create_monitor", + "datadog.datadog_list_monitors", + "datadog.datadog_list_logs", + "prometheus.prom_list_metrics", + "datadog.datadog_get_monitor", + "id_map.json", + "prometheus.prom_query", + "postgres.query", + "postgres.explain_query" + ], + "hit1": "datadog.datadog_query_metrics", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 449, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 668, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "kind": "verb_product", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "ranking": [ + "datadog.datadog_list_logs", + "datadog.datadog_query_metrics", + "datadog.datadog_list_monitors", + "datadog.datadog_create_monitor", + "github.actions_get_job_logs", + "datadog.datadog_get_monitor", + "kubernetes.k8s_pod_logs", + "docker.docker_logs", + "git.git_log", + "brave_search.brave_web_search" + ], + "hit1": "datadog.datadog_list_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 284, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 493, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 863, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_list_buckets", + "id_map.json", + "aws_s3.s3_list_objects", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "github.projects_list", + "github.list_milestones", + "github.list_hooks" + ], + "hit1": "aws_s3.s3_list_buckets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 133, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 590, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "kind": "paraphrase", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_put_object", + "slack.slack_upload_file", + "aws_s3.s3_delete_object", + "aws_s3.s3_get_object", + "github.add_comment_to_pending_review", + "slack.slack_reply_to_thread", + "github.assign_copilot_to_issue", + "slack.slack_add_reaction", + "id_map.json", + "twilio.twilio_send_sms" + ], + "hit1": "aws_s3.s3_put_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 487, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 857, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "id_map.json" + ], + "hit1": "cloudflare.cf_purge_cache", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 286, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 643, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "cloudflare.cf_list_zones", + "cloudflare.cf_purge_cache", + "salesforce.salesforce_get_record", + "salesforce.salesforce_update_record", + "salesforce.salesforce_create_record", + "salesforce.salesforce_delete_record", + "git.git_commit", + "id_map.json" + ], + "hit1": "cloudflare.cf_create_dns", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_deployments", + "vercel.vercel_list_projects", + "vercel.vercel_get_deployment", + "kubernetes.k8s_list_deployments", + "vercel.vercel_create_project", + "id_map.json", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_services", + "aws_s3.s3_list_buckets" + ], + "hit1": "vercel.vercel_list_deployments", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 513, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "ranking": [ + "supabase.supabase_insert", + "supabase.supabase_select", + "supabase.supabase_update", + "mongodb.mongo_insert", + "supabase.supabase_invoke_fn", + "supabase.supabase_list_tables", + "playwright.browser_type", + "id_map.json", + "playwright.browser_click", + "playwright.browser_evaluate" + ], + "hit1": "supabase.supabase_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 620, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "kind": "paraphrase", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "ranking": [ + "figma.figma_get_images", + "figma.figma_get_file", + "docker.docker_images", + "figma.figma_post_comment", + "figma.figma_get_comments", + "figma.figma_get_file_nodes", + "github.delete_file", + "elasticsearch.es_get_doc", + "mongodb.mongo_delete", + "aws_s3.s3_get_object" + ], + "hit1": "figma.figma_get_images", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 325, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 730, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "kind": "verb_product", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "ranking": [ + "twilio.twilio_send_sms", + "twilio.twilio_list_messages", + "twilio.twilio_make_call", + "filesystem.list_directory_with_sizes", + "puppeteer.puppeteer_click", + "id_map.json", + "gmail.gmail_send_email", + "asana.asana_get_task", + "puppeteer.puppeteer_fill", + "asana.asana_update_task" + ], + "hit1": "twilio.twilio_send_sms", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "kind": "verb_product", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "ranking": [ + "okta.okta_deactivate_user", + "okta.okta_get_user", + "okta.okta_create_user", + "okta.okta_list_users", + "linear.get_user", + "slack.slack_get_user_profile", + "asana.asana_create_task", + "linear.create_user", + "asana.asana_get_task", + "linear.update_user" + ], + "hit1": "okta.okta_deactivate_user", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 285, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 657, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query", + "prometheus.prom_list_metrics", + "prometheus.prom_query_range", + "id_map.json", + "postgres.query", + "postgres.explain_query", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "datadog.datadog_query_metrics" + ], + "hit1": "prometheus.prom_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 274, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 595, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "kind": "verb_product", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "ranking": [ + "grafana.grafana_search_dashboards", + "grafana.grafana_create_annotation", + "grafana.grafana_get_dashboard", + "id_map.json", + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "confluence.confluence_search", + "github.search_issues", + "github.search_users", + "github.search_orgs" + ], + "hit1": "grafana.grafana_search_dashboards", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 306, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 624, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "kind": "verb_product", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "ranking": [ + "circleci.circleci_trigger_pipeline", + "circleci.circleci_get_pipeline", + "circleci.circleci_list_pipelines", + "gitlab.gitlab_create_pipeline", + "github.actions_run_trigger", + "gitlab.create_pipeline", + "gitlab.get_pipeline", + "id_map.json", + "gitlab.update_pipeline", + "gitlab.delete_pipeline" + ], + "hit1": "circleci.circleci_trigger_pipeline", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 664, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "kind": "identifier", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "ranking": [ + "terraform.tf_plan", + "terraform.tf_apply", + "terraform.tf_state_list", + "postgres.explain_query", + "id_map.json", + "docker.docker_run", + "github.create_check_run", + "github.get_check_run", + "github.update_check_run", + "github.delete_check_run" + ], + "hit1": "terraform.tf_plan", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 227, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 598, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "kind": "verb_product", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "ranking": [ + "asana.asana_get_task", + "asana.asana_update_task", + "asana.asana_create_task", + "asana.asana_create_project", + "asana.asana_list_tasks", + "clickup.clickup_get_task", + "clickup.clickup_update_task", + "clickup.clickup_create_task", + "slack.slack_add_reaction", + "okta.okta_create_user" + ], + "hit1": "asana.asana_get_task", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 783, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "ranking": [ + "trello.trello_create_card", + "trello.trello_list_cards", + "trello.trello_add_comment", + "trello.trello_create_board", + "id_map.json", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment", + "github.create_package" + ], + "hit1": "trello.trello_create_card", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 784, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_update_task", + "clickup.clickup_get_task", + "clickup.clickup_create_task", + "clickup.clickup_list_tasks", + "asana.asana_get_task", + "puppeteer.puppeteer_click", + "playwright.browser_click", + "asana.asana_update_task", + "asana.asana_create_task", + "github.create_gist" + ], + "hit1": "clickup.clickup_update_task", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "kind": "verb_product", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "ranking": [ + "confluence.confluence_get_page", + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "confluence.confluence_search", + "notion.notion_get_page", + "notion.notion_update_page", + "notion.notion_create_page", + "playwright.browser_take_screenshot", + "notion.notion_get_comments", + "playwright.browser_press_key" + ], + "hit1": "confluence.confluence_get_page", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "kind": "verb_product", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "ranking": [ + "discord.discord_add_reaction", + "discord.discord_send_message", + "discord.discord_get_channel", + "slack.slack_post_message", + "discord.discord_list_channels", + "discord.discord_list_members", + "slack.slack_add_reaction", + "gmail.gmail_send_email", + "github.delete_file", + "twilio.twilio_send_sms" + ], + "hit1": "discord.discord_add_reaction", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 776, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_get_doc", + "elasticsearch.es_search", + "elasticsearch.es_index_doc", + "id_map.json", + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "confluence.confluence_search", + "github.search_issues", + "github.search_users", + "github.search_orgs" + ], + "hit1": "elasticsearch.es_get_doc", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 646, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "redis.redis_set", + "redis.redis_keys", + "redis.redis_delete", + "playwright.browser_press_key", + "jira.jira_get_issue", + "figma.figma_get_comments", + "figma.figma_post_comment", + "aws_s3.s3_get_object", + "figma.figma_get_file" + ], + "hit1": "redis.redis_get", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 586, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert", + "mongodb.mongo_find", + "mongodb.mongo_delete", + "mongodb.mongo_update", + "mongodb.mongo_list_collections" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "kind": "verb_product", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "ranking": [ + "sqlite.sqlite_list_tables", + "sqlite.sqlite_describe_table", + "postgres.list_tables", + "mysql.mysql_list_tables", + "sqlite.sqlite_query", + "supabase.supabase_list_tables", + "id_map.json", + "postgres.list_schemas", + "clickup.clickup_list_tasks", + "trello.trello_list_cards" + ], + "hit1": "sqlite.sqlite_list_tables", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 153, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 252, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 458, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "kind": "verb_product", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "ranking": [ + "postgres.describe_table", + "sqlite.sqlite_describe_table", + "postgres.list_schemas", + "postgres.list_tables", + "postgres.query", + "supabase.supabase_insert", + "postgres.explain_query", + "supabase.supabase_update", + "supabase.supabase_select", + "id_map.json" + ], + "hit1": "postgres.describe_table", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 170, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 272, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 543, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.issue_write", + "sentry.sentry_get_issue", + "github.list_check_runs", + "github.create_issue", + "jira.jira_get_issue", + "github.issue_read", + "github.delete_file", + "github.add_issue_comment", + "github.pull_request_review_write", + "github.update_issue_comment" + ], + "hit1": "github.issue_write", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7074887171046738, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 303, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 487, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1053, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "ranking": [ + "github.issue_write", + "jira.jira_get_issue", + "sentry.sentry_get_issue", + "gitlab.gitlab_get_issue", + "github.create_issue", + "github.issue_read", + "bitbucket.bitbucket_create_issue", + "sentry.sentry_update_issue", + "gitlab.gitlab_create_issue", + "jira.jira_assign_issue" + ], + "hit1": "github.issue_write", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.2, + "ndcg@10": 0.24711688711205218, + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 598, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 481, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 917, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "kind": "identifier", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "ranking": [ + "filesystem.create_directory", + "filesystem.list_directory", + "filesystem.directory_tree", + "filesystem.list_directory_with_sizes", + "filesystem.get_file_info", + "github.get_file_contents", + "filesystem.list_allowed_directories", + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file" + ], + "hit1": "filesystem.create_directory", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 644, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "kind": "paraphrase", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "ranking": [ + "sequential_thinking.sequentialthinking", + "id_map.json" + ], + "hit1": "sequential_thinking.sequentialthinking", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "kind": "paraphrase", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "ranking": [ + "time.convert_time", + "time.get_current_time", + "filesystem.list_allowed_directories", + "playwright.browser_wait_for", + "slack.slack_create_channel", + "jira.jira_add_worklog", + "github.create_hook", + "gitlab.list_pipelines", + "github.update_hook", + "google_calendar.gcal_list_events" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8597186998521972, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 748, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "kind": "verb_product", + "gold": [ + "time.convert_time" + ], + "alts": [], + "ranking": [ + "time.convert_time", + "salesforce.salesforce_convert_lead", + "time.get_current_time", + "playwright.browser_wait_for", + "git.git_diff", + "memory.create_relations", + "google_calendar.gcal_list_events", + "jira.jira_add_worklog", + "grafana.grafana_create_annotation", + "id_map.json" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 650, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "kind": "verb_product", + "gold": [ + "github.star_repository" + ], + "alts": [], + "ranking": [ + "github.star_repository", + "git.git_show", + "git.git_branch", + "git.git_status", + "git.git_diff_unstaged", + "github.unstar_repository", + "git.git_checkout", + "github.delete_file", + "github.pull_request_review_write", + "git.git_diff" + ], + "hit1": "github.star_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "kind": "resource", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "ranking": [ + "github.fork_repository", + "github.star_repository", + "github.unstar_repository", + "github.create_repository", + "github.get_repository_tree", + "github.delete_repository", + "id_map.json", + "github.list_repository_collaborators", + "github.list_dependabot_alerts", + "github.list_branches" + ], + "hit1": "github.fork_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 425, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 720, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "kind": "verb_product", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "ranking": [ + "github.get_latest_release", + "git.git_show", + "git.git_status", + "git.git_branch", + "github.get_release_by_tag", + "git.git_commit", + "git.git_diff_unstaged", + "git.git_add", + "git.git_diff_staged", + "git.git_checkout" + ], + "hit1": "github.get_latest_release", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 640, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "kind": "resource", + "gold": [ + "github.create_gist" + ], + "alts": [], + "ranking": [ + "github.create_gist", + "github.get_gist", + "github.update_gist", + "github.list_gists", + "github.create_milestone", + "id_map.json", + "github.create_hook", + "github.create_environment", + "github.create_package", + "github.create_artifact" + ], + "hit1": "github.create_gist", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 770, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "kind": "verb_product", + "gold": [ + "github.search_code" + ], + "alts": [], + "ranking": [ + "github.get_code_scanning_alert", + "git.git_show", + "github.search_code", + "github.list_code_scanning_alerts", + "git.git_branch", + "git.git_status", + "git.git_checkout", + "git.git_diff_unstaged", + "git.git_diff", + "git.git_diff_staged" + ], + "hit1": "github.get_code_scanning_alert", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 246, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "kind": "verb_product", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "ranking": [ + "github.list_notifications", + "git.git_show", + "github.mark_all_notifications_read", + "git.git_branch", + "git.git_status", + "git.git_checkout", + "git.git_diff_unstaged", + "git.git_diff", + "git.git_diff_staged", + "git.git_commit" + ], + "hit1": "github.list_notifications", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 282, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 577, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "kind": "verb_product", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "ranking": [ + "github.assign_copilot_to_issue", + "jira.jira_assign_issue", + "github.request_copilot_review", + "linear.linear_assign_issue", + "slack.slack_reply_to_thread", + "github.add_comment_to_pending_review", + "jira.jira_update_issue", + "asana.asana_create_task", + "jira.jira_transition_issue", + "github.create_issue" + ], + "hit1": "github.assign_copilot_to_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 457, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 983, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "kind": "resource", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "ranking": [ + "github.list_secret_scanning_alerts", + "github.get_secret_scanning_alert", + "github.list_code_scanning_alerts", + "github.list_dependabot_alerts", + "github.get_code_scanning_alert", + "id_map.json", + "kubernetes.k8s_get_pod", + "redis.redis_get", + "github.get_discussion", + "github.projects_get" + ], + "hit1": "github.list_secret_scanning_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 747, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "ranking": [ + "github.create_or_update_file", + "git.git_show", + "git.git_branch", + "google_drive.gdrive_get_file", + "github.delete_file", + "git.git_diff", + "google_drive.gdrive_update_file", + "git.git_add", + "github.get_file_contents", + "gitlab.gitlab_get_file" + ], + "hit1": "github.create_or_update_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 276, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 446, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 864, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "kind": "resource", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "ranking": [ + "github.list_repository_collaborators", + "github.get_discussion_comments", + "pagerduty.pagerduty_list_oncalls", + "notion.notion_get_comments", + "github.list_commits", + "github.star_repository", + "trello.trello_list_cards", + "github.unstar_repository", + "github.update_issue_comment", + "github.get_repository_tree" + ], + "hit1": "github.list_repository_collaborators", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 837, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "kind": "verb_product", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "ranking": [ + "slack.slack_reply_to_thread", + "slack.slack_get_thread_replies", + "slack.slack_search_messages", + "slack.slack_list_users", + "slack.slack_post_message", + "slack.slack_list_channels", + "slack.slack_add_reaction", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "github.get_notification_details" + ], + "hit1": "slack.slack_reply_to_thread", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "kind": "verb_product", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "ranking": [ + "slack.slack_add_reaction", + "discord.discord_add_reaction", + "slack.slack_join_channel", + "asana.asana_get_task", + "slack.slack_leave_channel", + "github.get_discussion_comments", + "slack.slack_create_channel", + "asana.asana_update_task", + "slack.slack_search_messages", + "pagerduty.pagerduty_list_oncalls" + ], + "hit1": "slack.slack_add_reaction", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 741, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "kind": "verb_product", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "ranking": [ + "slack.slack_search_messages", + "slack.slack_get_channel_history", + "slack.slack_list_users", + "slack.slack_post_message", + "slack.slack_add_reaction", + "twilio.twilio_list_messages", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_reply_to_thread", + "slack.slack_create_channel" + ], + "hit1": "slack.slack_search_messages", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "kind": "verb_product", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "ranking": [ + "slack.slack_set_channel_topic", + "slack.slack_search_messages", + "slack.slack_leave_channel", + "slack.slack_get_channel_history", + "slack.slack_join_channel", + "slack.slack_add_reaction", + "slack.slack_list_users", + "slack.slack_post_message", + "discord.discord_get_channel", + "slack.slack_create_channel" + ], + "hit1": "slack.slack_set_channel_topic", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.1, + "ndcg@10": 0.2890648263178879, + "hit_rank": 10, + "k3_calls": 4, + "k3_tokens": 837, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 2, + "k5_tokens": 705, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 685, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "kind": "verb_product", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_transition_issue", + "jira.jira_get_issue", + "jira.jira_search_issues", + "jira.jira_list_projects", + "jira.jira_add_worklog", + "jira.jira_add_comment", + "jira.jira_assign_issue", + "github.issue_write", + "jira.jira_update_issue", + "jira.jira_create_issue" + ], + "hit1": "jira.jira_transition_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 789, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "kind": "identifier", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "ranking": [ + "jira.jira_search_issues", + "jira.jira_list_projects", + "jira.jira_get_issue", + "jira.jira_add_worklog", + "jira.jira_add_comment", + "filesystem.list_directory_with_sizes", + "jira.jira_assign_issue", + "jira.jira_create_sprint", + "jira.jira_update_issue", + "github.update_pull_request_branch" + ], + "hit1": "jira.jira_search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 734, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "kind": "verb_product", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "ranking": [ + "jira.jira_add_worklog", + "jira.jira_add_comment", + "jira.jira_search_issues", + "jira.jira_list_projects", + "jira.jira_get_issue", + "github.add_comment_to_pending_review", + "jira.jira_transition_issue", + "slack.slack_reply_to_thread", + "jira.jira_assign_issue", + "github.assign_copilot_to_issue" + ], + "hit1": "jira.jira_add_worklog", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 821, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "kind": "verb_product", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "ranking": [ + "linear.linear_list_teams", + "linear.linear_create_project", + "linear.linear_update_issue", + "linear.linear_create_issue", + "linear.linear_get_issue", + "linear.linear_create_comment", + "github.get_teams", + "linear.linear_list_projects", + "linear.linear_list_issues", + "linear.linear_search_issues" + ], + "hit1": "linear.linear_list_teams", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "ranking": [ + "linear.linear_update_issue", + "linear.linear_get_issue", + "linear.create_comment", + "figma.figma_post_comment", + "linear.linear_create_comment", + "github.update_issue_comment", + "linear.linear_create_project", + "linear.linear_create_issue", + "jira.jira_get_issue", + "linear.linear_assign_issue" + ], + "hit1": "linear.linear_update_issue", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.2, + "ndcg@10": 0.38685280723454163, + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 552, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 426, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 871, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "kind": "paraphrase", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "ranking": [ + "filesystem.move_file", + "brave_search.brave_local_search", + "figma.figma_get_file", + "filesystem.read_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_update_file", + "filesystem.read_text_file", + "gitlab.gitlab_get_file", + "gmail.gmail_modify_email", + "filesystem.read_media_file" + ], + "hit1": "filesystem.move_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 744, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "kind": "paraphrase", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "ranking": [ + "filesystem.search_files", + "github.push_files", + "slack.list_files", + "github.get_release_by_tag", + "redis.redis_keys", + "filesystem.list_directory", + "filesystem.move_file", + "salesforce.salesforce_get_record", + "jira.jira_get_issue", + "filesystem.list_directory_with_sizes" + ], + "hit1": "filesystem.search_files", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 280, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 419, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "kind": "paraphrase", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "ranking": [ + "filesystem.edit_file", + "figma.figma_get_file", + "google_drive.gdrive_get_file", + "playwright.browser_tab_new", + "google_drive.gdrive_update_file", + "filesystem.list_directory_with_sizes", + "gitlab.gitlab_get_file", + "filesystem.write_file", + "slack.create_file", + "google_drive.gdrive_share_file" + ], + "hit1": "filesystem.edit_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 228, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 369, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 772, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "ranking": [ + "memory.read_graph", + "memory.create_relations", + "memory.delete_relations", + "memory.delete_entities", + "memory.create_entities", + "memory.add_observations", + "memory.search_nodes", + "memory.open_nodes", + "playwright.browser_navigate", + "memory.delete_observations" + ], + "hit1": "memory.read_graph", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 155, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 255, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 523, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "kind": "verb_product", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "figma.figma_get_file_nodes", + "memory.open_nodes", + "memory.delete_entities", + "memory.create_relations", + "memory.add_observations", + "memory.delete_observations", + "memory.create_entities", + "id_map.json", + "memory.delete_relations" + ], + "hit1": "memory.search_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 305, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 512, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "kind": "verb_product", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "ranking": [ + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_diff", + "git.git_show", + "git.git_branch", + "git.git_status", + "id_map.json", + "git.git_checkout", + "git.git_commit", + "git.git_add" + ], + "hit1": "git.git_diff_unstaged", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 297, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 540, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "kind": "identifier", + "gold": [ + "git.git_add" + ], + "alts": [], + "ranking": [ + "filesystem.list_directory_with_sizes", + "filesystem.search_files", + "git.git_show", + "git.git_branch", + "github.push_files", + "git.git_status", + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_add", + "git.git_checkout" + ], + "hit1": "filesystem.list_directory_with_sizes", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.1111111111111111, + "ndcg@10": 0.3010299956639812, + "hit_rank": 9, + "k3_calls": 3, + "k3_tokens": 634, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 675, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 654, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "kind": "verb_product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "ranking": [ + "shopify.shopify_create_product", + "shopify.shopify_update_product", + "shopify.shopify_get_product", + "shopify.shopify_list_products", + "shopify.shopify_list_orders", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "shopify.shopify_create_order", + "shopify.shopify_get_order", + "shopify.shopify_list_customers" + ], + "hit1": "shopify.shopify_create_product", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "kind": "verb_product", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "ranking": [ + "stripe.stripe_cancel_subscription", + "stripe.stripe_list_customers", + "stripe.stripe_list_subscriptions", + "stripe.stripe_create_invoice", + "stripe.stripe_create_price", + "stripe.stripe_create_product", + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_get_payment_intent", + "stripe.stripe_list_invoices" + ], + "hit1": "stripe.stripe_cancel_subscription", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 621, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "kind": "resource", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_refund", + "id_map.json", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment", + "github.create_package", + "github.create_artifact" + ], + "hit1": "stripe.stripe_create_payment_intent", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 802, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "id_map.json" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 626, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "kind": "verb_product", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "ranking": [ + "datadog.datadog_create_monitor", + "datadog.datadog_get_monitor", + "datadog.datadog_list_monitors", + "datadog.datadog_query_metrics", + "datadog.datadog_list_logs", + "id_map.json", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment" + ], + "hit1": "datadog.datadog_create_monitor", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 832, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "kind": "paraphrase", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "ranking": [ + "sentry.sentry_update_issue", + "sentry.sentry_get_issue", + "sentry.sentry_list_projects", + "pagerduty.pagerduty_resolve_incident", + "sentry.sentry_list_issues", + "sentry.sentry_list_events", + "jira.jira_get_issue", + "github.issue_write", + "github.issue_read", + "gitlab.gitlab_get_issue" + ], + "hit1": "sentry.sentry_update_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 823, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_delete", + "kubernetes.k8s_list_services", + "vercel.vercel_get_deployment", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_get_pod", + "id_map.json", + "kubernetes.k8s_apply", + "vercel.vercel_list_deployments" + ], + "hit1": "kubernetes.k8s_scale_deployment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 374, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_apply", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_delete", + "kubernetes.k8s_list_deployments", + "terraform.tf_apply", + "kubernetes.k8s_list_services", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_get_events", + "id_map.json" + ], + "hit1": "kubernetes.k8s_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 273, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 670, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "ranking": [ + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_run", + "docker.docker_logs", + "docker.docker_exec", + "docker.docker_build", + "docker.docker_ps", + "kubernetes.k8s_pod_logs", + "docker.docker_images", + "id_map.json" + ], + "hit1": "docker.docker_stop", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 587, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "kind": "verb_product", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "ranking": [ + "docker.docker_build", + "docker.docker_run", + "docker.docker_rm", + "docker.docker_stop", + "docker.docker_ps", + "docker.docker_images", + "filesystem.read_media_file", + "figma.figma_get_images", + "docker.docker_logs", + "id_map.json" + ], + "hit1": "docker.docker_build", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 571, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "kind": "verb_product", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "ranking": [ + "playwright.browser_click", + "puppeteer.puppeteer_click", + "playwright.browser_type", + "playwright.browser_take_screenshot", + "clickup.clickup_create_task", + "clickup.clickup_list_tasks", + "clickup.clickup_update_task", + "playwright.browser_snapshot", + "playwright.browser_navigate", + "asana.asana_get_task" + ], + "hit1": "playwright.browser_click", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "kind": "paraphrase", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "ranking": [ + "playwright.browser_type", + "playwright.browser_wait_for", + "playwright.browser_navigate", + "playwright.browser_tab_new", + "playwright.browser_tab_close", + "playwright.browser_tab_list", + "filesystem.read_text_file", + "playwright.browser_snapshot", + "mongodb.mongo_insert", + "playwright.browser_click" + ], + "hit1": "playwright.browser_type", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 614, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_list_projects", + "git.git_show", + "git.git_branch", + "git.git_status", + "github.projects_list", + "vercel.vercel_list_projects", + "git.git_checkout", + "sentry.sentry_list_projects", + "git.git_diff", + "git.git_diff_unstaged" + ], + "hit1": "gitlab.gitlab_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 565, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "kind": "verb_product", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "ranking": [ + "bitbucket.bitbucket_list_prs", + "bitbucket.bitbucket_create_issue", + "bitbucket.bitbucket_create_pr", + "github.pull_request_review_write", + "bitbucket.bitbucket_list_repos", + "github.update_pull_request", + "github.pull_request_read", + "github.update_pull_request_branch", + "gitlab.gitlab_get_merge_request", + "github.merge_pull_request" + ], + "hit1": "bitbucket.bitbucket_list_prs", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 509, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1020, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "kind": "verb_product", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "ranking": [ + "gmail.gmail_create_draft", + "gmail.gmail_send_email", + "gmail.gmail_list_labels", + "gmail.gmail_modify_email", + "gmail.gmail_trash_email", + "gmail.gmail_get_email", + "github.create_pull_request", + "gmail.gmail_search_emails", + "id_map.json", + "github.create_gist" + ], + "hit1": "gmail.gmail_create_draft", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 730, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "kind": "verb_product", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_get_file", + "google_drive.gdrive_share_file", + "google_drive.gdrive_search", + "google_drive.gdrive_update_file", + "google_drive.gdrive_create_file", + "gitlab.gitlab_get_file", + "google_drive.gdrive_list_folder", + "figma.figma_get_file", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event" + ], + "hit1": "google_drive.gdrive_get_file", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 778, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "kind": "verb_product", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "kubernetes.k8s_get_events", + "google_calendar.gcal_list_calendars", + "sentry.sentry_list_events", + "google_drive.gdrive_search", + "google_drive.gdrive_create_file", + "google_drive.gdrive_list_folder" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 441, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 770, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "id_map.json", + "kubernetes.k8s_delete", + "kubernetes.k8s_get_events", + "redis.redis_delete", + "github.pull_request_review_write" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 754, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_projects", + "vercel.vercel_get_deployment", + "vercel.vercel_list_deployments", + "vercel.vercel_create_project", + "sentry.sentry_list_projects", + "jira.jira_list_projects", + "github.projects_list", + "gitlab.gitlab_list_projects", + "linear.linear_list_projects", + "github.projects_get" + ], + "hit1": "vercel.vercel_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 286, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "ranking": [ + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update", + "supabase.supabase_list_tables", + "supabase.supabase_invoke_fn", + "github.delete_file", + "elasticsearch.es_get_doc", + "mongodb.mongo_delete", + "aws_s3.s3_get_object", + "puppeteer.puppeteer_click" + ], + "hit1": "supabase.supabase_select", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "kind": "verb_product", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "figma.figma_get_comments", + "figma.figma_get_file", + "figma.figma_get_images", + "figma.figma_get_file_nodes", + "linear.create_comment", + "notion.create_comment", + "github.update_issue_comment", + "linear.get_comment", + "linear.update_comment" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "kind": "verb_product", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "ranking": [ + "okta.okta_create_user", + "okta.okta_list_users", + "okta.okta_deactivate_user", + "slack.slack_list_users", + "okta.okta_get_user", + "linear.list_users", + "notion.list_users_tokens", + "github.search_users", + "notion.delete_users_token", + "notion.notion_list_users" + ], + "hit1": "okta.okta_create_user", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 298, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 595, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query_range", + "prometheus.prom_list_metrics", + "prometheus.prom_query", + "id_map.json", + "postgres.query", + "postgres.explain_query", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "datadog.datadog_query_metrics" + ], + "hit1": "prometheus.prom_query_range", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 274, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 595, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "kind": "verb_product", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "ranking": [ + "grafana.grafana_create_annotation", + "grafana.grafana_get_dashboard", + "grafana.grafana_search_dashboards", + "id_map.json", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment", + "github.create_package", + "github.create_artifact" + ], + "hit1": "grafana.grafana_create_annotation", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 786, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "kind": "identifier", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "ranking": [ + "terraform.tf_apply", + "terraform.tf_plan", + "kubernetes.k8s_apply", + "terraform.tf_state_list", + "id_map.json" + ], + "hit1": "terraform.tf_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 238, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 595, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "kind": "verb_product", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "ranking": [ + "asana.asana_list_tasks", + "asana.asana_get_task", + "asana.asana_create_task", + "asana.asana_update_task", + "clickup.clickup_list_tasks", + "asana.asana_create_project", + "id_map.json", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services" + ], + "hit1": "asana.asana_list_tasks", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "ranking": [ + "trello.trello_add_comment", + "trello.trello_list_cards", + "trello.trello_create_card", + "figma.figma_post_comment", + "trello.trello_create_board", + "github.update_issue_comment", + "linear.create_comment", + "notion.create_comment", + "linear.get_comment", + "github.add_comment_to_pending_review" + ], + "hit1": "trello.trello_add_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 892, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_update_task", + "clickup.clickup_create_task", + "clickup.clickup_get_task", + "clickup.clickup_list_tasks", + "asana.asana_get_task", + "puppeteer.puppeteer_click", + "asana.asana_update_task", + "playwright.browser_click", + "asana.asana_create_task", + "mongodb.mongo_update" + ], + "hit1": "clickup.clickup_update_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 761, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "kind": "verb_product", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "ranking": [ + "confluence.confluence_get_page", + "confluence.confluence_search", + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "github.search_issues", + "github.search_users", + "github.search_orgs", + "elasticsearch.es_search" + ], + "hit1": "confluence.confluence_get_page", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 731, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_deal", + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_search_contacts", + "github.delete_file", + "github.create_gist", + "github.pull_request_review_write", + "github.create_repository", + "github.get_discussion_comments" + ], + "hit1": "hubspot.hubspot_create_deal", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 872, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_search", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_create_ticket", + "id_map.json", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services" + ], + "hit1": "zendesk.zendesk_list_tickets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 604, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "kind": "verb_product", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "ranking": [ + "discord.discord_list_channels", + "discord.discord_get_channel", + "discord.discord_add_reaction", + "discord.discord_list_members", + "discord.discord_send_message", + "slack.slack_list_channels", + "id_map.json", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments" + ], + "hit1": "discord.discord_list_channels", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 573, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "redis.redis_set", + "redis.redis_keys", + "redis.redis_delete", + "playwright.browser_press_key", + "jira.jira_get_issue", + "figma.figma_post_comment", + "slack.slack_set_channel_topic", + "figma.figma_get_comments", + "aws_s3.s3_get_object" + ], + "hit1": "redis.redis_get", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 605, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert", + "mongodb.mongo_delete", + "mongodb.mongo_update", + "mongodb.mongo_find", + "mongodb.mongo_list_collections", + "supabase.supabase_insert", + "id_map.json" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 607, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "kind": "verb_product", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "ranking": [ + "mysql.mysql_query", + "postgres.query", + "postgres.explain_query", + "mysql.mysql_list_tables", + "postgres.list_schemas", + "sqlite.sqlite_query", + "postgres.list_tables", + "postgres.describe_table", + "salesforce.salesforce_query", + "prometheus.prom_query" + ], + "hit1": "mysql.mysql_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 266, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 543, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "elasticsearch.es_search", + "mongodb.mongo_find", + "playwright.browser_tab_close", + "id_map.json", + "github.create_milestone", + "mongodb.mongo_insert", + "github.get_milestone", + "jira.update_sprint" + ], + "hit1": "elasticsearch.es_index_doc", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_delete_object", + "aws_s3.s3_get_object", + "aws_s3.s3_put_object", + "asana.asana_get_task", + "aws_s3.s3_list_objects", + "asana.asana_create_task", + "id_map.json", + "okta.okta_get_user", + "asana.asana_update_task", + "elasticsearch.es_search" + ], + "hit1": "aws_s3.s3_delete_object", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones", + "cloudflare.cf_purge_cache", + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "time.convert_time", + "clickup.clickup_list_tasks", + "id_map.json", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "stripe.stripe_list_invoices", + "stripe.stripe_list_customers", + "stripe.stripe_create_customer", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_refund", + "stripe.stripe_get_customer", + "id_map.json" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 575, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "kind": "verb_product", + "gold": [ + "github.actions_list" + ], + "alts": [], + "ranking": [ + "github.actions_list", + "github.actions_run_trigger", + "git.git_show", + "git.git_branch", + "github.actions_get_job_logs", + "github.actions_get", + "git.git_status", + "git.git_diff_unstaged", + "git.git_checkout", + "git.git_diff" + ], + "hit1": "github.actions_list", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 408, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "kind": "verb_product", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "ranking": [ + "github.actions_get_job_logs", + "github.actions_run_trigger", + "git.git_show", + "github.actions_list", + "github.actions_get", + "git.git_branch", + "gitlab.delete_job", + "git.git_log", + "git.git_diff_unstaged", + "git.git_checkout" + ], + "hit1": "github.actions_get_job_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 296, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 449, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "kind": "verb_product", + "gold": [ + "github.create_repository" + ], + "alts": [], + "ranking": [ + "github.star_repository", + "github.create_repository", + "github.unstar_repository", + "github.fork_repository", + "github.get_repository_tree", + "git.git_status", + "git.git_show", + "github.delete_repository", + "git.git_branch", + "git.git_diff_unstaged" + ], + "hit1": "github.star_repository", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "kind": "verb_product", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "ranking": [ + "git.git_checkout", + "git.git_branch", + "git.git_create_branch", + "git.git_show", + "github.create_branch", + "git.git_status", + "git.git_diff_unstaged", + "git.git_diff", + "github.update_pull_request_branch", + "git.git_commit" + ], + "hit1": "git.git_checkout", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "kind": "identifier", + "gold": [ + "git.git_log" + ], + "alts": [], + "ranking": [ + "git.git_show", + "git.git_log", + "git.git_branch", + "git.git_status", + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_diff", + "git.git_checkout", + "datadog.datadog_list_logs", + "git.git_commit" + ], + "hit1": "git.git_show", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 634, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "kind": "verb_product", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "ranking": [ + "postgres.list_schemas", + "postgres.list_tables", + "postgres.describe_table", + "postgres.query", + "postgres.explain_query", + "id_map.json", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services" + ], + "hit1": "postgres.list_schemas", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 169, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 514, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "kind": "verb_product", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "ranking": [ + "memory.open_nodes", + "memory.search_nodes", + "memory.create_relations", + "memory.delete_entities", + "figma.figma_get_file_nodes", + "memory.read_graph", + "memory.create_entities", + "memory.delete_relations", + "id_map.json", + "jira.update_sprint" + ], + "hit1": "memory.open_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 308, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 507, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "kind": "paraphrase", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "ranking": [ + "filesystem.read_media_file", + "google_drive.gdrive_get_file", + "filesystem.get_file_info", + "gitlab.gitlab_get_file", + "filesystem.read_file", + "memory.delete_entities", + "filesystem.read_text_file", + "google_drive.gdrive_update_file", + "figma.figma_get_file", + "filesystem.write_file" + ], + "hit1": "filesystem.read_media_file", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 191, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 692, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "kind": "paraphrase", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "ranking": [ + "playwright.browser_tab_new", + "brave_search.brave_local_search", + "filesystem.edit_file", + "filesystem.read_file", + "figma.figma_get_file", + "filesystem.read_media_file", + "filesystem.write_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_update_file", + "gitlab.gitlab_get_file" + ], + "hit1": "playwright.browser_tab_new", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.14285714285714285, + "ndcg@10": 0.3333333333333333, + "hit_rank": 7, + "k3_calls": 3, + "k3_tokens": 667, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 737, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 716, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "kind": "verb_product", + "gold": [ + "github.search_issues" + ], + "alts": [], + "ranking": [ + "git.git_show", + "github.list_issues", + "git.git_branch", + "github.search_issues", + "git.git_status", + "jira.jira_search_issues", + "git.git_checkout", + "git.git_diff_unstaged", + "sentry.sentry_list_issues", + "git.git_diff" + ], + "hit1": "git.git_show", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 459, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "kind": "paraphrase", + "gold": [ + "github.get_me" + ], + "alts": [], + "ranking": [ + "git.git_show", + "git.git_status", + "git.git_branch", + "git.git_commit", + "github.list_notifications", + "git.git_add", + "git.git_diff_unstaged", + "git.git_checkout", + "git.git_diff_staged", + "playwright.browser_navigate" + ], + "hit1": "git.git_show", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 11, + "k3_calls": 4, + "k3_tokens": 751, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 3, + "k5_tokens": 994, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 1.0, + "k10_calls": 2, + "k10_tokens": 1377, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "cost": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.16, + "p95_calls": 2.0, + "success_within_1": 0.9, + "success_within_2": 0.9533333333333334, + "success_within_3": 0.9866666666666667, + "mean_tokens": 268.26666666666665, + "p95_tokens": 577.2999999999993, + "total_tokens": 40240, + "tokens_per_successful_query": 268.26666666666665, + "total_tokens_successful": 40240 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.0533333333333332, + "p95_calls": 1.0, + "success_within_1": 0.9533333333333334, + "success_within_2": 0.9933333333333333, + "success_within_3": 1.0, + "mean_tokens": 384.94, + "p95_tokens": 587.1999999999979, + "total_tokens": 57741, + "tokens_per_successful_query": 384.94, + "total_tokens_successful": 57741 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.0066666666666666, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 715.54, + "p95_tokens": 946.6999999999991, + "total_tokens": 107331, + "tokens_per_successful_query": 715.54, + "total_tokens_successful": 107331 + } + } +} diff --git a/experiments/mcp-tool-search/results/hybrid_codedb_weighted_bm25.csv b/experiments/mcp-tool-search/results/hybrid_codedb_weighted_bm25.csv new file mode 100644 index 00000000..3183bcfb --- /dev/null +++ b/experiments/mcp-tool-search/results/hybrid_codedb_weighted_bm25.csv @@ -0,0 +1,151 @@ +id,query,kind,hit1,recall@1,recall@3,recall@5,mrr@10,ndcg@10,hit_rank,k3_calls,k3_tokens,k3_success1,k5_calls,k5_tokens,k5_success1,k10_calls,k10_tokens,k10_success1,ranking +q001,Create a GitHub issue,verb_product,github.create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,388,1.0,1,598,1.0,1,1111,1.0,github.create_issue github.issue_write github.issue_read github.add_issue_comment github.update_issue_comment git.git_create_branch gitlab.gitlab_create_issue bitbucket.bitbucket_create_issue linear.linear_create_issue jira.jira_create_issue +q002,Create a Jira ticket,verb_product,jira.jira_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,333,1.0,1,458,1.0,1,795,1.0,jira.jira_create_issue jira.jira_create_sprint zendesk.zendesk_create_ticket jira.jira_assign_issue zendesk.zendesk_get_ticket jira.jira_list_projects zendesk.zendesk_update_ticket jira.jira_update_issue jira.create_attachment jira.jira_get_issue +q003,Send a Slack message,verb_product,slack.slack_post_message,1.0,1.0,1.0,1.0000,1.0000,1,1,232,1.0,1,356,1.0,1,755,1.0,slack.slack_post_message slack.slack_add_reaction slack.slack_join_channel slack.slack_leave_channel slack.slack_create_channel slack.slack_search_messages discord.discord_send_message twilio.twilio_send_sms gmail.gmail_send_email slack.slack_upload_file +q004,List open pull requests,resource,github.list_pull_requests,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,489,1.0,1,836,1.0,github.list_pull_requests github.search_pull_requests gitlab.gitlab_list_merge_requests github.create_pull_request github.merge_pull_request bitbucket.bitbucket_list_prs github.pull_request_read github.update_pull_request_branch playwright.browser_tab_list memory.open_nodes +q005,Open a pull request on GitHub,verb_product,github.create_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,368,1.0,1,653,1.0,1,1144,1.0,github.create_pull_request github.merge_pull_request github.pull_request_read github.pull_request_review_write github.update_pull_request github.update_pull_request_branch github.request_copilot_review github.search_pull_requests github.list_pull_requests github.add_comment_to_pending_review +q006,Run a SQL query against postgres,verb_product,postgres.query,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,272,1.0,1,551,1.0,postgres.query postgres.explain_query mysql.mysql_query postgres.list_schemas sqlite.sqlite_query postgres.list_tables salesforce.salesforce_query postgres.describe_table github.delete_check_run prometheus.prom_query +q007,Read a local file from disk,paraphrase,filesystem.read_file,1.0,1.0,1.0,1.0000,0.9502,1,1,223,1.0,1,428,1.0,1,797,1.0,filesystem.read_file filesystem.read_media_file filesystem.read_text_file gitlab.gitlab_get_file github.delete_file brave_search.brave_local_search figma.figma_get_file_nodes filesystem.write_file filesystem.edit_file filesystem.get_file_info +q008,What files are in this folder,paraphrase,google_drive.gdrive_list_folder,0.0,0.0,0.0,0.1667,0.4348,6,2,473,0.0,2,714,0.0,1,692,1.0,google_drive.gdrive_list_folder filesystem.search_files slack.list_files github.push_files filesystem.move_file filesystem.list_directory filesystem.list_directory_with_sizes filesystem.directory_tree cloudflare.cf_purge_cache filesystem.read_file +q009,Remember this fact in the knowledge graph,paraphrase,memory.read_graph,0.0,1.0,1.0,0.5000,0.5672,2,1,156,1.0,1,261,1.0,1,511,1.0,memory.read_graph memory.create_entities memory.create_relations memory.add_observations memory.delete_relations memory.open_nodes memory.search_nodes memory.delete_entities postgres.list_schemas memory.delete_observations +q010,Search the web with Brave,verb_product,brave_search.brave_web_search,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,374,1.0,1,712,1.0,brave_search.brave_web_search brave_search.brave_local_search jira.jira_search_issues filesystem.list_directory_with_sizes memory.search_nodes github.update_pull_request_branch grafana.grafana_search_dashboards puppeteer.puppeteer_screenshot github.search_users puppeteer.puppeteer_navigate +q011,Fetch the contents of a URL,verb_product,fetch.fetch,1.0,1.0,1.0,1.0000,1.0000,1,1,264,1.0,1,380,1.0,1,707,1.0,fetch.fetch github.get_file_contents filesystem.read_file playwright.browser_navigate git.git_add slack.slack_set_channel_topic playwright.browser_snapshot git.git_status puppeteer.puppeteer_navigate github.get_repository_tree +q012,Show git status,identifier,git.git_status,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,312,1.0,1,615,1.0,git.git_status git.git_show git.git_diff git.git_diff_unstaged git.git_log git.git_diff_staged git.git_checkout git.git_branch git.git_commit gitlab.list_releases +q013,Commit staged changes,paraphrase,git.git_commit,1.0,1.0,1.0,1.0000,1.0000,1,1,215,1.0,1,327,1.0,1,596,1.0,git.git_commit git.git_reset github.get_commit git.git_diff_staged git.git_show git.git_log github.push_files id_map.json +q014,Create a Linear issue,verb_product,linear.linear_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,293,1.0,1,463,1.0,1,876,1.0,linear.linear_create_issue linear.linear_assign_issue linear.linear_create_comment linear.linear_update_issue linear.linear_get_issue linear.linear_create_project bitbucket.bitbucket_create_issue gitlab.gitlab_create_issue linear.linear_list_teams jira.jira_create_issue +q015,Make a task in my project tracker Linear,paraphrase,linear.linear_create_project,0.0,1.0,1.0,0.3333,0.5000,3,1,326,1.0,1,494,1.0,1,806,1.0,linear.linear_create_project twilio.twilio_make_call linear.linear_create_issue clickup.clickup_get_task linear.linear_update_issue asana.asana_get_task linear.linear_list_teams clickup.clickup_update_task linear.linear_list_projects asana.asana_update_task +q016,Post a comment on a GitHub issue,verb_product,github.update_issue_comment,0.0,1.0,1.0,0.5000,0.6309,2,1,333,1.0,1,569,1.0,1,1072,1.0,github.update_issue_comment github.add_issue_comment github.issue_read github.assign_copilot_to_issue github.create_issue github.issue_write figma.figma_post_comment github.add_comment_to_pending_review github.list_repository_collaborators git.git_checkout +q017,Merge the pull request,ambiguous,github.merge_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,283,1.0,1,468,1.0,1,1121,1.0,github.merge_pull_request gitlab.gitlab_merge_merge_request gitlab.gitlab_get_merge_request github.update_pull_request_branch github.pull_request_read gitlab.gitlab_create_merge_request github.create_pull_request github.update_pull_request github.pull_request_review_write github.request_copilot_review +q018,Trigger a GitHub Actions workflow,verb_product,github.actions_run_trigger,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,458,1.0,1,729,1.0,github.actions_run_trigger github.actions_get github.actions_list github.actions_get_job_logs circleci.circleci_trigger_pipeline git.git_checkout github.search_users git.git_show github.search_orgs git.git_branch +q019,Get Dependabot alerts,resource,github.list_dependabot_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,412,1.0,1,686,1.0,github.list_dependabot_alerts github.get_dependabot_alert github.list_secret_scanning_alerts github.list_code_scanning_alerts redis.redis_get id_map.json kubernetes.k8s_get_pod discord.discord_get_channel vercel.vercel_get_deployment github.get_discussion +q020,List Sentry issues,verb_product,sentry.sentry_list_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,380,1.0,1,796,1.0,sentry.sentry_list_issues sentry.sentry_list_projects sentry.sentry_list_events sentry.sentry_get_issue gitlab.gitlab_list_issues github.list_issues sentry.sentry_update_issue linear.linear_list_issues github.search_issues linear.linear_search_issues +q021,Create a Stripe customer,verb_product,stripe.stripe_create_customer,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,341,1.0,1,662,1.0,stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_create_invoice stripe.stripe_create_product stripe.stripe_create_payment_intent stripe.stripe_list_invoices stripe.stripe_list_subscriptions stripe.stripe_create_refund stripe.stripe_create_price stripe.stripe_get_payment_intent +q022,Refund a Stripe payment,paraphrase,stripe.stripe_create_refund,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,332,1.0,1,632,1.0,stripe.stripe_create_refund stripe.stripe_get_payment_intent stripe.stripe_create_payment_intent stripe.stripe_get_customer stripe.stripe_create_invoice stripe.stripe_list_customers stripe.stripe_list_invoices stripe.stripe_create_product stripe.stripe_create_customer stripe.stripe_cancel_subscription +q023,Send an email with Gmail,verb_product,gmail.gmail_send_email,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,389,1.0,1,716,1.0,gmail.gmail_send_email gmail.gmail_trash_email gmail.gmail_get_email gmail.gmail_modify_email gmail.gmail_search_emails twilio.twilio_send_sms gmail.gmail_list_labels gmail.gmail_create_draft filesystem.list_directory_with_sizes discord.discord_send_message +q024,Search my Gmail inbox,verb_product,gmail.gmail_search_emails,1.0,1.0,1.0,1.0000,1.0000,1,1,205,1.0,1,315,1.0,1,700,1.0,gmail.gmail_search_emails gmail.gmail_list_labels gmail.gmail_create_draft gmail.gmail_trash_email gmail.gmail_get_email gmail.gmail_send_email gmail.gmail_modify_email grafana.grafana_search_dashboards brave_search.brave_web_search github.search_users +q025,Upload a file to Google Drive,paraphrase,google_drive.gdrive_update_file,0.0,0.0,1.0,0.2000,0.3869,5,2,523,0.0,1,446,1.0,1,827,1.0,google_drive.gdrive_update_file google_drive.gdrive_get_file google_drive.gdrive_share_file slack.slack_upload_file google_drive.gdrive_create_file google_drive.gdrive_search google_drive.gdrive_list_folder filesystem.edit_file gitlab.gitlab_get_file aws_s3.s3_put_object +q026,Create a calendar event,resource,google_calendar.gcal_create_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,427,1.0,1,810,1.0,google_calendar.gcal_create_event google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars github.pull_request_review_write github.create_gist id_map.json github.create_milestone trello.trello_create_board +q027,List Kubernetes pods,verb_product,kubernetes.k8s_list_pods,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,303,1.0,1,665,1.0,kubernetes.k8s_list_pods kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_apply kubernetes.k8s_get_pod kubernetes.k8s_scale_deployment kubernetes.k8s_delete kubernetes.k8s_pod_logs id_map.json +q028,Get logs from a k8s pod,verb_product,kubernetes.k8s_pod_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,455,1.0,1,822,1.0,kubernetes.k8s_pod_logs kubernetes.k8s_get_pod github.actions_get_job_logs docker.docker_logs datadog.datadog_list_logs elasticsearch.es_get_doc aws_s3.s3_get_object gitlab.gitlab_get_file git.git_log slack.slack_get_channel_history +q029,Run a docker container,verb_product,docker.docker_run,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,345,1.0,1,597,1.0,docker.docker_run docker.docker_stop docker.docker_exec docker.docker_rm docker.docker_logs docker.docker_images docker.docker_ps docker.docker_build github.get_check_run github.delete_check_run +q030,Show docker container logs,verb_product,docker.docker_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,345,1.0,1,702,1.0,docker.docker_logs docker.docker_stop docker.docker_rm docker.docker_exec docker.docker_run kubernetes.k8s_pod_logs datadog.datadog_list_logs docker.docker_images git.git_log docker.docker_build +q031,Navigate the browser to a URL,ambiguous,playwright.browser_navigate,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,294,1.0,1,536,1.0,playwright.browser_navigate puppeteer.puppeteer_navigate playwright.browser_tab_new playwright.browser_evaluate playwright.browser_click playwright.browser_press_key playwright.browser_snapshot playwright.browser_wait_for playwright.browser_tab_close playwright.browser_tab_list +q032,Take a screenshot of the page,ambiguous,playwright.browser_take_screenshot,1.0,1.0,1.0,1.0000,1.0000,1,1,226,1.0,1,352,1.0,1,711,1.0,playwright.browser_take_screenshot puppeteer.puppeteer_screenshot notion.notion_get_page notion.notion_update_page confluence.confluence_get_page playwright.browser_snapshot confluence.confluence_create_page confluence.confluence_update_page notion.notion_create_page playwright.browser_evaluate +q033,Create a GitLab merge request,verb_product,gitlab.gitlab_create_merge_request,1.0,1.0,1.0,1.0000,1.0000,1,1,297,1.0,1,475,1.0,1,844,1.0,gitlab.gitlab_create_merge_request gitlab.gitlab_merge_merge_request gitlab.gitlab_get_merge_request gitlab.gitlab_list_merge_requests github.merge_pull_request git.git_create_branch gitlab.gitlab_create_pipeline git.git_checkout gitlab.gitlab_create_issue git.git_show +q034,Search Notion pages,verb_product,notion.notion_search,1.0,1.0,1.0,1.0000,1.0000,1,1,195,1.0,1,318,1.0,1,639,1.0,notion.notion_search notion.notion_get_page notion.notion_get_block notion.notion_list_users notion.notion_query_database grafana.grafana_search_dashboards notion.notion_create_database github.search_users notion.notion_update_page hubspot.hubspot_search_contacts +q035,Create a Notion page,verb_product,notion.notion_get_page,0.0,1.0,1.0,0.5000,0.6309,2,1,230,1.0,1,371,1.0,1,784,1.0,notion.notion_get_page notion.notion_create_page notion.notion_update_page notion.notion_get_comments notion.notion_create_database confluence.confluence_create_page notion.notion_search notion.create_comment notion.notion_get_block notion.create_data_source +q036,Query a Notion database,verb_product,notion.notion_query_database,1.0,1.0,1.0,1.0000,1.0000,1,1,279,1.0,1,401,1.0,1,669,1.0,notion.notion_query_database notion.notion_create_database notion.notion_create_page notion.notion_search notion.notion_get_page notion.notion_get_block notion.notion_list_users notion.notion_update_page notion.notion_get_comments postgres.query +q037,Create a Zendesk ticket,verb_product,zendesk.zendesk_create_ticket,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,429,1.0,1,723,1.0,zendesk.zendesk_create_ticket zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket jira.jira_create_issue zendesk.zendesk_search zendesk.zendesk_list_tickets id_map.json github.create_gist trello.trello_create_board github.create_milestone +q038,Create a HubSpot contact,verb_product,hubspot.hubspot_create_contact,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,899,1.0,hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_create_deal hubspot.hubspot_search_contacts hubspot.hubspot_list_contacts github.create_gist github.create_repository github.create_branch github.create_issue github.create_or_update_file +q039,Create a Salesforce lead,verb_product,salesforce.salesforce_create_lead,1.0,1.0,1.0,1.0000,1.0000,1,1,252,1.0,1,364,1.0,1,682,1.0,salesforce.salesforce_create_lead salesforce.salesforce_convert_lead salesforce.salesforce_create_record salesforce.salesforce_query salesforce.salesforce_list_opportunities salesforce.salesforce_delete_record salesforce.salesforce_update_record salesforce.salesforce_get_record id_map.json github.create_gist +q040,Run a SOQL query,identifier,salesforce.salesforce_query,1.0,1.0,1.0,1.0000,1.0000,1,1,187,1.0,1,332,1.0,1,616,1.0,salesforce.salesforce_query postgres.query mysql.mysql_query sqlite.sqlite_query docker.docker_run github.delete_check_run prometheus.prom_query github.get_check_run github.update_check_run postgres.explain_query +q041,List Shopify orders,verb_product,shopify.shopify_list_orders,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,320,1.0,1,606,1.0,shopify.shopify_list_orders shopify.shopify_list_customers shopify.shopify_list_products shopify.shopify_get_product shopify.shopify_get_order shopify.shopify_update_product shopify.shopify_create_product shopify.shopify_create_order clickup.clickup_list_tasks id_map.json +q042,Create a PagerDuty incident,verb_product,pagerduty.pagerduty_create_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,403,1.0,1,705,1.0,pagerduty.pagerduty_create_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls id_map.json github.create_gist trello.trello_create_board github.create_milestone +q043,Who is on call in PagerDuty,paraphrase,pagerduty.pagerduty_list_oncalls,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,363,1.0,1,738,1.0,pagerduty.pagerduty_list_oncalls pagerduty.pagerduty_get_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_create_incident twilio.twilio_make_call filesystem.list_allowed_directories slack.slack_create_channel trello.trello_add_comment +q044,Query Datadog metrics,verb_product,datadog.datadog_query_metrics,1.0,1.0,1.0,1.0000,1.0000,1,1,284,1.0,1,449,1.0,1,767,1.0,datadog.datadog_query_metrics datadog.datadog_list_monitors datadog.datadog_list_logs datadog.datadog_create_monitor prometheus.prom_list_metrics datadog.datadog_get_monitor postgres.query prometheus.prom_query postgres.explain_query prometheus.prom_query_range +q045,Search Datadog logs,verb_product,datadog.datadog_list_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,458,1.0,1,860,1.0,datadog.datadog_list_logs datadog.datadog_list_monitors datadog.datadog_get_monitor datadog.datadog_query_metrics datadog.datadog_create_monitor docker.docker_logs kubernetes.k8s_pod_logs github.actions_get_job_logs grafana.grafana_search_dashboards brave_search.brave_web_search +q046,List S3 buckets,verb_product,aws_s3.s3_list_buckets,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,327,1.0,1,598,1.0,aws_s3.s3_list_buckets aws_s3.s3_list_objects clickup.clickup_list_tasks aws_s3.s3_delete_object aws_s3.s3_get_object trello.trello_list_cards aws_s3.s3_put_object kubernetes.k8s_list_deployments kubernetes.k8s_list_services id_map.json +q047,Upload an object to S3,paraphrase,aws_s3.s3_put_object,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,437,1.0,1,868,1.0,aws_s3.s3_put_object aws_s3.s3_delete_object aws_s3.s3_get_object slack.slack_upload_file github.assign_copilot_to_issue twilio.twilio_send_sms slack.slack_add_reaction gmail.gmail_send_email aws_s3.s3_list_objects slack.slack_reply_to_thread +q048,Purge Cloudflare cache,verb_product,cloudflare.cf_purge_cache,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,286,1.0,1,643,1.0,cloudflare.cf_purge_cache cloudflare.cf_list_zones cloudflare.cf_list_dns cloudflare.cf_create_dns id_map.json +q049,Create a Cloudflare DNS record,verb_product,cloudflare.cf_create_dns,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,356,1.0,1,660,1.0,cloudflare.cf_create_dns cloudflare.cf_list_dns salesforce.salesforce_create_record cloudflare.cf_list_zones cloudflare.cf_purge_cache salesforce.salesforce_delete_record salesforce.salesforce_update_record salesforce.salesforce_get_record git.git_commit id_map.json +q050,List Vercel deployments,verb_product,vercel.vercel_list_deployments,1.0,1.0,1.0,1.0000,1.0000,1,1,173,1.0,1,291,1.0,1,506,1.0,vercel.vercel_list_deployments vercel.vercel_list_projects kubernetes.k8s_list_deployments vercel.vercel_get_deployment vercel.vercel_create_project clickup.clickup_list_tasks id_map.json trello.trello_list_cards jira.jira_list_projects kubernetes.k8s_list_services +q051,Insert a row into Supabase,verb_product,supabase.supabase_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,230,1.0,1,347,1.0,1,598,1.0,supabase.supabase_insert supabase.supabase_select mongodb.mongo_insert supabase.supabase_list_tables supabase.supabase_invoke_fn supabase.supabase_update playwright.browser_type id_map.json +q052,Export images from Figma,paraphrase,figma.figma_get_images,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,328,1.0,1,695,1.0,figma.figma_get_images figma.figma_get_file_nodes figma.figma_get_file docker.docker_images figma.figma_get_comments figma.figma_post_comment elasticsearch.es_get_doc memory.delete_observations aws_s3.s3_get_object datadog.datadog_list_logs +q053,Send an SMS with Twilio,verb_product,twilio.twilio_send_sms,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,401,1.0,1,668,1.0,twilio.twilio_send_sms twilio.twilio_list_messages twilio.twilio_make_call gmail.gmail_send_email filesystem.list_directory_with_sizes discord.discord_send_message puppeteer.puppeteer_click puppeteer.puppeteer_fill id_map.json puppeteer.puppeteer_screenshot +q054,Create an Okta user,verb_product,okta.okta_create_user,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,329,1.0,1,676,1.0,okta.okta_create_user okta.okta_get_user okta.okta_deactivate_user linear.create_user okta.okta_list_users linear.get_user asana.asana_create_task slack.slack_get_user_profile asana.asana_create_project linear.update_user +q055,Instant query Prometheus,verb_product,prometheus.prom_query,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,329,1.0,1,680,1.0,prometheus.prom_query prometheus.prom_query_range prometheus.prom_list_metrics postgres.query postgres.explain_query datadog.datadog_query_metrics salesforce.salesforce_query mysql.mysql_query sqlite.sqlite_query notion.notion_query_database +q056,Search Grafana dashboards,verb_product,grafana.grafana_search_dashboards,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,321,1.0,1,681,1.0,grafana.grafana_search_dashboards grafana.grafana_get_dashboard grafana.grafana_create_annotation github.search_users confluence.confluence_search github.search_orgs brave_search.brave_web_search hubspot.hubspot_search_contacts brave_search.brave_local_search github.search_issues +q057,Trigger a CircleCI pipeline,verb_product,circleci.circleci_trigger_pipeline,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,390,1.0,1,664,1.0,circleci.circleci_trigger_pipeline circleci.circleci_get_pipeline circleci.circleci_list_pipelines gitlab.gitlab_create_pipeline github.actions_run_trigger gitlab.get_pipeline gitlab.create_pipeline gitlab.delete_pipeline gitlab.update_pipeline id_map.json +q058,Run terraform plan,identifier,terraform.tf_plan,1.0,1.0,1.0,1.0000,1.0000,1,1,172,1.0,1,319,1.0,1,717,1.0,terraform.tf_plan terraform.tf_apply terraform.tf_state_list github.get_check_run docker.docker_run github.delete_check_run postgres.explain_query github.update_check_run github.create_check_run github.actions_run_trigger +q059,Create an Asana task,verb_product,asana.asana_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,405,1.0,1,825,1.0,asana.asana_create_task asana.asana_get_task asana.asana_update_task asana.asana_create_project asana.asana_list_tasks clickup.clickup_create_task clickup.clickup_get_task clickup.clickup_update_task okta.okta_create_user github.create_issue +q060,Create a Trello card,verb_product,trello.trello_create_card,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,316,1.0,1,700,1.0,trello.trello_create_card trello.trello_create_board trello.trello_add_comment trello.trello_list_cards id_map.json github.create_gist stripe.stripe_create_invoice github.create_milestone vercel.vercel_create_project github.create_hook +q061,Create a ClickUp task,verb_product,clickup.clickup_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,415,1.0,1,673,1.0,clickup.clickup_create_task clickup.clickup_get_task clickup.clickup_update_task clickup.clickup_list_tasks asana.asana_create_task puppeteer.puppeteer_click asana.asana_get_task playwright.browser_click asana.asana_update_task id_map.json +q062,Create a Confluence page,verb_product,confluence.confluence_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,401,1.0,1,652,1.0,confluence.confluence_create_page confluence.confluence_get_page confluence.confluence_update_page confluence.confluence_search notion.notion_create_page notion.notion_get_page notion.notion_update_page playwright.browser_take_screenshot id_map.json notion.notion_get_comments +q063,Send a Discord message,verb_product,discord.discord_send_message,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,360,1.0,1,628,1.0,discord.discord_send_message discord.discord_add_reaction discord.discord_get_channel discord.discord_list_channels twilio.twilio_send_sms gmail.gmail_send_email discord.discord_list_members slack.slack_post_message gmail.gmail_trash_email id_map.json +q064,Search Elasticsearch,verb_product,elasticsearch.es_search,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,343,1.0,1,672,1.0,elasticsearch.es_search elasticsearch.es_get_doc elasticsearch.es_index_doc github.search_users confluence.confluence_search github.search_orgs github.search_commits grafana.grafana_search_dashboards brave_search.brave_web_search brave_search.brave_local_search +q065,Get a Redis key,verb_product,redis.redis_get,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,281,1.0,1,589,1.0,redis.redis_get redis.redis_set redis.redis_delete jira.jira_get_issue redis.redis_keys figma.figma_get_file aws_s3.s3_get_object figma.figma_get_comments figma.figma_get_file_nodes playwright.browser_press_key +q066,Find MongoDB documents,verb_product,mongodb.mongo_find,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,381,1.0,1,631,1.0,mongodb.mongo_find mongodb.mongo_insert mongodb.mongo_delete mongodb.mongo_update mongodb.mongo_list_collections id_map.json +q067,List SQLite tables,verb_product,sqlite.sqlite_list_tables,1.0,1.0,1.0,1.0000,1.0000,1,1,151,1.0,1,240,1.0,1,420,1.0,sqlite.sqlite_list_tables mysql.mysql_list_tables postgres.list_tables supabase.supabase_list_tables sqlite.sqlite_describe_table sqlite.sqlite_query clickup.clickup_list_tasks id_map.json postgres.list_schemas vercel.vercel_list_projects +q068,Describe a postgres table,verb_product,postgres.describe_table,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,278,1.0,1,543,1.0,postgres.describe_table sqlite.sqlite_describe_table postgres.explain_query postgres.query postgres.list_schemas postgres.list_tables supabase.supabase_insert supabase.supabase_select supabase.supabase_update id_map.json +q069,github_create_issue,identifier,github.create_issue,1.0,1.0,1.0,1.0000,0.9502,1,1,386,1.0,1,596,1.0,1,1019,1.0,github.create_issue github.issue_read github.issue_write github.add_issue_comment github.update_issue_comment github.assign_copilot_to_issue bitbucket.bitbucket_create_issue sentry.sentry_get_issue gitlab.gitlab_create_issue jira.jira_assign_issue +q070,createIssue,identifier,github.create_issue,1.0,1.0,1.0,1.0000,0.7349,1,1,340,1.0,1,546,1.0,1,961,1.0,github.create_issue github.issue_read bitbucket.bitbucket_create_issue github.issue_write sentry.sentry_get_issue gitlab.gitlab_create_issue jira.jira_assign_issue jira.jira_get_issue linear.linear_create_issue gitlab.gitlab_get_issue +q071,list-directory,identifier,filesystem.create_directory,0.0,1.0,1.0,0.5000,0.6309,2,1,182,1.0,1,296,1.0,1,607,1.0,filesystem.create_directory filesystem.list_directory filesystem.directory_tree filesystem.list_directory_with_sizes filesystem.get_file_info filesystem.list_allowed_directories filesystem.read_file filesystem.read_text_file filesystem.read_media_file filesystem.write_file +q072,sequential thinking next thought,paraphrase,sequential_thinking.sequentialthinking,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,291,1.0,1,713,1.0,sequential_thinking.sequentialthinking id_map.json +q073,What time is it in Tokyo,paraphrase,time.get_current_time,1.0,1.0,1.0,1.0000,1.0000,1,1,222,1.0,1,329,1.0,1,678,1.0,time.get_current_time time.convert_time slack.slack_create_channel playwright.browser_wait_for filesystem.list_allowed_directories grafana.grafana_create_annotation google_calendar.gcal_list_events jira.jira_add_worklog id_map.json elasticsearch.es_index_doc +q074,Convert time between timezones,verb_product,time.convert_time,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,322,1.0,1,650,1.0,time.convert_time salesforce.salesforce_convert_lead time.get_current_time memory.create_relations playwright.browser_wait_for git.git_diff grafana.grafana_create_annotation google_calendar.gcal_list_events jira.jira_add_worklog id_map.json +q075,Star a GitHub repo,verb_product,github.star_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,205,1.0,1,333,1.0,1,684,1.0,github.star_repository github.unstar_repository git.git_branch github.delete_repository git.git_show github.actions_list git.git_checkout github.get_label git.git_status github.get_discussion +q076,Fork a repository,resource,github.fork_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,235,1.0,1,384,1.0,1,789,1.0,github.fork_repository github.star_repository github.delete_repository github.unstar_repository github.list_repository_collaborators github.create_repository github.get_repository_tree github.list_releases github.list_dependabot_alerts github.list_labels +q077,Get the latest GitHub release,verb_product,github.get_latest_release,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,341,1.0,1,680,1.0,github.get_latest_release github.get_release_by_tag gitlab.get_release git.git_commit github.get_teams git.git_status github.get_me git.git_add github.get_repository_tree git.git_checkout +q078,Create a gist,resource,github.create_gist,1.0,1.0,1.0,1.0000,1.0000,1,1,229,1.0,1,391,1.0,1,739,1.0,github.create_gist github.get_gist github.update_gist trello.trello_create_board github.create_milestone stripe.stripe_create_invoice github.create_hook vercel.vercel_create_project github.create_environment okta.okta_create_user +q079,Search GitHub code,verb_product,github.search_code,1.0,1.0,1.0,1.0000,1.0000,1,1,283,1.0,1,395,1.0,1,660,1.0,github.search_code github.list_code_scanning_alerts github.get_code_scanning_alert github.search_users git.git_checkout github.search_orgs git.git_show github.search_commits git.git_branch github.search_pull_requests +q080,List GitHub notifications,verb_product,github.list_notifications,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,286,1.0,1,595,1.0,github.list_notifications github.mark_all_notifications_read github.projects_list git.git_branch github.list_gists git.git_checkout github.actions_list git.git_show github.list_starred_repositories git.git_diff +q081,Assign Copilot to an issue,verb_product,github.assign_copilot_to_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,402,1.0,1,896,1.0,github.assign_copilot_to_issue linear.linear_assign_issue jira.jira_assign_issue jira.jira_update_issue jira.jira_transition_issue github.request_copilot_review github.add_issue_comment github.create_issue linear.linear_create_comment slack.slack_reply_to_thread +q082,Get secret scanning alerts,resource,github.list_secret_scanning_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,295,1.0,1,470,1.0,1,702,1.0,github.list_secret_scanning_alerts github.get_secret_scanning_alert github.list_code_scanning_alerts github.get_code_scanning_alert github.list_dependabot_alerts redis.redis_get id_map.json kubernetes.k8s_get_pod discord.discord_get_channel vercel.vercel_get_deployment +q083,Create or update a file on GitHub,verb_product,github.create_or_update_file,1.0,1.0,1.0,1.0000,0.7602,1,1,367,1.0,1,548,1.0,1,966,1.0,github.create_or_update_file github.get_file_contents google_drive.gdrive_update_file github.delete_file slack.update_file git.git_diff slack.create_file git.git_add google_drive.gdrive_create_file github.projects_write +q084,List collaborators on a repo,resource,github.list_repository_collaborators,1.0,1.0,1.0,1.0000,1.0000,1,1,264,1.0,1,416,1.0,1,824,1.0,github.list_repository_collaborators github.list_commits pagerduty.pagerduty_list_oncalls trello.trello_list_cards bitbucket.bitbucket_list_prs github.get_discussion_comments github.actions_list github.update_issue_comment github.list_releases trello.trello_add_comment +q085,Reply in a Slack thread,verb_product,slack.slack_reply_to_thread,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,359,1.0,1,655,1.0,slack.slack_reply_to_thread slack.slack_get_thread_replies slack.slack_list_channels slack.delete_file slack.slack_join_channel slack.get_canvase slack.slack_leave_channel slack.delete_canvase slack.slack_create_channel slack.delete_usergroup +q086,Add an emoji reaction on Slack,verb_product,slack.slack_add_reaction,1.0,1.0,1.0,1.0000,1.0000,1,1,252,1.0,1,375,1.0,1,768,1.0,slack.slack_add_reaction discord.discord_add_reaction slack.slack_join_channel trello.trello_add_comment slack.slack_leave_channel gmail.gmail_modify_email slack.slack_create_channel jira.jira_add_comment slack.slack_search_messages jira.jira_add_worklog +q087,Search Slack messages,verb_product,slack.slack_search_messages,1.0,1.0,1.0,1.0000,1.0000,1,1,232,1.0,1,336,1.0,1,652,1.0,slack.slack_search_messages slack.slack_get_channel_history gmail.gmail_search_emails slack.slack_join_channel twilio.twilio_list_messages slack.slack_leave_channel grafana.grafana_search_dashboards slack.slack_create_channel github.search_users slack.slack_upload_file +q088,Create a Slack channel,verb_product,slack.slack_create_channel,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,342,1.0,1,751,1.0,slack.slack_create_channel slack.slack_join_channel slack.slack_leave_channel slack.slack_set_channel_topic slack.slack_get_channel_history slack.slack_post_message slack.slack_upload_file slack.slack_add_reaction slack.slack_get_thread_replies slack.slack_reply_to_thread +q089,Transition a Jira issue,verb_product,jira.jira_transition_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,389,1.0,1,724,1.0,jira.jira_transition_issue jira.jira_assign_issue jira.jira_get_issue jira.jira_update_issue jira.jira_create_issue jira.jira_add_comment jira.jira_add_worklog gitlab.gitlab_get_issue jira.jira_list_projects sentry.sentry_get_issue +q090,Search Jira with JQL,identifier,jira.jira_search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,312,1.0,1,647,1.0,jira.jira_search_issues filesystem.list_directory_with_sizes jira.jira_assign_issue grafana.grafana_search_dashboards jira.jira_list_projects github.search_users jira.jira_create_sprint hubspot.hubspot_search_contacts jira.jira_update_issue github.search_orgs +q091,Add a worklog to Jira,verb_product,jira.jira_add_worklog,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,491,1.0,1,823,1.0,jira.jira_add_worklog jira.jira_add_comment jira.jira_transition_issue github.add_comment_to_pending_review discord.discord_add_reaction jira.jira_assign_issue memory.add_observations jira.jira_list_projects git.git_add jira.jira_create_sprint +q092,List Linear teams,verb_product,linear.linear_list_teams,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,277,1.0,1,755,1.0,linear.linear_list_teams linear.linear_list_projects linear.linear_list_issues linear.linear_search_issues github.get_teams linear.linear_update_issue linear.list_comments linear.linear_create_project linear.list_attachments linear.linear_create_issue +q093,Comment on a Linear issue,verb_product,linear.linear_create_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,276,1.0,1,464,1.0,1,888,1.0,linear.linear_create_comment linear.linear_update_issue linear.linear_assign_issue linear.linear_create_issue linear.linear_get_issue github.update_issue_comment github.add_issue_comment linear.create_comment linear.get_comment jira.jira_add_comment +q094,Move a local file,paraphrase,filesystem.move_file,1.0,1.0,1.0,1.0000,1.0000,1,1,214,1.0,1,358,1.0,1,688,1.0,filesystem.move_file filesystem.read_file brave_search.brave_local_search filesystem.read_media_file filesystem.read_text_file figma.figma_get_file slack.delete_file filesystem.write_file google_drive.gdrive_get_file slack.create_file +q095,Recursively search files by pattern,paraphrase,filesystem.search_files,1.0,1.0,1.0,1.0000,1.0000,1,1,280,1.0,1,382,1.0,1,735,1.0,filesystem.search_files slack.list_files github.push_files redis.redis_keys google_drive.gdrive_search linear.linear_search_issues github.search_repositories github.get_release_by_tag filesystem.move_file filesystem.list_directory +q096,Edit a file in place with old and new string,paraphrase,filesystem.edit_file,1.0,1.0,1.0,1.0000,1.0000,1,1,227,1.0,1,350,1.0,1,682,1.0,filesystem.edit_file filesystem.list_directory_with_sizes playwright.browser_tab_new filesystem.write_file slack.get_file filesystem.move_file slack.create_file slack.delete_file slack.update_file figma.figma_get_file +q097,Read the whole knowledge graph,paraphrase,memory.read_graph,1.0,1.0,1.0,1.0000,1.0000,1,1,152,1.0,1,260,1.0,1,562,1.0,memory.read_graph memory.delete_relations memory.create_entities memory.create_relations filesystem.read_file memory.add_observations memory.delete_entities memory.open_nodes filesystem.read_media_file filesystem.read_text_file +q098,Search memory nodes,verb_product,memory.search_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,202,1.0,1,304,1.0,1,546,1.0,memory.search_nodes memory.open_nodes figma.figma_get_file_nodes memory.create_entities memory.delete_observations memory.create_relations memory.read_graph memory.delete_relations memory.add_observations memory.delete_entities +q099,Show unstaged git diff,verb_product,git.git_diff_unstaged,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,303,1.0,1,537,1.0,git.git_diff_unstaged git.git_diff_staged git.git_diff git.git_show git.git_status git.git_log git.git_checkout git.git_branch id_map.json git.git_commit +q100,Stage files with git add,identifier,git.git_add,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,433,1.0,1,834,1.0,git.git_add filesystem.list_directory_with_sizes github.push_files filesystem.search_files github.add_issue_comment github.add_comment_to_pending_review git.git_checkout git.git_show github.create_gist git.git_branch +q101,Create a Shopify product,verb_product,shopify.shopify_create_product,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,377,1.0,1,702,1.0,shopify.shopify_create_product shopify.shopify_get_product shopify.shopify_update_product shopify.shopify_create_order stripe.stripe_create_product shopify.shopify_list_products shopify.shopify_list_customers shopify.shopify_get_order stripe.stripe_create_price shopify.shopify_list_orders +q102,Cancel a Stripe subscription,verb_product,stripe.stripe_cancel_subscription,1.0,1.0,1.0,1.0000,1.0000,1,1,168,1.0,1,282,1.0,1,610,1.0,stripe.stripe_cancel_subscription stripe.stripe_get_customer stripe.stripe_create_invoice stripe.stripe_list_customers stripe.stripe_list_invoices stripe.stripe_create_customer stripe.stripe_create_product stripe.stripe_create_refund stripe.stripe_list_subscriptions stripe.stripe_get_payment_intent +q103,Create a payment intent,resource,stripe.stripe_create_payment_intent,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,312,1.0,1,675,1.0,stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent stripe.stripe_create_refund id_map.json github.create_gist trello.trello_create_board github.create_milestone stripe.stripe_create_invoice github.create_hook vercel.vercel_create_project +q104,Acknowledge a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,405,1.0,1,626,1.0,pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls id_map.json +q105,Create a Datadog monitor,verb_product,datadog.datadog_create_monitor,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,459,1.0,1,747,1.0,datadog.datadog_create_monitor datadog.datadog_get_monitor datadog.datadog_list_monitors datadog.datadog_list_logs datadog.datadog_query_metrics id_map.json github.create_gist trello.trello_create_board github.create_milestone stripe.stripe_create_invoice +q106,Resolve a Sentry issue,paraphrase,sentry.sentry_get_issue,0.0,1.0,1.0,0.5000,0.6309,2,1,251,1.0,1,368,1.0,1,754,1.0,sentry.sentry_get_issue sentry.sentry_update_issue sentry.sentry_list_events sentry.sentry_list_projects pagerduty.pagerduty_resolve_incident sentry.sentry_list_issues jira.jira_assign_issue jira.jira_get_issue gitlab.gitlab_get_issue github.issue_read +q107,Scale a Kubernetes deployment,verb_product,kubernetes.k8s_scale_deployment,1.0,1.0,1.0,1.0000,1.0000,1,1,252,1.0,1,371,1.0,1,726,1.0,kubernetes.k8s_scale_deployment kubernetes.k8s_list_deployments kubernetes.k8s_get_pod kubernetes.k8s_apply kubernetes.k8s_list_services vercel.vercel_get_deployment kubernetes.k8s_get_events kubernetes.k8s_delete kubernetes.k8s_list_pods kubernetes.k8s_pod_logs +q108,Apply a Kubernetes manifest,verb_product,kubernetes.k8s_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,334,1.0,1,727,1.0,kubernetes.k8s_apply kubernetes.k8s_list_deployments kubernetes.k8s_get_pod kubernetes.k8s_list_services terraform.tf_apply kubernetes.k8s_delete kubernetes.k8s_get_events kubernetes.k8s_scale_deployment kubernetes.k8s_list_pods kubernetes.k8s_pod_logs +q109,Stop a docker container,verb_product,docker.docker_stop,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,345,1.0,1,587,1.0,docker.docker_stop docker.docker_rm docker.docker_logs docker.docker_exec docker.docker_run docker.docker_images docker.docker_ps docker.docker_build kubernetes.k8s_pod_logs id_map.json +q110,Build a docker image,verb_product,docker.docker_build,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,317,1.0,1,547,1.0,docker.docker_build docker.docker_run docker.docker_stop docker.docker_images docker.docker_rm docker.docker_ps docker.docker_logs docker.docker_exec filesystem.read_media_file id_map.json +q111,Click an element in Playwright,verb_product,playwright.browser_click,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,329,1.0,1,573,1.0,playwright.browser_click puppeteer.puppeteer_click playwright.browser_type playwright.browser_take_screenshot playwright.browser_snapshot playwright.browser_tab_list playwright.browser_tab_new clickup.clickup_list_tasks playwright.browser_navigate clickup.clickup_get_task +q112,Type text into a browser input,paraphrase,playwright.browser_type,1.0,1.0,1.0,1.0000,0.7602,1,1,217,1.0,1,302,1.0,1,577,1.0,playwright.browser_type playwright.browser_wait_for playwright.browser_navigate playwright.browser_tab_list playwright.browser_tab_close playwright.browser_tab_new playwright.browser_fill_form playwright.browser_evaluate filesystem.read_text_file playwright.browser_snapshot +q113,List GitLab projects,verb_product,gitlab.gitlab_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,209,1.0,1,328,1.0,1,587,1.0,gitlab.gitlab_list_projects github.projects_list gitlab.gitlab_list_issues git.git_branch gitlab.gitlab_list_merge_requests git.git_checkout git.git_show vercel.vercel_list_projects git.git_diff jira.jira_list_projects +q114,Create a Bitbucket pull request,verb_product,bitbucket.bitbucket_create_pr,1.0,1.0,1.0,1.0000,1.0000,1,1,401,1.0,1,582,1.0,1,1091,1.0,bitbucket.bitbucket_create_pr github.create_pull_request github.merge_pull_request bitbucket.bitbucket_create_issue github.pull_request_read bitbucket.bitbucket_list_prs github.pull_request_review_write github.update_pull_request_branch github.update_pull_request bitbucket.bitbucket_list_repos +q115,Create a Gmail draft,verb_product,gmail.gmail_create_draft,1.0,1.0,1.0,1.0000,1.0000,1,1,205,1.0,1,448,1.0,1,730,1.0,gmail.gmail_create_draft gmail.gmail_search_emails gmail.gmail_list_labels github.create_pull_request gmail.gmail_send_email gmail.gmail_trash_email gmail.gmail_get_email gmail.gmail_modify_email id_map.json github.create_gist +q116,Share a Google Drive file,verb_product,google_drive.gdrive_share_file,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,417,1.0,1,728,1.0,google_drive.gdrive_share_file google_drive.gdrive_get_file google_drive.gdrive_update_file google_drive.gdrive_create_file google_drive.gdrive_search google_drive.gdrive_list_folder google_calendar.gcal_list_calendars gitlab.gitlab_get_file google_calendar.gcal_delete_event figma.figma_get_file +q117,List Google Calendar events,verb_product,google_calendar.gcal_list_events,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,379,1.0,1,753,1.0,google_calendar.gcal_list_events google_calendar.gcal_delete_event google_calendar.gcal_list_calendars google_calendar.gcal_update_event sentry.sentry_list_events google_calendar.gcal_create_event kubernetes.k8s_get_events google_drive.gdrive_list_folder google_drive.gdrive_search google_drive.gdrive_update_file +q118,Delete a calendar event,resource,google_calendar.gcal_delete_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,427,1.0,1,731,1.0,google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars redis.redis_delete kubernetes.k8s_delete linear.delete_attachment jira.delete_attachment aws_s3.s3_delete_object +q119,List Vercel projects,verb_product,vercel.vercel_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,151,1.0,1,262,1.0,1,550,1.0,vercel.vercel_list_projects vercel.vercel_list_deployments jira.jira_list_projects vercel.vercel_get_deployment sentry.sentry_list_projects github.projects_list linear.linear_list_projects vercel.vercel_create_project gitlab.gitlab_list_projects github.projects_get +q120,Select rows from Supabase,verb_product,supabase.supabase_select,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,354,1.0,1,697,1.0,supabase.supabase_select supabase.supabase_insert supabase.supabase_update supabase.supabase_list_tables supabase.supabase_invoke_fn elasticsearch.es_get_doc memory.delete_observations aws_s3.s3_get_object datadog.datadog_list_logs memory.delete_relations +q121,Post a Figma comment,verb_product,figma.figma_post_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,368,1.0,1,688,1.0,figma.figma_post_comment figma.figma_get_file figma.figma_get_comments figma.figma_get_images figma.figma_get_file_nodes linear.delete_comment linear.get_comment slack.slack_post_message linear.create_comment notion.get_comment +q122,List Okta users,verb_product,okta.okta_list_users,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,299,1.0,1,606,1.0,okta.okta_list_users linear.list_users notion.notion_list_users slack.slack_list_users okta.okta_create_user okta.okta_get_user okta.okta_deactivate_user notion.list_users_tokens slack.slack_get_users github.search_users +q123,Range query Prometheus,verb_product,prometheus.prom_query_range,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,329,1.0,1,680,1.0,prometheus.prom_query_range prometheus.prom_query prometheus.prom_list_metrics postgres.query postgres.explain_query datadog.datadog_query_metrics salesforce.salesforce_query mysql.mysql_query sqlite.sqlite_query notion.notion_query_database +q124,Create a Grafana annotation,verb_product,grafana.grafana_create_annotation,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,296,1.0,1,659,1.0,grafana.grafana_create_annotation grafana.grafana_search_dashboards grafana.grafana_get_dashboard id_map.json github.create_gist trello.trello_create_board github.create_milestone stripe.stripe_create_invoice github.create_hook vercel.vercel_create_project +q125,Apply terraform,identifier,terraform.tf_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,238,1.0,1,595,1.0,terraform.tf_apply terraform.tf_plan kubernetes.k8s_apply terraform.tf_state_list id_map.json +q126,List Asana tasks,verb_product,asana.asana_list_tasks,1.0,1.0,1.0,1.0000,1.0000,1,1,192,1.0,1,368,1.0,1,627,1.0,asana.asana_list_tasks asana.asana_get_task clickup.clickup_list_tasks asana.asana_create_task asana.asana_create_project asana.asana_update_task id_map.json trello.trello_list_cards vercel.vercel_list_projects kubernetes.k8s_list_deployments +q127,Comment on a Trello card,verb_product,trello.trello_add_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,390,1.0,1,745,1.0,trello.trello_add_comment trello.trello_create_card trello.trello_list_cards figma.figma_post_comment trello.trello_create_board github.update_issue_comment linear.delete_comment linear.get_comment linear.create_comment notion.get_comment +q128,Update a ClickUp task,verb_product,clickup.clickup_update_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,399,1.0,1,673,1.0,clickup.clickup_update_task clickup.clickup_get_task clickup.clickup_create_task clickup.clickup_list_tasks asana.asana_update_task puppeteer.puppeteer_click asana.asana_get_task playwright.browser_click asana.asana_create_task id_map.json +q129,Search Confluence,verb_product,confluence.confluence_search,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,362,1.0,1,695,1.0,confluence.confluence_search confluence.confluence_get_page confluence.confluence_create_page confluence.confluence_update_page github.search_users github.search_orgs grafana.grafana_search_dashboards brave_search.brave_web_search brave_search.brave_local_search hubspot.hubspot_search_contacts +q130,Create a HubSpot deal,verb_product,hubspot.hubspot_create_deal,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,899,1.0,hubspot.hubspot_create_deal hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_search_contacts hubspot.hubspot_list_contacts github.create_gist github.create_repository github.create_branch github.create_issue github.create_or_update_file +q131,List Zendesk tickets,verb_product,zendesk.zendesk_list_tickets,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,367,1.0,1,590,1.0,zendesk.zendesk_list_tickets zendesk.zendesk_search zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_create_ticket clickup.clickup_list_tasks id_map.json trello.trello_list_cards vercel.vercel_list_projects kubernetes.k8s_list_deployments +q132,List Discord channels,verb_product,discord.discord_list_channels,1.0,1.0,1.0,1.0000,1.0000,1,1,166,1.0,1,300,1.0,1,555,1.0,discord.discord_list_channels discord.discord_list_members discord.discord_get_channel slack.slack_list_channels discord.discord_send_message discord.discord_add_reaction clickup.clickup_list_tasks id_map.json trello.trello_list_cards vercel.vercel_list_projects +q133,Set a Redis key,verb_product,redis.redis_set,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,302,1.0,1,530,1.0,redis.redis_set redis.redis_get redis.redis_delete redis.redis_keys slack.slack_set_channel_topic playwright.browser_press_key jira.jira_get_issue figma.figma_get_file id_map.json aws_s3.s3_get_object +q134,Insert MongoDB documents,verb_product,mongodb.mongo_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,256,1.0,1,381,1.0,1,607,1.0,mongodb.mongo_insert mongodb.mongo_delete mongodb.mongo_update mongodb.mongo_find mongodb.mongo_list_collections supabase.supabase_insert id_map.json +q135,Run a MySQL query,verb_product,postgres.query,0.0,1.0,1.0,0.5000,0.6309,2,1,182,1.0,1,281,1.0,1,545,1.0,postgres.query mysql.mysql_query postgres.explain_query mysql.mysql_list_tables sqlite.sqlite_query salesforce.salesforce_query postgres.list_schemas postgres.list_tables postgres.describe_table github.delete_check_run +q136,Index a document in Elasticsearch,verb_product,elasticsearch.es_index_doc,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,291,1.0,1,623,1.0,elasticsearch.es_index_doc elasticsearch.es_get_doc elasticsearch.es_search playwright.browser_tab_close id_map.json github.create_milestone supabase.supabase_list_tables github.get_milestone jira.get_attachment github.list_milestones +q137,Get an S3 object,verb_product,aws_s3.s3_get_object,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,375,1.0,1,635,1.0,aws_s3.s3_get_object aws_s3.s3_delete_object aws_s3.s3_put_object aws_s3.s3_list_objects okta.okta_get_user asana.asana_get_task aws_s3.s3_list_buckets elasticsearch.es_search discord.discord_get_channel okta.okta_create_user +q138,List Cloudflare zones,verb_product,cloudflare.cf_list_zones,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,338,1.0,1,542,1.0,cloudflare.cf_list_zones cloudflare.cf_list_dns cloudflare.cf_purge_cache cloudflare.cf_create_dns clickup.clickup_list_tasks id_map.json trello.trello_list_cards vercel.vercel_list_projects kubernetes.k8s_list_deployments jira.jira_list_projects +q139,Create a Stripe invoice,verb_product,stripe.stripe_create_invoice,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,353,1.0,1,647,1.0,stripe.stripe_create_invoice stripe.stripe_create_product stripe.stripe_create_customer stripe.stripe_create_refund stripe.stripe_create_price stripe.stripe_get_customer stripe.stripe_list_invoices stripe.stripe_cancel_subscription stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent +q140,List GitHub Actions workflows,verb_product,github.actions_list,1.0,1.0,1.0,1.0000,1.0000,1,1,268,1.0,1,453,1.0,1,728,1.0,github.actions_list github.actions_get github.actions_get_job_logs github.actions_run_trigger github.projects_list git.git_branch github.list_gists git.git_checkout github.list_starred_repositories git.git_show +q141,Get GitHub Actions job logs,verb_product,github.actions_get_job_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,442,1.0,1,746,1.0,github.actions_get_job_logs github.actions_get github.actions_list github.actions_run_trigger gitlab.get_job git.git_log github.projects_get git.git_checkout github.get_gist git.git_show +q142,Create a GitHub repository,verb_product,github.create_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,235,1.0,1,388,1.0,1,927,1.0,github.create_repository github.delete_repository github.star_repository github.fork_repository github.unstar_repository github.list_repository_collaborators github.create_branch github.create_issue github.create_or_update_file git.git_create_branch +q143,Checkout a git branch,verb_product,git.git_checkout,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,373,1.0,1,835,1.0,git.git_checkout git.git_create_branch git.git_branch github.create_branch git.git_show gitlab.gitlab_create_merge_request github.list_commits git.git_diff github.delete_file git.git_commit +q144,Show git log,identifier,git.git_log,1.0,1.0,1.0,1.0000,1.0000,1,1,206,1.0,1,312,1.0,1,600,1.0,git.git_log git.git_show git.git_diff git.git_status git.git_diff_unstaged git.git_checkout git.git_branch git.git_diff_staged git.git_commit git.git_add +q145,List postgres schemas,verb_product,postgres.list_schemas,1.0,1.0,1.0,1.0000,1.0000,1,1,158,1.0,1,277,1.0,1,500,1.0,postgres.list_schemas postgres.list_tables postgres.query postgres.explain_query postgres.describe_table clickup.clickup_list_tasks id_map.json trello.trello_list_cards vercel.vercel_list_projects kubernetes.k8s_list_deployments +q146,Open nodes in the knowledge graph,verb_product,memory.open_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,163,1.0,1,262,1.0,1,549,1.0,memory.open_nodes memory.search_nodes memory.read_graph memory.create_entities memory.create_relations memory.delete_relations memory.add_observations memory.delete_entities figma.figma_get_file_nodes memory.delete_observations +q147,Get file metadata from the filesystem,paraphrase,filesystem.get_file_info,1.0,1.0,1.0,1.0000,1.0000,1,1,192,1.0,1,334,1.0,1,770,1.0,filesystem.get_file_info filesystem.read_file filesystem.read_media_file filesystem.read_text_file google_drive.gdrive_get_file gitlab.gitlab_get_file figma.figma_get_file_nodes github.get_file_contents filesystem.write_file filesystem.edit_file +q148,Write a new local file,paraphrase,filesystem.write_file,1.0,1.0,1.0,1.0000,1.0000,1,1,198,1.0,1,366,1.0,1,784,1.0,filesystem.write_file filesystem.read_file playwright.browser_tab_new brave_search.brave_local_search filesystem.edit_file filesystem.read_media_file filesystem.read_text_file github.issue_write figma.figma_get_file github.projects_write +q149,Search GitHub issues,verb_product,github.search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,263,1.0,1,397,1.0,1,670,1.0,github.search_issues github.list_issues linear.linear_search_issues jira.jira_search_issues git.git_checkout git.git_show github.search_users git.git_branch github.search_orgs git.git_diff +q150,Get the authenticated GitHub user,paraphrase,github.get_teams,0.0,1.0,1.0,0.3333,0.5000,3,1,178,1.0,1,295,1.0,1,592,1.0,github.get_teams git.git_commit github.get_me git.git_status github.list_notifications git.git_add slack.slack_get_user_profile git.git_checkout okta.okta_get_user git.git_show diff --git a/experiments/mcp-tool-search/results/hybrid_codedb_weighted_bm25.json b/experiments/mcp-tool-search/results/hybrid_codedb_weighted_bm25.json new file mode 100644 index 00000000..9bba7027 --- /dev/null +++ b/experiments/mcp-tool-search/results/hybrid_codedb_weighted_bm25.json @@ -0,0 +1,6981 @@ +{ + "summary": { + "n": 150, + "index_s": 0.4446164010000757, + "rss_mb": null, + "p50_ms": 0.006783499998164189, + "p95_ms": 0.015479500012816057, + "recall@1": 0.9333333333333333, + "recall@1_lo": 0.8933333333333333, + "recall@1_hi": 0.9733333333333334, + "recall@3": 0.9866666666666667, + "recall@3_lo": 0.9666666666666667, + "recall@3_hi": 1.0, + "recall@5": 0.9933333333333333, + "recall@5_lo": 0.98, + "recall@5_hi": 1.0, + "mrr@10": 0.9602222222222222, + "mrr@10_lo": 0.9346666666666666, + "mrr@10_hi": 0.9833333333333333, + "ndcg@10": 0.9646616732747915, + "ndcg@10_lo": 0.945318156083469, + "ndcg@10_hi": 0.9824338047871932 + }, + "meta": { + "name": "hybrid_codedb_weighted_bm25", + "needs_model": false, + "needs_api": false, + "notes": "RRF codedb_search + weighted BM25" + }, + "rows": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "github.issue_write", + "github.issue_read", + "github.add_issue_comment", + "github.update_issue_comment", + "git.git_create_branch", + "gitlab.gitlab_create_issue", + "bitbucket.bitbucket_create_issue", + "linear.linear_create_issue", + "jira.jira_create_issue" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 388, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 598, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1111, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "kind": "verb_product", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_create_issue", + "jira.jira_create_sprint", + "zendesk.zendesk_create_ticket", + "jira.jira_assign_issue", + "zendesk.zendesk_get_ticket", + "jira.jira_list_projects", + "zendesk.zendesk_update_ticket", + "jira.jira_update_issue", + "jira.create_attachment", + "jira.jira_get_issue" + ], + "hit1": "jira.jira_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 795, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "kind": "verb_product", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "ranking": [ + "slack.slack_post_message", + "slack.slack_add_reaction", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_create_channel", + "slack.slack_search_messages", + "discord.discord_send_message", + "twilio.twilio_send_sms", + "gmail.gmail_send_email", + "slack.slack_upload_file" + ], + "hit1": "slack.slack_post_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "kind": "resource", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "ranking": [ + "github.list_pull_requests", + "github.search_pull_requests", + "gitlab.gitlab_list_merge_requests", + "github.create_pull_request", + "github.merge_pull_request", + "bitbucket.bitbucket_list_prs", + "github.pull_request_read", + "github.update_pull_request_branch", + "playwright.browser_tab_list", + "memory.open_nodes" + ], + "hit1": "github.list_pull_requests", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 489, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 836, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "ranking": [ + "github.create_pull_request", + "github.merge_pull_request", + "github.pull_request_read", + "github.pull_request_review_write", + "github.update_pull_request", + "github.update_pull_request_branch", + "github.request_copilot_review", + "github.search_pull_requests", + "github.list_pull_requests", + "github.add_comment_to_pending_review" + ], + "hit1": "github.create_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 368, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 653, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1144, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "kind": "verb_product", + "gold": [ + "postgres.query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "postgres.explain_query", + "mysql.mysql_query", + "postgres.list_schemas", + "sqlite.sqlite_query", + "postgres.list_tables", + "salesforce.salesforce_query", + "postgres.describe_table", + "github.delete_check_run", + "prometheus.prom_query" + ], + "hit1": "postgres.query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 272, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 551, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "kind": "paraphrase", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "ranking": [ + "filesystem.read_file", + "filesystem.read_media_file", + "filesystem.read_text_file", + "gitlab.gitlab_get_file", + "github.delete_file", + "brave_search.brave_local_search", + "figma.figma_get_file_nodes", + "filesystem.write_file", + "filesystem.edit_file", + "filesystem.get_file_info" + ], + "hit1": "filesystem.read_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "kind": "paraphrase", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "ranking": [ + "google_drive.gdrive_list_folder", + "filesystem.search_files", + "slack.list_files", + "github.push_files", + "filesystem.move_file", + "filesystem.list_directory", + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree", + "cloudflare.cf_purge_cache", + "filesystem.read_file" + ], + "hit1": "google_drive.gdrive_list_folder", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.16666666666666666, + "ndcg@10": 0.43476305489842704, + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 473, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 714, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 692, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "ranking": [ + "memory.read_graph", + "memory.create_entities", + "memory.create_relations", + "memory.add_observations", + "memory.delete_relations", + "memory.open_nodes", + "memory.search_nodes", + "memory.delete_entities", + "postgres.list_schemas", + "memory.delete_observations" + ], + "hit1": "memory.read_graph", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.5672074169568709, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 156, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 261, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 511, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "kind": "verb_product", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "ranking": [ + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "jira.jira_search_issues", + "filesystem.list_directory_with_sizes", + "memory.search_nodes", + "github.update_pull_request_branch", + "grafana.grafana_search_dashboards", + "puppeteer.puppeteer_screenshot", + "github.search_users", + "puppeteer.puppeteer_navigate" + ], + "hit1": "brave_search.brave_web_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 374, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 712, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "kind": "verb_product", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "ranking": [ + "fetch.fetch", + "github.get_file_contents", + "filesystem.read_file", + "playwright.browser_navigate", + "git.git_add", + "slack.slack_set_channel_topic", + "playwright.browser_snapshot", + "git.git_status", + "puppeteer.puppeteer_navigate", + "github.get_repository_tree" + ], + "hit1": "fetch.fetch", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 264, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "kind": "identifier", + "gold": [ + "git.git_status" + ], + "alts": [], + "ranking": [ + "git.git_status", + "git.git_show", + "git.git_diff", + "git.git_diff_unstaged", + "git.git_log", + "git.git_diff_staged", + "git.git_checkout", + "git.git_branch", + "git.git_commit", + "gitlab.list_releases" + ], + "hit1": "git.git_status", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 615, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "kind": "paraphrase", + "gold": [ + "git.git_commit" + ], + "alts": [], + "ranking": [ + "git.git_commit", + "git.git_reset", + "github.get_commit", + "git.git_diff_staged", + "git.git_show", + "git.git_log", + "github.push_files", + "id_map.json" + ], + "hit1": "git.git_commit", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 596, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_issue", + "linear.linear_assign_issue", + "linear.linear_create_comment", + "linear.linear_update_issue", + "linear.linear_get_issue", + "linear.linear_create_project", + "bitbucket.bitbucket_create_issue", + "gitlab.gitlab_create_issue", + "linear.linear_list_teams", + "jira.jira_create_issue" + ], + "hit1": "linear.linear_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 876, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "kind": "paraphrase", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_project", + "twilio.twilio_make_call", + "linear.linear_create_issue", + "clickup.clickup_get_task", + "linear.linear_update_issue", + "asana.asana_get_task", + "linear.linear_list_teams", + "clickup.clickup_update_task", + "linear.linear_list_projects", + "asana.asana_update_task" + ], + "hit1": "linear.linear_create_project", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 326, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 494, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 806, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "ranking": [ + "github.update_issue_comment", + "github.add_issue_comment", + "github.issue_read", + "github.assign_copilot_to_issue", + "github.create_issue", + "github.issue_write", + "figma.figma_post_comment", + "github.add_comment_to_pending_review", + "github.list_repository_collaborators", + "git.git_checkout" + ], + "hit1": "github.update_issue_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 569, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1072, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "kind": "ambiguous", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "ranking": [ + "github.merge_pull_request", + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_get_merge_request", + "github.update_pull_request_branch", + "github.pull_request_read", + "gitlab.gitlab_create_merge_request", + "github.create_pull_request", + "github.update_pull_request", + "github.pull_request_review_write", + "github.request_copilot_review" + ], + "hit1": "github.merge_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 468, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1121, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "kind": "verb_product", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "ranking": [ + "github.actions_run_trigger", + "github.actions_get", + "github.actions_list", + "github.actions_get_job_logs", + "circleci.circleci_trigger_pipeline", + "git.git_checkout", + "github.search_users", + "git.git_show", + "github.search_orgs", + "git.git_branch" + ], + "hit1": "github.actions_run_trigger", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "kind": "resource", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "ranking": [ + "github.list_dependabot_alerts", + "github.get_dependabot_alert", + "github.list_secret_scanning_alerts", + "github.list_code_scanning_alerts", + "redis.redis_get", + "id_map.json", + "kubernetes.k8s_get_pod", + "discord.discord_get_channel", + "vercel.vercel_get_deployment", + "github.get_discussion" + ], + "hit1": "github.list_dependabot_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "kind": "verb_product", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_issues", + "sentry.sentry_list_projects", + "sentry.sentry_list_events", + "sentry.sentry_get_issue", + "gitlab.gitlab_list_issues", + "github.list_issues", + "sentry.sentry_update_issue", + "linear.linear_list_issues", + "github.search_issues", + "linear.linear_search_issues" + ], + "hit1": "sentry.sentry_list_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_create_invoice", + "stripe.stripe_create_product", + "stripe.stripe_create_payment_intent", + "stripe.stripe_list_invoices", + "stripe.stripe_list_subscriptions", + "stripe.stripe_create_refund", + "stripe.stripe_create_price", + "stripe.stripe_get_payment_intent" + ], + "hit1": "stripe.stripe_create_customer", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "kind": "paraphrase", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_refund", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_customer", + "stripe.stripe_create_invoice", + "stripe.stripe_list_customers", + "stripe.stripe_list_invoices", + "stripe.stripe_create_product", + "stripe.stripe_create_customer", + "stripe.stripe_cancel_subscription" + ], + "hit1": "stripe.stripe_create_refund", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 332, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "kind": "verb_product", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "ranking": [ + "gmail.gmail_send_email", + "gmail.gmail_trash_email", + "gmail.gmail_get_email", + "gmail.gmail_modify_email", + "gmail.gmail_search_emails", + "twilio.twilio_send_sms", + "gmail.gmail_list_labels", + "gmail.gmail_create_draft", + "filesystem.list_directory_with_sizes", + "discord.discord_send_message" + ], + "hit1": "gmail.gmail_send_email", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 716, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "kind": "verb_product", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "ranking": [ + "gmail.gmail_search_emails", + "gmail.gmail_list_labels", + "gmail.gmail_create_draft", + "gmail.gmail_trash_email", + "gmail.gmail_get_email", + "gmail.gmail_send_email", + "gmail.gmail_modify_email", + "grafana.grafana_search_dashboards", + "brave_search.brave_web_search", + "github.search_users" + ], + "hit1": "gmail.gmail_search_emails", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 700, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "kind": "paraphrase", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_update_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_share_file", + "slack.slack_upload_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "filesystem.edit_file", + "gitlab.gitlab_get_file", + "aws_s3.s3_put_object" + ], + "hit1": "google_drive.gdrive_update_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.2, + "ndcg@10": 0.38685280723454163, + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 523, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 446, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 827, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_create_event", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "github.pull_request_review_write", + "github.create_gist", + "id_map.json", + "github.create_milestone", + "trello.trello_create_board" + ], + "hit1": "google_calendar.gcal_create_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 810, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_list_pods", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_apply", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_delete", + "kubernetes.k8s_pod_logs", + "id_map.json" + ], + "hit1": "kubernetes.k8s_list_pods", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_get_pod", + "github.actions_get_job_logs", + "docker.docker_logs", + "datadog.datadog_list_logs", + "elasticsearch.es_get_doc", + "aws_s3.s3_get_object", + "gitlab.gitlab_get_file", + "git.git_log", + "slack.slack_get_channel_history" + ], + "hit1": "kubernetes.k8s_pod_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 822, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "ranking": [ + "docker.docker_run", + "docker.docker_stop", + "docker.docker_exec", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_images", + "docker.docker_ps", + "docker.docker_build", + "github.get_check_run", + "github.delete_check_run" + ], + "hit1": "docker.docker_run", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "kind": "verb_product", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "ranking": [ + "docker.docker_logs", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_exec", + "docker.docker_run", + "kubernetes.k8s_pod_logs", + "datadog.datadog_list_logs", + "docker.docker_images", + "git.git_log", + "docker.docker_build" + ], + "hit1": "docker.docker_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "kind": "ambiguous", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "ranking": [ + "playwright.browser_navigate", + "puppeteer.puppeteer_navigate", + "playwright.browser_tab_new", + "playwright.browser_evaluate", + "playwright.browser_click", + "playwright.browser_press_key", + "playwright.browser_snapshot", + "playwright.browser_wait_for", + "playwright.browser_tab_close", + "playwright.browser_tab_list" + ], + "hit1": "playwright.browser_navigate", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 536, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "kind": "ambiguous", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "ranking": [ + "playwright.browser_take_screenshot", + "puppeteer.puppeteer_screenshot", + "notion.notion_get_page", + "notion.notion_update_page", + "confluence.confluence_get_page", + "playwright.browser_snapshot", + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "notion.notion_create_page", + "playwright.browser_evaluate" + ], + "hit1": "playwright.browser_take_screenshot", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 711, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_create_merge_request", + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_get_merge_request", + "gitlab.gitlab_list_merge_requests", + "github.merge_pull_request", + "git.git_create_branch", + "gitlab.gitlab_create_pipeline", + "git.git_checkout", + "gitlab.gitlab_create_issue", + "git.git_show" + ], + "hit1": "gitlab.gitlab_create_merge_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 844, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "kind": "verb_product", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "ranking": [ + "notion.notion_search", + "notion.notion_get_page", + "notion.notion_get_block", + "notion.notion_list_users", + "notion.notion_query_database", + "grafana.grafana_search_dashboards", + "notion.notion_create_database", + "github.search_users", + "notion.notion_update_page", + "hubspot.hubspot_search_contacts" + ], + "hit1": "notion.notion_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 318, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 639, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "kind": "verb_product", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "ranking": [ + "notion.notion_get_page", + "notion.notion_create_page", + "notion.notion_update_page", + "notion.notion_get_comments", + "notion.notion_create_database", + "confluence.confluence_create_page", + "notion.notion_search", + "notion.create_comment", + "notion.notion_get_block", + "notion.create_data_source" + ], + "hit1": "notion.notion_get_page", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 784, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "kind": "verb_product", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "ranking": [ + "notion.notion_query_database", + "notion.notion_create_database", + "notion.notion_create_page", + "notion.notion_search", + "notion.notion_get_page", + "notion.notion_get_block", + "notion.notion_list_users", + "notion.notion_update_page", + "notion.notion_get_comments", + "postgres.query" + ], + "hit1": "notion.notion_query_database", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 279, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "jira.jira_create_issue", + "zendesk.zendesk_search", + "zendesk.zendesk_list_tickets", + "id_map.json", + "github.create_gist", + "trello.trello_create_board", + "github.create_milestone" + ], + "hit1": "zendesk.zendesk_create_ticket", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 723, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_create_deal", + "hubspot.hubspot_search_contacts", + "hubspot.hubspot_list_contacts", + "github.create_gist", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.create_or_update_file" + ], + "hit1": "hubspot.hubspot_create_contact", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 899, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "kind": "verb_product", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_create_lead", + "salesforce.salesforce_convert_lead", + "salesforce.salesforce_create_record", + "salesforce.salesforce_query", + "salesforce.salesforce_list_opportunities", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_update_record", + "salesforce.salesforce_get_record", + "id_map.json", + "github.create_gist" + ], + "hit1": "salesforce.salesforce_create_lead", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "kind": "identifier", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_query", + "postgres.query", + "mysql.mysql_query", + "sqlite.sqlite_query", + "docker.docker_run", + "github.delete_check_run", + "prometheus.prom_query", + "github.get_check_run", + "github.update_check_run", + "postgres.explain_query" + ], + "hit1": "salesforce.salesforce_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 187, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 332, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 616, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "kind": "verb_product", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "ranking": [ + "shopify.shopify_list_orders", + "shopify.shopify_list_customers", + "shopify.shopify_list_products", + "shopify.shopify_get_product", + "shopify.shopify_get_order", + "shopify.shopify_update_product", + "shopify.shopify_create_product", + "shopify.shopify_create_order", + "clickup.clickup_list_tasks", + "id_map.json" + ], + "hit1": "shopify.shopify_list_orders", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "id_map.json", + "github.create_gist", + "trello.trello_create_board", + "github.create_milestone" + ], + "hit1": "pagerduty.pagerduty_create_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "kind": "paraphrase", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_list_oncalls", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_create_incident", + "twilio.twilio_make_call", + "filesystem.list_allowed_directories", + "slack.slack_create_channel", + "trello.trello_add_comment" + ], + "hit1": "pagerduty.pagerduty_list_oncalls", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "kind": "verb_product", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "ranking": [ + "datadog.datadog_query_metrics", + "datadog.datadog_list_monitors", + "datadog.datadog_list_logs", + "datadog.datadog_create_monitor", + "prometheus.prom_list_metrics", + "datadog.datadog_get_monitor", + "postgres.query", + "prometheus.prom_query", + "postgres.explain_query", + "prometheus.prom_query_range" + ], + "hit1": "datadog.datadog_query_metrics", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 284, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 449, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 767, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "kind": "verb_product", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "ranking": [ + "datadog.datadog_list_logs", + "datadog.datadog_list_monitors", + "datadog.datadog_get_monitor", + "datadog.datadog_query_metrics", + "datadog.datadog_create_monitor", + "docker.docker_logs", + "kubernetes.k8s_pod_logs", + "github.actions_get_job_logs", + "grafana.grafana_search_dashboards", + "brave_search.brave_web_search" + ], + "hit1": "datadog.datadog_list_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 860, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_list_buckets", + "aws_s3.s3_list_objects", + "clickup.clickup_list_tasks", + "aws_s3.s3_delete_object", + "aws_s3.s3_get_object", + "trello.trello_list_cards", + "aws_s3.s3_put_object", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "id_map.json" + ], + "hit1": "aws_s3.s3_list_buckets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 598, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "kind": "paraphrase", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_put_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_get_object", + "slack.slack_upload_file", + "github.assign_copilot_to_issue", + "twilio.twilio_send_sms", + "slack.slack_add_reaction", + "gmail.gmail_send_email", + "aws_s3.s3_list_objects", + "slack.slack_reply_to_thread" + ], + "hit1": "aws_s3.s3_put_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 868, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "cloudflare.cf_list_dns", + "cloudflare.cf_create_dns", + "id_map.json" + ], + "hit1": "cloudflare.cf_purge_cache", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 286, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 643, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "salesforce.salesforce_create_record", + "cloudflare.cf_list_zones", + "cloudflare.cf_purge_cache", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_update_record", + "salesforce.salesforce_get_record", + "git.git_commit", + "id_map.json" + ], + "hit1": "cloudflare.cf_create_dns", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_deployments", + "vercel.vercel_list_projects", + "kubernetes.k8s_list_deployments", + "vercel.vercel_get_deployment", + "vercel.vercel_create_project", + "clickup.clickup_list_tasks", + "id_map.json", + "trello.trello_list_cards", + "jira.jira_list_projects", + "kubernetes.k8s_list_services" + ], + "hit1": "vercel.vercel_list_deployments", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 173, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 506, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "ranking": [ + "supabase.supabase_insert", + "supabase.supabase_select", + "mongodb.mongo_insert", + "supabase.supabase_list_tables", + "supabase.supabase_invoke_fn", + "supabase.supabase_update", + "playwright.browser_type", + "id_map.json" + ], + "hit1": "supabase.supabase_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 347, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 598, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "kind": "paraphrase", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "ranking": [ + "figma.figma_get_images", + "figma.figma_get_file_nodes", + "figma.figma_get_file", + "docker.docker_images", + "figma.figma_get_comments", + "figma.figma_post_comment", + "elasticsearch.es_get_doc", + "memory.delete_observations", + "aws_s3.s3_get_object", + "datadog.datadog_list_logs" + ], + "hit1": "figma.figma_get_images", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 695, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "kind": "verb_product", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "ranking": [ + "twilio.twilio_send_sms", + "twilio.twilio_list_messages", + "twilio.twilio_make_call", + "gmail.gmail_send_email", + "filesystem.list_directory_with_sizes", + "discord.discord_send_message", + "puppeteer.puppeteer_click", + "puppeteer.puppeteer_fill", + "id_map.json", + "puppeteer.puppeteer_screenshot" + ], + "hit1": "twilio.twilio_send_sms", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 668, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "kind": "verb_product", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "ranking": [ + "okta.okta_create_user", + "okta.okta_get_user", + "okta.okta_deactivate_user", + "linear.create_user", + "okta.okta_list_users", + "linear.get_user", + "asana.asana_create_task", + "slack.slack_get_user_profile", + "asana.asana_create_project", + "linear.update_user" + ], + "hit1": "okta.okta_create_user", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 676, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query", + "prometheus.prom_query_range", + "prometheus.prom_list_metrics", + "postgres.query", + "postgres.explain_query", + "datadog.datadog_query_metrics", + "salesforce.salesforce_query", + "mysql.mysql_query", + "sqlite.sqlite_query", + "notion.notion_query_database" + ], + "hit1": "prometheus.prom_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "kind": "verb_product", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "ranking": [ + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "grafana.grafana_create_annotation", + "github.search_users", + "confluence.confluence_search", + "github.search_orgs", + "brave_search.brave_web_search", + "hubspot.hubspot_search_contacts", + "brave_search.brave_local_search", + "github.search_issues" + ], + "hit1": "grafana.grafana_search_dashboards", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 681, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "kind": "verb_product", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "ranking": [ + "circleci.circleci_trigger_pipeline", + "circleci.circleci_get_pipeline", + "circleci.circleci_list_pipelines", + "gitlab.gitlab_create_pipeline", + "github.actions_run_trigger", + "gitlab.get_pipeline", + "gitlab.create_pipeline", + "gitlab.delete_pipeline", + "gitlab.update_pipeline", + "id_map.json" + ], + "hit1": "circleci.circleci_trigger_pipeline", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 664, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "kind": "identifier", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "ranking": [ + "terraform.tf_plan", + "terraform.tf_apply", + "terraform.tf_state_list", + "github.get_check_run", + "docker.docker_run", + "github.delete_check_run", + "postgres.explain_query", + "github.update_check_run", + "github.create_check_run", + "github.actions_run_trigger" + ], + "hit1": "terraform.tf_plan", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "kind": "verb_product", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "ranking": [ + "asana.asana_create_task", + "asana.asana_get_task", + "asana.asana_update_task", + "asana.asana_create_project", + "asana.asana_list_tasks", + "clickup.clickup_create_task", + "clickup.clickup_get_task", + "clickup.clickup_update_task", + "okta.okta_create_user", + "github.create_issue" + ], + "hit1": "asana.asana_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "ranking": [ + "trello.trello_create_card", + "trello.trello_create_board", + "trello.trello_add_comment", + "trello.trello_list_cards", + "id_map.json", + "github.create_gist", + "stripe.stripe_create_invoice", + "github.create_milestone", + "vercel.vercel_create_project", + "github.create_hook" + ], + "hit1": "trello.trello_create_card", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 700, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_create_task", + "clickup.clickup_get_task", + "clickup.clickup_update_task", + "clickup.clickup_list_tasks", + "asana.asana_create_task", + "puppeteer.puppeteer_click", + "asana.asana_get_task", + "playwright.browser_click", + "asana.asana_update_task", + "id_map.json" + ], + "hit1": "clickup.clickup_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 415, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "kind": "verb_product", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "ranking": [ + "confluence.confluence_create_page", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "confluence.confluence_search", + "notion.notion_create_page", + "notion.notion_get_page", + "notion.notion_update_page", + "playwright.browser_take_screenshot", + "id_map.json", + "notion.notion_get_comments" + ], + "hit1": "confluence.confluence_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 652, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "kind": "verb_product", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "discord.discord_add_reaction", + "discord.discord_get_channel", + "discord.discord_list_channels", + "twilio.twilio_send_sms", + "gmail.gmail_send_email", + "discord.discord_list_members", + "slack.slack_post_message", + "gmail.gmail_trash_email", + "id_map.json" + ], + "hit1": "discord.discord_send_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 628, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_search", + "elasticsearch.es_get_doc", + "elasticsearch.es_index_doc", + "github.search_users", + "confluence.confluence_search", + "github.search_orgs", + "github.search_commits", + "grafana.grafana_search_dashboards", + "brave_search.brave_web_search", + "brave_search.brave_local_search" + ], + "hit1": "elasticsearch.es_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 672, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "redis.redis_set", + "redis.redis_delete", + "jira.jira_get_issue", + "redis.redis_keys", + "figma.figma_get_file", + "aws_s3.s3_get_object", + "figma.figma_get_comments", + "figma.figma_get_file_nodes", + "playwright.browser_press_key" + ], + "hit1": "redis.redis_get", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 589, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_find", + "mongodb.mongo_insert", + "mongodb.mongo_delete", + "mongodb.mongo_update", + "mongodb.mongo_list_collections", + "id_map.json" + ], + "hit1": "mongodb.mongo_find", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 631, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "kind": "verb_product", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "ranking": [ + "sqlite.sqlite_list_tables", + "mysql.mysql_list_tables", + "postgres.list_tables", + "supabase.supabase_list_tables", + "sqlite.sqlite_describe_table", + "sqlite.sqlite_query", + "clickup.clickup_list_tasks", + "id_map.json", + "postgres.list_schemas", + "vercel.vercel_list_projects" + ], + "hit1": "sqlite.sqlite_list_tables", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 240, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 420, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "kind": "verb_product", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "ranking": [ + "postgres.describe_table", + "sqlite.sqlite_describe_table", + "postgres.explain_query", + "postgres.query", + "postgres.list_schemas", + "postgres.list_tables", + "supabase.supabase_insert", + "supabase.supabase_select", + "supabase.supabase_update", + "id_map.json" + ], + "hit1": "postgres.describe_table", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 278, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 543, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "github.issue_read", + "github.issue_write", + "github.add_issue_comment", + "github.update_issue_comment", + "github.assign_copilot_to_issue", + "bitbucket.bitbucket_create_issue", + "sentry.sentry_get_issue", + "gitlab.gitlab_create_issue", + "jira.jira_assign_issue" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 386, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 596, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1019, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "ranking": [ + "github.create_issue", + "github.issue_read", + "bitbucket.bitbucket_create_issue", + "github.issue_write", + "sentry.sentry_get_issue", + "gitlab.gitlab_create_issue", + "jira.jira_assign_issue", + "jira.jira_get_issue", + "linear.linear_create_issue", + "gitlab.gitlab_get_issue" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7349350438281765, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 340, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 546, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 961, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "kind": "identifier", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "ranking": [ + "filesystem.create_directory", + "filesystem.list_directory", + "filesystem.directory_tree", + "filesystem.list_directory_with_sizes", + "filesystem.get_file_info", + "filesystem.list_allowed_directories", + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "filesystem.write_file" + ], + "hit1": "filesystem.create_directory", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 607, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "kind": "paraphrase", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "ranking": [ + "sequential_thinking.sequentialthinking", + "id_map.json" + ], + "hit1": "sequential_thinking.sequentialthinking", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "kind": "paraphrase", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "ranking": [ + "time.get_current_time", + "time.convert_time", + "slack.slack_create_channel", + "playwright.browser_wait_for", + "filesystem.list_allowed_directories", + "grafana.grafana_create_annotation", + "google_calendar.gcal_list_events", + "jira.jira_add_worklog", + "id_map.json", + "elasticsearch.es_index_doc" + ], + "hit1": "time.get_current_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 222, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 678, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "kind": "verb_product", + "gold": [ + "time.convert_time" + ], + "alts": [], + "ranking": [ + "time.convert_time", + "salesforce.salesforce_convert_lead", + "time.get_current_time", + "memory.create_relations", + "playwright.browser_wait_for", + "git.git_diff", + "grafana.grafana_create_annotation", + "google_calendar.gcal_list_events", + "jira.jira_add_worklog", + "id_map.json" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 650, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "kind": "verb_product", + "gold": [ + "github.star_repository" + ], + "alts": [], + "ranking": [ + "github.star_repository", + "github.unstar_repository", + "git.git_branch", + "github.delete_repository", + "git.git_show", + "github.actions_list", + "git.git_checkout", + "github.get_label", + "git.git_status", + "github.get_discussion" + ], + "hit1": "github.star_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "kind": "resource", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "ranking": [ + "github.fork_repository", + "github.star_repository", + "github.delete_repository", + "github.unstar_repository", + "github.list_repository_collaborators", + "github.create_repository", + "github.get_repository_tree", + "github.list_releases", + "github.list_dependabot_alerts", + "github.list_labels" + ], + "hit1": "github.fork_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 789, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "kind": "verb_product", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "ranking": [ + "github.get_latest_release", + "github.get_release_by_tag", + "gitlab.get_release", + "git.git_commit", + "github.get_teams", + "git.git_status", + "github.get_me", + "git.git_add", + "github.get_repository_tree", + "git.git_checkout" + ], + "hit1": "github.get_latest_release", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "kind": "resource", + "gold": [ + "github.create_gist" + ], + "alts": [], + "ranking": [ + "github.create_gist", + "github.get_gist", + "github.update_gist", + "trello.trello_create_board", + "github.create_milestone", + "stripe.stripe_create_invoice", + "github.create_hook", + "vercel.vercel_create_project", + "github.create_environment", + "okta.okta_create_user" + ], + "hit1": "github.create_gist", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 739, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "kind": "verb_product", + "gold": [ + "github.search_code" + ], + "alts": [], + "ranking": [ + "github.search_code", + "github.list_code_scanning_alerts", + "github.get_code_scanning_alert", + "github.search_users", + "git.git_checkout", + "github.search_orgs", + "git.git_show", + "github.search_commits", + "git.git_branch", + "github.search_pull_requests" + ], + "hit1": "github.search_code", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 395, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "kind": "verb_product", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "ranking": [ + "github.list_notifications", + "github.mark_all_notifications_read", + "github.projects_list", + "git.git_branch", + "github.list_gists", + "git.git_checkout", + "github.actions_list", + "git.git_show", + "github.list_starred_repositories", + "git.git_diff" + ], + "hit1": "github.list_notifications", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 286, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 595, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "kind": "verb_product", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "ranking": [ + "github.assign_copilot_to_issue", + "linear.linear_assign_issue", + "jira.jira_assign_issue", + "jira.jira_update_issue", + "jira.jira_transition_issue", + "github.request_copilot_review", + "github.add_issue_comment", + "github.create_issue", + "linear.linear_create_comment", + "slack.slack_reply_to_thread" + ], + "hit1": "github.assign_copilot_to_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 896, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "kind": "resource", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "ranking": [ + "github.list_secret_scanning_alerts", + "github.get_secret_scanning_alert", + "github.list_code_scanning_alerts", + "github.get_code_scanning_alert", + "github.list_dependabot_alerts", + "redis.redis_get", + "id_map.json", + "kubernetes.k8s_get_pod", + "discord.discord_get_channel", + "vercel.vercel_get_deployment" + ], + "hit1": "github.list_secret_scanning_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "ranking": [ + "github.create_or_update_file", + "github.get_file_contents", + "google_drive.gdrive_update_file", + "github.delete_file", + "slack.update_file", + "git.git_diff", + "slack.create_file", + "git.git_add", + "google_drive.gdrive_create_file", + "github.projects_write" + ], + "hit1": "github.create_or_update_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 367, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 548, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 966, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "kind": "resource", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "ranking": [ + "github.list_repository_collaborators", + "github.list_commits", + "pagerduty.pagerduty_list_oncalls", + "trello.trello_list_cards", + "bitbucket.bitbucket_list_prs", + "github.get_discussion_comments", + "github.actions_list", + "github.update_issue_comment", + "github.list_releases", + "trello.trello_add_comment" + ], + "hit1": "github.list_repository_collaborators", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 264, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 824, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "kind": "verb_product", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "ranking": [ + "slack.slack_reply_to_thread", + "slack.slack_get_thread_replies", + "slack.slack_list_channels", + "slack.delete_file", + "slack.slack_join_channel", + "slack.get_canvase", + "slack.slack_leave_channel", + "slack.delete_canvase", + "slack.slack_create_channel", + "slack.delete_usergroup" + ], + "hit1": "slack.slack_reply_to_thread", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 359, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 655, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "kind": "verb_product", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "ranking": [ + "slack.slack_add_reaction", + "discord.discord_add_reaction", + "slack.slack_join_channel", + "trello.trello_add_comment", + "slack.slack_leave_channel", + "gmail.gmail_modify_email", + "slack.slack_create_channel", + "jira.jira_add_comment", + "slack.slack_search_messages", + "jira.jira_add_worklog" + ], + "hit1": "slack.slack_add_reaction", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 768, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "kind": "verb_product", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "ranking": [ + "slack.slack_search_messages", + "slack.slack_get_channel_history", + "gmail.gmail_search_emails", + "slack.slack_join_channel", + "twilio.twilio_list_messages", + "slack.slack_leave_channel", + "grafana.grafana_search_dashboards", + "slack.slack_create_channel", + "github.search_users", + "slack.slack_upload_file" + ], + "hit1": "slack.slack_search_messages", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 336, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 652, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "kind": "verb_product", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "ranking": [ + "slack.slack_create_channel", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_set_channel_topic", + "slack.slack_get_channel_history", + "slack.slack_post_message", + "slack.slack_upload_file", + "slack.slack_add_reaction", + "slack.slack_get_thread_replies", + "slack.slack_reply_to_thread" + ], + "hit1": "slack.slack_create_channel", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "kind": "verb_product", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_transition_issue", + "jira.jira_assign_issue", + "jira.jira_get_issue", + "jira.jira_update_issue", + "jira.jira_create_issue", + "jira.jira_add_comment", + "jira.jira_add_worklog", + "gitlab.gitlab_get_issue", + "jira.jira_list_projects", + "sentry.sentry_get_issue" + ], + "hit1": "jira.jira_transition_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "kind": "identifier", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "ranking": [ + "jira.jira_search_issues", + "filesystem.list_directory_with_sizes", + "jira.jira_assign_issue", + "grafana.grafana_search_dashboards", + "jira.jira_list_projects", + "github.search_users", + "jira.jira_create_sprint", + "hubspot.hubspot_search_contacts", + "jira.jira_update_issue", + "github.search_orgs" + ], + "hit1": "jira.jira_search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "kind": "verb_product", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "ranking": [ + "jira.jira_add_worklog", + "jira.jira_add_comment", + "jira.jira_transition_issue", + "github.add_comment_to_pending_review", + "discord.discord_add_reaction", + "jira.jira_assign_issue", + "memory.add_observations", + "jira.jira_list_projects", + "git.git_add", + "jira.jira_create_sprint" + ], + "hit1": "jira.jira_add_worklog", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 491, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 823, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "kind": "verb_product", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "ranking": [ + "linear.linear_list_teams", + "linear.linear_list_projects", + "linear.linear_list_issues", + "linear.linear_search_issues", + "github.get_teams", + "linear.linear_update_issue", + "linear.list_comments", + "linear.linear_create_project", + "linear.list_attachments", + "linear.linear_create_issue" + ], + "hit1": "linear.linear_list_teams", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "ranking": [ + "linear.linear_create_comment", + "linear.linear_update_issue", + "linear.linear_assign_issue", + "linear.linear_create_issue", + "linear.linear_get_issue", + "github.update_issue_comment", + "github.add_issue_comment", + "linear.create_comment", + "linear.get_comment", + "jira.jira_add_comment" + ], + "hit1": "linear.linear_create_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 276, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 464, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 888, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "kind": "paraphrase", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "ranking": [ + "filesystem.move_file", + "filesystem.read_file", + "brave_search.brave_local_search", + "filesystem.read_media_file", + "filesystem.read_text_file", + "figma.figma_get_file", + "slack.delete_file", + "filesystem.write_file", + "google_drive.gdrive_get_file", + "slack.create_file" + ], + "hit1": "filesystem.move_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 688, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "kind": "paraphrase", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "ranking": [ + "filesystem.search_files", + "slack.list_files", + "github.push_files", + "redis.redis_keys", + "google_drive.gdrive_search", + "linear.linear_search_issues", + "github.search_repositories", + "github.get_release_by_tag", + "filesystem.move_file", + "filesystem.list_directory" + ], + "hit1": "filesystem.search_files", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 280, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 735, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "kind": "paraphrase", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "ranking": [ + "filesystem.edit_file", + "filesystem.list_directory_with_sizes", + "playwright.browser_tab_new", + "filesystem.write_file", + "slack.get_file", + "filesystem.move_file", + "slack.create_file", + "slack.delete_file", + "slack.update_file", + "figma.figma_get_file" + ], + "hit1": "filesystem.edit_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "ranking": [ + "memory.read_graph", + "memory.delete_relations", + "memory.create_entities", + "memory.create_relations", + "filesystem.read_file", + "memory.add_observations", + "memory.delete_entities", + "memory.open_nodes", + "filesystem.read_media_file", + "filesystem.read_text_file" + ], + "hit1": "memory.read_graph", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 260, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 562, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "kind": "verb_product", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "memory.open_nodes", + "figma.figma_get_file_nodes", + "memory.create_entities", + "memory.delete_observations", + "memory.create_relations", + "memory.read_graph", + "memory.delete_relations", + "memory.add_observations", + "memory.delete_entities" + ], + "hit1": "memory.search_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 304, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 546, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "kind": "verb_product", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "ranking": [ + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_diff", + "git.git_show", + "git.git_status", + "git.git_log", + "git.git_checkout", + "git.git_branch", + "id_map.json", + "git.git_commit" + ], + "hit1": "git.git_diff_unstaged", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 537, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "kind": "identifier", + "gold": [ + "git.git_add" + ], + "alts": [], + "ranking": [ + "git.git_add", + "filesystem.list_directory_with_sizes", + "github.push_files", + "filesystem.search_files", + "github.add_issue_comment", + "github.add_comment_to_pending_review", + "git.git_checkout", + "git.git_show", + "github.create_gist", + "git.git_branch" + ], + "hit1": "git.git_add", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 433, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 834, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "kind": "verb_product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "ranking": [ + "shopify.shopify_create_product", + "shopify.shopify_get_product", + "shopify.shopify_update_product", + "shopify.shopify_create_order", + "stripe.stripe_create_product", + "shopify.shopify_list_products", + "shopify.shopify_list_customers", + "shopify.shopify_get_order", + "stripe.stripe_create_price", + "shopify.shopify_list_orders" + ], + "hit1": "shopify.shopify_create_product", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "kind": "verb_product", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "ranking": [ + "stripe.stripe_cancel_subscription", + "stripe.stripe_get_customer", + "stripe.stripe_create_invoice", + "stripe.stripe_list_customers", + "stripe.stripe_list_invoices", + "stripe.stripe_create_customer", + "stripe.stripe_create_product", + "stripe.stripe_create_refund", + "stripe.stripe_list_subscriptions", + "stripe.stripe_get_payment_intent" + ], + "hit1": "stripe.stripe_cancel_subscription", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 168, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 282, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 610, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "kind": "resource", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_refund", + "id_map.json", + "github.create_gist", + "trello.trello_create_board", + "github.create_milestone", + "stripe.stripe_create_invoice", + "github.create_hook", + "vercel.vercel_create_project" + ], + "hit1": "stripe.stripe_create_payment_intent", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "id_map.json" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 626, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "kind": "verb_product", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "ranking": [ + "datadog.datadog_create_monitor", + "datadog.datadog_get_monitor", + "datadog.datadog_list_monitors", + "datadog.datadog_list_logs", + "datadog.datadog_query_metrics", + "id_map.json", + "github.create_gist", + "trello.trello_create_board", + "github.create_milestone", + "stripe.stripe_create_invoice" + ], + "hit1": "datadog.datadog_create_monitor", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 747, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "kind": "paraphrase", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "ranking": [ + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "sentry.sentry_list_events", + "sentry.sentry_list_projects", + "pagerduty.pagerduty_resolve_incident", + "sentry.sentry_list_issues", + "jira.jira_assign_issue", + "jira.jira_get_issue", + "gitlab.gitlab_get_issue", + "github.issue_read" + ], + "hit1": "sentry.sentry_get_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 754, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_apply", + "kubernetes.k8s_list_services", + "vercel.vercel_get_deployment", + "kubernetes.k8s_get_events", + "kubernetes.k8s_delete", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_pod_logs" + ], + "hit1": "kubernetes.k8s_scale_deployment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_apply", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_list_services", + "terraform.tf_apply", + "kubernetes.k8s_delete", + "kubernetes.k8s_get_events", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_pod_logs" + ], + "hit1": "kubernetes.k8s_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "ranking": [ + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_exec", + "docker.docker_run", + "docker.docker_images", + "docker.docker_ps", + "docker.docker_build", + "kubernetes.k8s_pod_logs", + "id_map.json" + ], + "hit1": "docker.docker_stop", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 587, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "kind": "verb_product", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "ranking": [ + "docker.docker_build", + "docker.docker_run", + "docker.docker_stop", + "docker.docker_images", + "docker.docker_rm", + "docker.docker_ps", + "docker.docker_logs", + "docker.docker_exec", + "filesystem.read_media_file", + "id_map.json" + ], + "hit1": "docker.docker_build", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 547, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "kind": "verb_product", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "ranking": [ + "playwright.browser_click", + "puppeteer.puppeteer_click", + "playwright.browser_type", + "playwright.browser_take_screenshot", + "playwright.browser_snapshot", + "playwright.browser_tab_list", + "playwright.browser_tab_new", + "clickup.clickup_list_tasks", + "playwright.browser_navigate", + "clickup.clickup_get_task" + ], + "hit1": "playwright.browser_click", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 573, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "kind": "paraphrase", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "ranking": [ + "playwright.browser_type", + "playwright.browser_wait_for", + "playwright.browser_navigate", + "playwright.browser_tab_list", + "playwright.browser_tab_close", + "playwright.browser_tab_new", + "playwright.browser_fill_form", + "playwright.browser_evaluate", + "filesystem.read_text_file", + "playwright.browser_snapshot" + ], + "hit1": "playwright.browser_type", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 577, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_list_projects", + "github.projects_list", + "gitlab.gitlab_list_issues", + "git.git_branch", + "gitlab.gitlab_list_merge_requests", + "git.git_checkout", + "git.git_show", + "vercel.vercel_list_projects", + "git.git_diff", + "jira.jira_list_projects" + ], + "hit1": "gitlab.gitlab_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 209, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 587, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "kind": "verb_product", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "ranking": [ + "bitbucket.bitbucket_create_pr", + "github.create_pull_request", + "github.merge_pull_request", + "bitbucket.bitbucket_create_issue", + "github.pull_request_read", + "bitbucket.bitbucket_list_prs", + "github.pull_request_review_write", + "github.update_pull_request_branch", + "github.update_pull_request", + "bitbucket.bitbucket_list_repos" + ], + "hit1": "bitbucket.bitbucket_create_pr", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 401, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 582, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1091, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "kind": "verb_product", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "ranking": [ + "gmail.gmail_create_draft", + "gmail.gmail_search_emails", + "gmail.gmail_list_labels", + "github.create_pull_request", + "gmail.gmail_send_email", + "gmail.gmail_trash_email", + "gmail.gmail_get_email", + "gmail.gmail_modify_email", + "id_map.json", + "github.create_gist" + ], + "hit1": "gmail.gmail_create_draft", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 448, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 730, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "kind": "verb_product", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_share_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_update_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "google_calendar.gcal_list_calendars", + "gitlab.gitlab_get_file", + "google_calendar.gcal_delete_event", + "figma.figma_get_file" + ], + "hit1": "google_drive.gdrive_share_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "kind": "verb_product", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_list_events", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_list_calendars", + "google_calendar.gcal_update_event", + "sentry.sentry_list_events", + "google_calendar.gcal_create_event", + "kubernetes.k8s_get_events", + "google_drive.gdrive_list_folder", + "google_drive.gdrive_search", + "google_drive.gdrive_update_file" + ], + "hit1": "google_calendar.gcal_list_events", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "redis.redis_delete", + "kubernetes.k8s_delete", + "linear.delete_attachment", + "jira.delete_attachment", + "aws_s3.s3_delete_object" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 731, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_projects", + "vercel.vercel_list_deployments", + "jira.jira_list_projects", + "vercel.vercel_get_deployment", + "sentry.sentry_list_projects", + "github.projects_list", + "linear.linear_list_projects", + "vercel.vercel_create_project", + "gitlab.gitlab_list_projects", + "github.projects_get" + ], + "hit1": "vercel.vercel_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "ranking": [ + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update", + "supabase.supabase_list_tables", + "supabase.supabase_invoke_fn", + "elasticsearch.es_get_doc", + "memory.delete_observations", + "aws_s3.s3_get_object", + "datadog.datadog_list_logs", + "memory.delete_relations" + ], + "hit1": "supabase.supabase_select", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 697, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "kind": "verb_product", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "figma.figma_get_file", + "figma.figma_get_comments", + "figma.figma_get_images", + "figma.figma_get_file_nodes", + "linear.delete_comment", + "linear.get_comment", + "slack.slack_post_message", + "linear.create_comment", + "notion.get_comment" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 688, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "kind": "verb_product", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "ranking": [ + "okta.okta_list_users", + "linear.list_users", + "notion.notion_list_users", + "slack.slack_list_users", + "okta.okta_create_user", + "okta.okta_get_user", + "okta.okta_deactivate_user", + "notion.list_users_tokens", + "slack.slack_get_users", + "github.search_users" + ], + "hit1": "okta.okta_list_users", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query_range", + "prometheus.prom_query", + "prometheus.prom_list_metrics", + "postgres.query", + "postgres.explain_query", + "datadog.datadog_query_metrics", + "salesforce.salesforce_query", + "mysql.mysql_query", + "sqlite.sqlite_query", + "notion.notion_query_database" + ], + "hit1": "prometheus.prom_query_range", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "kind": "verb_product", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "ranking": [ + "grafana.grafana_create_annotation", + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "id_map.json", + "github.create_gist", + "trello.trello_create_board", + "github.create_milestone", + "stripe.stripe_create_invoice", + "github.create_hook", + "vercel.vercel_create_project" + ], + "hit1": "grafana.grafana_create_annotation", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 659, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "kind": "identifier", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "ranking": [ + "terraform.tf_apply", + "terraform.tf_plan", + "kubernetes.k8s_apply", + "terraform.tf_state_list", + "id_map.json" + ], + "hit1": "terraform.tf_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 238, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 595, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "kind": "verb_product", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "ranking": [ + "asana.asana_list_tasks", + "asana.asana_get_task", + "clickup.clickup_list_tasks", + "asana.asana_create_task", + "asana.asana_create_project", + "asana.asana_update_task", + "id_map.json", + "trello.trello_list_cards", + "vercel.vercel_list_projects", + "kubernetes.k8s_list_deployments" + ], + "hit1": "asana.asana_list_tasks", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 627, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "ranking": [ + "trello.trello_add_comment", + "trello.trello_create_card", + "trello.trello_list_cards", + "figma.figma_post_comment", + "trello.trello_create_board", + "github.update_issue_comment", + "linear.delete_comment", + "linear.get_comment", + "linear.create_comment", + "notion.get_comment" + ], + "hit1": "trello.trello_add_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 745, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_update_task", + "clickup.clickup_get_task", + "clickup.clickup_create_task", + "clickup.clickup_list_tasks", + "asana.asana_update_task", + "puppeteer.puppeteer_click", + "asana.asana_get_task", + "playwright.browser_click", + "asana.asana_create_task", + "id_map.json" + ], + "hit1": "clickup.clickup_update_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "kind": "verb_product", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "ranking": [ + "confluence.confluence_search", + "confluence.confluence_get_page", + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "github.search_users", + "github.search_orgs", + "grafana.grafana_search_dashboards", + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "hubspot.hubspot_search_contacts" + ], + "hit1": "confluence.confluence_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 695, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_deal", + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_search_contacts", + "hubspot.hubspot_list_contacts", + "github.create_gist", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.create_or_update_file" + ], + "hit1": "hubspot.hubspot_create_deal", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 899, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_search", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_create_ticket", + "clickup.clickup_list_tasks", + "id_map.json", + "trello.trello_list_cards", + "vercel.vercel_list_projects", + "kubernetes.k8s_list_deployments" + ], + "hit1": "zendesk.zendesk_list_tickets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 590, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "kind": "verb_product", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "ranking": [ + "discord.discord_list_channels", + "discord.discord_list_members", + "discord.discord_get_channel", + "slack.slack_list_channels", + "discord.discord_send_message", + "discord.discord_add_reaction", + "clickup.clickup_list_tasks", + "id_map.json", + "trello.trello_list_cards", + "vercel.vercel_list_projects" + ], + "hit1": "discord.discord_list_channels", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 166, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 555, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "ranking": [ + "redis.redis_set", + "redis.redis_get", + "redis.redis_delete", + "redis.redis_keys", + "slack.slack_set_channel_topic", + "playwright.browser_press_key", + "jira.jira_get_issue", + "figma.figma_get_file", + "id_map.json", + "aws_s3.s3_get_object" + ], + "hit1": "redis.redis_set", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 530, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert", + "mongodb.mongo_delete", + "mongodb.mongo_update", + "mongodb.mongo_find", + "mongodb.mongo_list_collections", + "supabase.supabase_insert", + "id_map.json" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 607, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "kind": "verb_product", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "mysql.mysql_query", + "postgres.explain_query", + "mysql.mysql_list_tables", + "sqlite.sqlite_query", + "salesforce.salesforce_query", + "postgres.list_schemas", + "postgres.list_tables", + "postgres.describe_table", + "github.delete_check_run" + ], + "hit1": "postgres.query", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 545, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "elasticsearch.es_search", + "playwright.browser_tab_close", + "id_map.json", + "github.create_milestone", + "supabase.supabase_list_tables", + "github.get_milestone", + "jira.get_attachment", + "github.list_milestones" + ], + "hit1": "elasticsearch.es_index_doc", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 623, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_put_object", + "aws_s3.s3_list_objects", + "okta.okta_get_user", + "asana.asana_get_task", + "aws_s3.s3_list_buckets", + "elasticsearch.es_search", + "discord.discord_get_channel", + "okta.okta_create_user" + ], + "hit1": "aws_s3.s3_get_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 635, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones", + "cloudflare.cf_list_dns", + "cloudflare.cf_purge_cache", + "cloudflare.cf_create_dns", + "clickup.clickup_list_tasks", + "id_map.json", + "trello.trello_list_cards", + "vercel.vercel_list_projects", + "kubernetes.k8s_list_deployments", + "jira.jira_list_projects" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 542, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_create_product", + "stripe.stripe_create_customer", + "stripe.stripe_create_refund", + "stripe.stripe_create_price", + "stripe.stripe_get_customer", + "stripe.stripe_list_invoices", + "stripe.stripe_cancel_subscription", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "kind": "verb_product", + "gold": [ + "github.actions_list" + ], + "alts": [], + "ranking": [ + "github.actions_list", + "github.actions_get", + "github.actions_get_job_logs", + "github.actions_run_trigger", + "github.projects_list", + "git.git_branch", + "github.list_gists", + "git.git_checkout", + "github.list_starred_repositories", + "git.git_show" + ], + "hit1": "github.actions_list", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 453, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "kind": "verb_product", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "ranking": [ + "github.actions_get_job_logs", + "github.actions_get", + "github.actions_list", + "github.actions_run_trigger", + "gitlab.get_job", + "git.git_log", + "github.projects_get", + "git.git_checkout", + "github.get_gist", + "git.git_show" + ], + "hit1": "github.actions_get_job_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 442, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "kind": "verb_product", + "gold": [ + "github.create_repository" + ], + "alts": [], + "ranking": [ + "github.create_repository", + "github.delete_repository", + "github.star_repository", + "github.fork_repository", + "github.unstar_repository", + "github.list_repository_collaborators", + "github.create_branch", + "github.create_issue", + "github.create_or_update_file", + "git.git_create_branch" + ], + "hit1": "github.create_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 927, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "kind": "verb_product", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "ranking": [ + "git.git_checkout", + "git.git_create_branch", + "git.git_branch", + "github.create_branch", + "git.git_show", + "gitlab.gitlab_create_merge_request", + "github.list_commits", + "git.git_diff", + "github.delete_file", + "git.git_commit" + ], + "hit1": "git.git_checkout", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 835, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "kind": "identifier", + "gold": [ + "git.git_log" + ], + "alts": [], + "ranking": [ + "git.git_log", + "git.git_show", + "git.git_diff", + "git.git_status", + "git.git_diff_unstaged", + "git.git_checkout", + "git.git_branch", + "git.git_diff_staged", + "git.git_commit", + "git.git_add" + ], + "hit1": "git.git_log", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "kind": "verb_product", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "ranking": [ + "postgres.list_schemas", + "postgres.list_tables", + "postgres.query", + "postgres.explain_query", + "postgres.describe_table", + "clickup.clickup_list_tasks", + "id_map.json", + "trello.trello_list_cards", + "vercel.vercel_list_projects", + "kubernetes.k8s_list_deployments" + ], + "hit1": "postgres.list_schemas", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 158, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 500, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "kind": "verb_product", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "ranking": [ + "memory.open_nodes", + "memory.search_nodes", + "memory.read_graph", + "memory.create_entities", + "memory.create_relations", + "memory.delete_relations", + "memory.add_observations", + "memory.delete_entities", + "figma.figma_get_file_nodes", + "memory.delete_observations" + ], + "hit1": "memory.open_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 549, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "kind": "paraphrase", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "ranking": [ + "filesystem.get_file_info", + "filesystem.read_file", + "filesystem.read_media_file", + "filesystem.read_text_file", + "google_drive.gdrive_get_file", + "gitlab.gitlab_get_file", + "figma.figma_get_file_nodes", + "github.get_file_contents", + "filesystem.write_file", + "filesystem.edit_file" + ], + "hit1": "filesystem.get_file_info", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 770, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "kind": "paraphrase", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "ranking": [ + "filesystem.write_file", + "filesystem.read_file", + "playwright.browser_tab_new", + "brave_search.brave_local_search", + "filesystem.edit_file", + "filesystem.read_media_file", + "filesystem.read_text_file", + "github.issue_write", + "figma.figma_get_file", + "github.projects_write" + ], + "hit1": "filesystem.write_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 198, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 784, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "kind": "verb_product", + "gold": [ + "github.search_issues" + ], + "alts": [], + "ranking": [ + "github.search_issues", + "github.list_issues", + "linear.linear_search_issues", + "jira.jira_search_issues", + "git.git_checkout", + "git.git_show", + "github.search_users", + "git.git_branch", + "github.search_orgs", + "git.git_diff" + ], + "hit1": "github.search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 263, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 670, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "kind": "paraphrase", + "gold": [ + "github.get_me" + ], + "alts": [], + "ranking": [ + "github.get_teams", + "git.git_commit", + "github.get_me", + "git.git_status", + "github.list_notifications", + "git.git_add", + "slack.slack_get_user_profile", + "git.git_checkout", + "okta.okta_get_user", + "git.git_show" + ], + "hit1": "github.get_teams", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 592, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "cost": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.0133333333333334, + "p95_calls": 1.0, + "success_within_1": 0.9866666666666667, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 236.3, + "p95_tokens": 336.8499999999999, + "total_tokens": 35445, + "tokens_per_successful_query": 236.3, + "total_tokens_successful": 35445 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.0066666666666666, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 372.78, + "p95_tokens": 522.5999999999991, + "total_tokens": 55917, + "tokens_per_successful_query": 372.78, + "total_tokens_successful": 55917 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.0, + "p95_calls": 1.0, + "success_within_1": 1.0, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 705.18, + "p95_tokens": 945.6999999999994, + "total_tokens": 105777, + "tokens_per_successful_query": 705.18, + "total_tokens_successful": 105777 + } + } +} diff --git a/experiments/mcp-tool-search/results/hybrid_rrf_bm25_embed.csv b/experiments/mcp-tool-search/results/hybrid_rrf_bm25_embed.csv new file mode 100644 index 00000000..ecdfc0f5 --- /dev/null +++ b/experiments/mcp-tool-search/results/hybrid_rrf_bm25_embed.csv @@ -0,0 +1,151 @@ +id,query,kind,hit1,recall@1,recall@3,recall@5,mrr@10,ndcg@10,hit_rank,k3_calls,k3_tokens,k3_success1,k5_calls,k5_tokens,k5_success1,k10_calls,k10_tokens,k10_success1,ranking +q001,Create a GitHub issue,verb_product,github.create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,380,1.0,1,584,1.0,1,1042,1.0,github.create_issue github.issue_write gitlab.gitlab_create_issue github.add_issue_comment github.issue_read github.assign_copilot_to_issue github.create_repository github.create_gist github.update_issue_comment bitbucket.bitbucket_create_issue +q002,Create a Jira ticket,verb_product,jira.jira_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,330,1.0,1,535,1.0,1,929,1.0,jira.jira_create_issue jira.create_sprint jira.create_board jira.jira_create_sprint jira.create_version jira.create_filter jira.create_webhook zendesk.zendesk_create_ticket jira.get_sprint zendesk.zendesk_get_ticket +q003,Send a Slack message,verb_product,slack.slack_post_message,1.0,1.0,1.0,1.0000,1.0000,1,1,252,1.0,1,374,1.0,1,719,1.0,slack.slack_post_message slack.slack_add_reaction slack.slack_create_channel slack.get_reminder discord.discord_send_message slack.slack_upload_file slack.slack_leave_channel slack.slack_join_channel slack.slack_search_messages slack.slack_reply_to_thread +q004,List open pull requests,resource,github.list_pull_requests,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,477,1.0,1,976,1.0,github.list_pull_requests github.search_pull_requests bitbucket.bitbucket_list_prs github.create_pull_request github.pull_request_read gitlab.gitlab_list_merge_requests github.pull_request_review_write github.update_pull_request playwright.browser_tab_list github.merge_pull_request +q005,Open a pull request on GitHub,verb_product,github.create_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,406,1.0,1,564,1.0,1,1144,1.0,github.create_pull_request github.pull_request_read github.pull_request_review_write github.search_pull_requests github.merge_pull_request github.request_copilot_review github.list_pull_requests github.update_pull_request github.update_pull_request_branch github.add_comment_to_pending_review +q006,Run a SQL query against postgres,verb_product,postgres.query,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,295,1.0,1,596,1.0,postgres.query mysql.mysql_query sqlite.sqlite_query postgres.explain_query salesforce.salesforce_query postgres.list_schemas postgres.describe_table postgres.list_tables prometheus.prom_query prometheus.prom_query_range +q007,Read a local file from disk,paraphrase,filesystem.read_file,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,406,1.0,1,747,1.0,filesystem.read_file filesystem.read_text_file filesystem.read_media_file figma.figma_get_file_nodes github.get_file_contents figma.figma_get_file gitlab.gitlab_get_file figma.figma_get_images filesystem.get_file_info slack.get_file +q008,What files are in this folder,paraphrase,google_drive.gdrive_list_folder,0.0,1.0,1.0,0.5000,0.6863,2,1,191,1.0,1,314,1.0,1,648,1.0,google_drive.gdrive_list_folder filesystem.list_directory filesystem.list_directory_with_sizes filesystem.search_files filesystem.directory_tree git.git_add filesystem.move_file github.create_gist google_drive.gdrive_search slack.list_files +q009,Remember this fact in the knowledge graph,paraphrase,memory.read_graph,0.0,1.0,1.0,0.3333,0.5438,3,1,164,1.0,1,261,1.0,1,568,1.0,memory.read_graph memory.create_relations memory.add_observations memory.create_entities memory.delete_relations memory.open_nodes memory.delete_entities memory.search_nodes postgres.list_schemas sequential_thinking.sequentialthinking +q010,Search the web with Brave,verb_product,brave_search.brave_web_search,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,351,1.0,1,637,1.0,brave_search.brave_web_search brave_search.brave_local_search confluence.confluence_search grafana.grafana_search_dashboards github.search_orgs memory.search_nodes hubspot.hubspot_search_contacts slack.slack_search_messages github.search_users github.search_pull_requests +q011,Fetch the contents of a URL,verb_product,fetch.fetch,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,372,1.0,1,700,1.0,fetch.fetch playwright.browser_navigate puppeteer.puppeteer_navigate filesystem.read_file github.get_file_contents playwright.browser_take_screenshot docker.docker_logs playwright.browser_tab_new filesystem.list_directory twilio.twilio_make_call +q012,Show git status,identifier,git.git_status,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,312,1.0,1,616,1.0,git.git_status git.git_show git.git_diff_unstaged git.git_log git.git_diff git.git_branch git.git_diff_staged github.list_milestones git.git_commit git.git_checkout +q013,Commit staged changes,paraphrase,git.git_commit,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,327,1.0,1,706,1.0,git.git_commit git.git_reset git.git_diff_staged git.git_show github.get_commit github.update_milestone filesystem.move_file git.git_log github.push_files slack.update_reminder +q014,Create a Linear issue,verb_product,linear.linear_create_comment,0.0,1.0,1.0,0.5000,0.6309,2,1,275,1.0,1,463,1.0,1,925,1.0,linear.linear_create_comment linear.linear_create_issue linear.linear_get_issue linear.linear_update_issue linear.linear_assign_issue linear.linear_create_project linear.create_label linear.create_cycle linear.create_attachment linear.create_comment +q015,Make a task in my project tracker Linear,paraphrase,linear.linear_create_project,0.0,0.0,1.0,0.2500,0.4307,4,2,515,0.0,1,440,1.0,1,801,1.0,linear.linear_create_project linear.linear_list_projects asana.asana_create_task linear.linear_create_issue linear.get_cycle linear.get_label clickup.clickup_update_task linear.get_attachment linear.create_label linear.list_labels +q016,Post a comment on a GitHub issue,verb_product,github.add_issue_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,302,1.0,1,542,1.0,1,1028,1.0,github.add_issue_comment github.update_issue_comment jira.jira_add_comment github.assign_copilot_to_issue github.issue_write figma.figma_post_comment github.issue_read github.add_comment_to_pending_review github.get_discussion_comments linear.linear_create_comment +q017,Merge the pull request,ambiguous,github.merge_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,283,1.0,1,495,1.0,1,1093,1.0,github.merge_pull_request gitlab.gitlab_merge_merge_request gitlab.gitlab_get_merge_request github.update_pull_request_branch gitlab.gitlab_create_merge_request github.pull_request_read github.create_pull_request github.update_pull_request gitlab.gitlab_list_merge_requests github.pull_request_review_write +q018,Trigger a GitHub Actions workflow,verb_product,github.actions_run_trigger,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,458,1.0,1,734,1.0,github.actions_run_trigger github.actions_get github.actions_list github.actions_get_job_logs circleci.circleci_trigger_pipeline github.search_commits github.dismiss_notification github.update_gist github.mark_all_notifications_read github.search_pull_requests +q019,Get Dependabot alerts,resource,github.list_dependabot_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,427,1.0,1,706,1.0,github.list_dependabot_alerts github.get_dependabot_alert github.list_secret_scanning_alerts github.list_code_scanning_alerts vercel.vercel_get_deployment slack.get_reminder github.get_milestone github.get_hook gitlab.get_release pagerduty.pagerduty_get_incident +q020,List Sentry issues,verb_product,sentry.sentry_list_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,393,1.0,1,772,1.0,sentry.sentry_list_issues sentry.sentry_list_projects sentry.sentry_list_events sentry.sentry_get_issue linear.linear_list_issues sentry.sentry_update_issue gitlab.gitlab_list_issues github.list_issues stripe.stripe_list_customers hubspot.hubspot_list_contacts +q021,Create a Stripe customer,verb_product,stripe.stripe_create_customer,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,322,1.0,1,669,1.0,stripe.stripe_create_customer stripe.stripe_create_invoice stripe.stripe_create_payment_intent stripe.stripe_get_customer stripe.stripe_list_invoices stripe.stripe_create_product stripe.stripe_list_subscriptions stripe.stripe_create_price stripe.stripe_create_refund stripe.stripe_list_customers +q022,Refund a Stripe payment,paraphrase,stripe.stripe_create_refund,1.0,1.0,1.0,1.0000,1.0000,1,1,195,1.0,1,332,1.0,1,632,1.0,stripe.stripe_create_refund stripe.stripe_get_payment_intent stripe.stripe_create_invoice stripe.stripe_create_payment_intent stripe.stripe_get_customer stripe.stripe_list_invoices stripe.stripe_cancel_subscription stripe.stripe_create_customer stripe.stripe_create_product stripe.stripe_list_customers +q023,Send an email with Gmail,verb_product,gmail.gmail_send_email,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,399,1.0,1,743,1.0,gmail.gmail_send_email gmail.gmail_get_email gmail.gmail_trash_email gmail.gmail_modify_email gmail.gmail_create_draft gmail.gmail_list_labels twilio.twilio_send_sms gmail.gmail_search_emails discord.discord_send_message google_drive.gdrive_share_file +q024,Search my Gmail inbox,verb_product,gmail.gmail_search_emails,1.0,1.0,1.0,1.0000,1.0000,1,1,178,1.0,1,315,1.0,1,678,1.0,gmail.gmail_search_emails gmail.gmail_list_labels gmail.gmail_get_email gmail.gmail_trash_email gmail.gmail_create_draft gmail.gmail_modify_email gmail.gmail_send_email hubspot.hubspot_search_contacts github.search_users slack.slack_search_messages +q025,Upload a file to Google Drive,paraphrase,google_drive.gdrive_update_file,0.0,1.0,1.0,0.3333,0.5000,3,1,307,1.0,1,418,1.0,1,806,1.0,google_drive.gdrive_update_file google_drive.gdrive_share_file google_drive.gdrive_create_file google_drive.gdrive_get_file google_drive.gdrive_search slack.slack_upload_file google_drive.gdrive_list_folder aws_s3.s3_put_object filesystem.edit_file filesystem.write_file +q026,Create a calendar event,resource,google_calendar.gcal_create_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,427,1.0,1,824,1.0,google_calendar.gcal_create_event google_calendar.gcal_update_event google_calendar.gcal_delete_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars trello.trello_create_board grafana.grafana_create_annotation shopify.shopify_create_order gmail.gmail_create_draft github.create_milestone +q027,List Kubernetes pods,verb_product,kubernetes.k8s_list_pods,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,314,1.0,1,706,1.0,kubernetes.k8s_list_pods kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_get_pod kubernetes.k8s_apply kubernetes.k8s_pod_logs kubernetes.k8s_scale_deployment kubernetes.k8s_delete aws_s3.s3_list_buckets +q028,Get logs from a k8s pod,verb_product,kubernetes.k8s_pod_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,262,1.0,1,455,1.0,1,740,1.0,kubernetes.k8s_pod_logs kubernetes.k8s_get_pod docker.docker_logs github.actions_get_job_logs datadog.datadog_list_logs git.git_log slack.slack_get_channel_history memory.delete_observations redis.redis_get github.get_runner +q029,Run a docker container,verb_product,docker.docker_run,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,331,1.0,1,587,1.0,docker.docker_run docker.docker_exec docker.docker_stop docker.docker_ps docker.docker_logs docker.docker_rm docker.docker_images docker.docker_build github.get_check_run terraform.tf_plan +q030,Show docker container logs,verb_product,docker.docker_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,226,1.0,1,352,1.0,1,683,1.0,docker.docker_logs kubernetes.k8s_pod_logs docker.docker_stop docker.docker_run docker.docker_images docker.docker_exec git.git_log docker.docker_ps docker.docker_rm datadog.datadog_list_logs +q031,Navigate the browser to a URL,ambiguous,playwright.browser_navigate,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,309,1.0,1,624,1.0,playwright.browser_navigate puppeteer.puppeteer_navigate playwright.browser_tab_new fetch.fetch playwright.browser_press_key playwright.browser_click playwright.browser_evaluate twilio.twilio_make_call playwright.browser_snapshot playwright.browser_take_screenshot +q032,Take a screenshot of the page,ambiguous,playwright.browser_take_screenshot,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,314,1.0,1,629,1.0,playwright.browser_take_screenshot puppeteer.puppeteer_screenshot playwright.browser_snapshot playwright.browser_evaluate playwright.browser_press_key playwright.browser_click notion.notion_get_page confluence.confluence_get_page confluence.confluence_create_page playwright.browser_navigate +q033,Create a GitLab merge request,verb_product,gitlab.gitlab_merge_merge_request,0.0,1.0,1.0,0.5000,0.6309,2,1,297,1.0,1,475,1.0,1,929,1.0,gitlab.gitlab_merge_merge_request gitlab.gitlab_create_merge_request gitlab.gitlab_get_merge_request github.merge_pull_request gitlab.gitlab_list_merge_requests gitlab.gitlab_create_pipeline gitlab.gitlab_create_issue github.create_pull_request gitlab.gitlab_list_projects gitlab.gitlab_get_file +q034,Search Notion pages,verb_product,notion.notion_search,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,334,1.0,1,667,1.0,notion.notion_search notion.notion_get_page notion.notion_list_users notion.notion_query_database notion.notion_update_page notion.notion_get_block notion.notion_create_database notion.notion_get_comments notion.notion_create_page github.search_orgs +q035,Create a Notion page,verb_product,notion.notion_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,230,1.0,1,371,1.0,1,781,1.0,notion.notion_create_page notion.notion_get_page notion.notion_update_page notion.notion_create_database notion.notion_get_comments notion.create_data_source notion.create_comment notion.create_users_token notion.notion_search notion.notion_list_users +q036,Query a Notion database,verb_product,notion.notion_query_database,1.0,1.0,1.0,1.0000,1.0000,1,1,261,1.0,1,386,1.0,1,656,1.0,notion.notion_query_database notion.notion_create_database notion.notion_search notion.notion_create_page notion.notion_list_users notion.notion_get_page notion.notion_get_block notion.notion_get_comments notion.get_data_source notion.get_comment +q037,Create a Zendesk ticket,verb_product,zendesk.zendesk_create_ticket,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,368,1.0,1,782,1.0,zendesk.zendesk_create_ticket zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_list_tickets zendesk.zendesk_search jira.jira_create_issue vercel.vercel_create_project gmail.gmail_create_draft hubspot.hubspot_create_deal stripe.stripe_create_refund +q038,Create a HubSpot contact,verb_product,hubspot.hubspot_create_contact,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,866,1.0,hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_create_deal hubspot.hubspot_search_contacts hubspot.hubspot_list_contacts slack.slack_create_channel github.create_pull_request github.pull_request_review_write shopify.shopify_create_order stripe.stripe_create_customer +q039,Create a Salesforce lead,verb_product,salesforce.salesforce_convert_lead,0.0,1.0,1.0,0.5000,0.6309,2,1,252,1.0,1,380,1.0,1,798,1.0,salesforce.salesforce_convert_lead salesforce.salesforce_create_lead salesforce.salesforce_create_record salesforce.salesforce_list_opportunities salesforce.salesforce_get_record salesforce.salesforce_query salesforce.salesforce_update_record hubspot.hubspot_create_deal linear.linear_create_project github.create_hook +q040,Run a SOQL query,identifier,salesforce.salesforce_query,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,291,1.0,1,694,1.0,salesforce.salesforce_query mysql.mysql_query sqlite.sqlite_query postgres.query prometheus.prom_query prometheus.prom_query_range postgres.explain_query datadog.datadog_query_metrics notion.notion_query_database datadog.datadog_list_monitors +q041,List Shopify orders,verb_product,shopify.shopify_list_orders,1.0,1.0,1.0,1.0000,1.0000,1,1,193,1.0,1,333,1.0,1,647,1.0,shopify.shopify_list_orders shopify.shopify_list_customers shopify.shopify_get_order shopify.shopify_list_products shopify.shopify_create_order shopify.shopify_get_product shopify.shopify_update_product shopify.shopify_create_product clickup.clickup_list_tasks aws_s3.s3_list_buckets +q042,Create a PagerDuty incident,verb_product,pagerduty.pagerduty_create_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,403,1.0,1,821,1.0,pagerduty.pagerduty_create_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls shopify.shopify_create_order gmail.gmail_create_draft confluence.confluence_create_page datadog.datadog_create_monitor +q043,Who is on call in PagerDuty,paraphrase,pagerduty.pagerduty_list_oncalls,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,389,1.0,1,725,1.0,pagerduty.pagerduty_list_oncalls pagerduty.pagerduty_get_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents twilio.twilio_make_call playwright.browser_press_key playwright.browser_click gitlab.get_snippet +q044,Query Datadog metrics,verb_product,datadog.datadog_query_metrics,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,458,1.0,1,785,1.0,datadog.datadog_query_metrics datadog.datadog_list_monitors datadog.datadog_create_monitor datadog.datadog_list_logs datadog.datadog_get_monitor prometheus.prom_list_metrics memory.search_nodes prometheus.prom_query prometheus.prom_query_range slack.slack_search_messages +q045,Search Datadog logs,verb_product,datadog.datadog_list_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,458,1.0,1,785,1.0,datadog.datadog_list_logs datadog.datadog_list_monitors datadog.datadog_get_monitor datadog.datadog_create_monitor datadog.datadog_query_metrics docker.docker_logs github.search_users github.search_orgs github.actions_get_job_logs slack.slack_search_messages +q046,List S3 buckets,verb_product,aws_s3.s3_list_buckets,1.0,1.0,1.0,1.0000,1.0000,1,1,128,1.0,1,233,1.0,1,482,1.0,aws_s3.s3_list_buckets jira.jira_list_projects gmail.gmail_list_labels kubernetes.k8s_list_services bitbucket.bitbucket_list_repos salesforce.salesforce_list_opportunities cloudflare.cf_list_zones clickup.clickup_list_tasks okta.okta_list_users kubernetes.k8s_list_deployments +q047,Upload an object to S3,paraphrase,aws_s3.s3_put_object,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,420,1.0,1,835,1.0,aws_s3.s3_put_object aws_s3.s3_get_object aws_s3.s3_delete_object slack.slack_upload_file twilio.twilio_send_sms aws_s3.s3_list_objects twilio.twilio_make_call filesystem.write_file gmail.gmail_create_draft github.assign_copilot_to_issue +q048,Purge Cloudflare cache,verb_product,cloudflare.cf_purge_cache,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,341,1.0,1,618,1.0,cloudflare.cf_purge_cache cloudflare.cf_list_zones cloudflare.cf_list_dns cloudflare.cf_create_dns slack.delete_bookmark slack.delete_canvase memory.delete_observations redis.redis_delete aws_s3.s3_delete_object linear.delete_attachment +q049,Create a Cloudflare DNS record,verb_product,cloudflare.cf_create_dns,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,356,1.0,1,819,1.0,cloudflare.cf_create_dns cloudflare.cf_list_dns salesforce.salesforce_create_record cloudflare.cf_purge_cache cloudflare.cf_list_zones salesforce.salesforce_get_record salesforce.salesforce_update_record datadog.datadog_create_monitor hubspot.hubspot_create_deal confluence.confluence_create_page +q050,List Vercel deployments,verb_product,vercel.vercel_list_deployments,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,291,1.0,1,542,1.0,vercel.vercel_list_deployments vercel.vercel_list_projects vercel.vercel_get_deployment kubernetes.k8s_list_deployments vercel.vercel_create_project kubernetes.k8s_list_services jira.jira_list_projects sentry.sentry_list_projects okta.okta_list_users datadog.datadog_list_monitors +q051,Insert a row into Supabase,verb_product,supabase.supabase_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,230,1.0,1,352,1.0,1,701,1.0,supabase.supabase_insert supabase.supabase_select mongodb.mongo_insert supabase.supabase_update supabase.supabase_list_tables supabase.supabase_invoke_fn playwright.browser_type linear.update_cycle linear.update_attachment linear.update_user +q052,Export images from Figma,paraphrase,figma.figma_get_images,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,328,1.0,1,685,1.0,figma.figma_get_images figma.figma_get_file_nodes figma.figma_get_file docker.docker_images figma.figma_get_comments figma.figma_post_comment filesystem.read_media_file github.get_file_contents memory.delete_observations linear.update_attachment +q053,Send an SMS with Twilio,verb_product,twilio.twilio_send_sms,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,409,1.0,1,806,1.0,twilio.twilio_send_sms twilio.twilio_list_messages twilio.twilio_make_call gmail.gmail_send_email discord.discord_send_message slack.slack_add_reaction aws_s3.s3_put_object playwright.browser_type slack.slack_reply_to_thread puppeteer.puppeteer_click +q054,Create an Okta user,verb_product,okta.okta_create_user,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,329,1.0,1,676,1.0,okta.okta_create_user okta.okta_get_user okta.okta_deactivate_user okta.okta_list_users linear.create_user asana.asana_create_project asana.asana_create_task linear.get_user linear.update_user slack.slack_get_user_profile +q055,Instant query Prometheus,verb_product,prometheus.prom_query,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,379,1.0,1,661,1.0,prometheus.prom_query prometheus.prom_query_range prometheus.prom_list_metrics datadog.datadog_query_metrics memory.search_nodes salesforce.salesforce_query postgres.query sqlite.sqlite_query datadog.datadog_list_monitors confluence.confluence_search +q056,Search Grafana dashboards,verb_product,grafana.grafana_search_dashboards,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,319,1.0,1,677,1.0,grafana.grafana_search_dashboards grafana.grafana_get_dashboard grafana.grafana_create_annotation github.search_orgs github.search_users confluence.confluence_search github.search_pull_requests brave_search.brave_web_search brave_search.brave_local_search github.search_issues +q057,Trigger a CircleCI pipeline,verb_product,circleci.circleci_trigger_pipeline,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,390,1.0,1,755,1.0,circleci.circleci_trigger_pipeline circleci.circleci_get_pipeline circleci.circleci_list_pipelines github.actions_run_trigger gitlab.gitlab_create_pipeline gitlab.update_pipeline hubspot.hubspot_create_deal gitlab.get_pipeline gitlab.delete_pipeline gitlab.create_pipeline +q058,Run terraform plan,identifier,terraform.tf_plan,1.0,1.0,1.0,1.0000,1.0000,1,1,172,1.0,1,334,1.0,1,758,1.0,terraform.tf_plan terraform.tf_apply terraform.tf_state_list github.update_check_run docker.docker_run github.get_check_run github.create_check_run github.actions_run_trigger docker.docker_exec github.actions_get +q059,Create an Asana task,verb_product,asana.asana_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,405,1.0,1,820,1.0,asana.asana_create_task asana.asana_get_task asana.asana_create_project asana.asana_update_task asana.asana_list_tasks clickup.clickup_create_task okta.okta_create_user github.create_issue notion.notion_create_database clickup.clickup_get_task +q060,Create a Trello card,verb_product,trello.trello_create_card,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,364,1.0,1,734,1.0,trello.trello_create_card trello.trello_add_comment trello.trello_create_board trello.trello_list_cards stripe.stripe_create_invoice stripe.stripe_create_customer stripe.stripe_create_product grafana.grafana_create_annotation stripe.stripe_create_refund stripe.stripe_create_price +q061,Create a ClickUp task,verb_product,clickup.clickup_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,364,1.0,1,734,1.0,clickup.clickup_create_task clickup.clickup_get_task clickup.clickup_list_tasks clickup.clickup_update_task puppeteer.puppeteer_click asana.asana_create_task playwright.browser_click gmail.gmail_create_draft asana.asana_get_task shopify.shopify_create_order +q062,Create a Confluence page,verb_product,confluence.confluence_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,401,1.0,1,751,1.0,confluence.confluence_create_page confluence.confluence_get_page confluence.confluence_update_page confluence.confluence_search notion.notion_create_page github.search_code notion.notion_get_page playwright.browser_snapshot shopify.shopify_create_product github.create_artifact +q063,Send a Discord message,verb_product,discord.discord_send_message,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,343,1.0,1,718,1.0,discord.discord_send_message discord.discord_add_reaction discord.discord_get_channel slack.slack_post_message discord.discord_list_channels twilio.twilio_send_sms gmail.gmail_trash_email discord.discord_list_members gmail.gmail_send_email slack.slack_add_reaction +q064,Search Elasticsearch,verb_product,elasticsearch.es_search,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,310,1.0,1,653,1.0,elasticsearch.es_search confluence.confluence_search elasticsearch.es_get_doc github.search_orgs github.search_users elasticsearch.es_index_doc hubspot.hubspot_search_contacts grafana.grafana_search_dashboards brave_search.brave_web_search github.search_pull_requests +q065,Get a Redis key,verb_product,redis.redis_get,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,280,1.0,1,566,1.0,redis.redis_get redis.redis_set redis.redis_delete redis.redis_keys playwright.browser_press_key figma.figma_get_file jira.jira_get_issue aws_s3.s3_get_object github.get_milestone gitlab.get_pipeline +q066,Find MongoDB documents,verb_product,mongodb.mongo_find,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,381,1.0,1,729,1.0,mongodb.mongo_find mongodb.mongo_delete mongodb.mongo_insert mongodb.mongo_list_collections mongodb.mongo_update elasticsearch.es_get_doc filesystem.search_files confluence.confluence_search datadog.datadog_list_logs hubspot.hubspot_search_contacts +q067,List SQLite tables,verb_product,sqlite.sqlite_list_tables,1.0,1.0,1.0,1.0000,1.0000,1,1,151,1.0,1,240,1.0,1,517,1.0,sqlite.sqlite_list_tables mysql.mysql_list_tables sqlite.sqlite_describe_table supabase.supabase_list_tables postgres.list_tables okta.okta_list_users trello.trello_list_cards datadog.datadog_list_monitors jira.jira_list_projects clickup.clickup_list_tasks +q068,Describe a postgres table,verb_product,postgres.describe_table,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,292,1.0,1,590,1.0,postgres.describe_table sqlite.sqlite_describe_table postgres.explain_query postgres.list_schemas supabase.supabase_select supabase.supabase_insert postgres.list_tables postgres.query supabase.supabase_update mysql.mysql_list_tables +q069,github_create_issue,identifier,github.create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,378,1.0,1,564,1.0,1,1048,1.0,github.create_issue github.issue_write gitlab.gitlab_create_issue bitbucket.bitbucket_create_issue github.create_environment github.assign_copilot_to_issue github.create_repository github.add_issue_comment github.create_package github.create_artifact +q070,createIssue,identifier,gmail.gmail_create_draft,0.0,0.0,0.0,0.0000,0.0000,39,13,3322,0.0,8,3368,0.0,4,3296,0.0,gmail.gmail_create_draft stripe.stripe_create_product okta.okta_create_user stripe.stripe_create_invoice salesforce.salesforce_create_record trello.trello_create_board notion.notion_create_database vercel.vercel_create_project filesystem.create_directory github.create_gist +q071,list-directory,identifier,filesystem.list_directory,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,269,1.0,1,508,1.0,filesystem.list_directory filesystem.list_directory_with_sizes clickup.clickup_list_tasks okta.okta_list_users jira.jira_list_projects gmail.gmail_list_labels aws_s3.s3_list_buckets sentry.sentry_list_projects vercel.vercel_list_projects trello.trello_list_cards +q072,sequential thinking next thought,paraphrase,sequential_thinking.sequentialthinking,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,344,1.0,1,650,1.0,sequential_thinking.sequentialthinking memory.create_relations memory.add_observations memory.search_nodes memory.create_entities memory.delete_relations notion.update_comment notion.notion_search memory.open_nodes notion.notion_append_block +q073,What time is it in Tokyo,paraphrase,time.get_current_time,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,391,1.0,1,696,1.0,time.get_current_time time.convert_time google_calendar.gcal_list_events playwright.browser_wait_for grafana.grafana_create_annotation jira.jira_add_worklog jira.get_sprint jira.get_webhook jira.get_board gitlab.get_milestone +q074,Convert time between timezones,verb_product,time.convert_time,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,390,1.0,1,698,1.0,time.convert_time time.get_current_time playwright.browser_wait_for grafana.grafana_create_annotation google_calendar.gcal_list_events salesforce.salesforce_convert_lead datadog.datadog_query_metrics memory.create_relations cloudflare.cf_list_zones git.git_diff +q075,Star a GitHub repo,verb_product,github.star_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,245,1.0,1,400,1.0,1,801,1.0,github.star_repository github.unstar_repository github.search_issues github.get_label github.list_releases github.get_file_blame github.actions_list github.fork_repository github.list_tags github.get_tag +q076,Fork a repository,resource,github.fork_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,386,1.0,1,875,1.0,github.fork_repository github.unstar_repository github.star_repository github.delete_repository github.create_repository github.get_repository_tree github.list_releases github.list_dependabot_alerts github.list_pull_requests github.create_or_update_file +q077,Get the latest GitHub release,verb_product,github.get_latest_release,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,344,1.0,1,714,1.0,github.get_latest_release github.get_release_by_tag gitlab.get_release github.get_gist github.get_notification_details github.get_teams github.get_me github.get_code_scanning_alert github.get_commit github.get_secret_scanning_alert +q078,Create a gist,resource,github.create_gist,1.0,1.0,1.0,1.0000,1.0000,1,1,229,1.0,1,437,1.0,1,877,1.0,github.create_gist github.get_gist github.update_gist github.create_milestone gitlab.create_snippet github.create_hook github.create_repository gmail.gmail_create_draft notion.notion_create_database gitlab.create_label +q079,Search GitHub code,verb_product,github.search_code,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,385,1.0,1,719,1.0,github.search_code github.search_orgs github.search_users github.get_code_scanning_alert github.list_code_scanning_alerts github.search_pull_requests github.search_commits github.search_repositories github.search_issues gitlab.gitlab_list_projects +q080,List GitHub notifications,verb_product,github.list_notifications,1.0,1.0,1.0,1.0000,1.0000,1,1,174,1.0,1,321,1.0,1,704,1.0,github.list_notifications github.mark_all_notifications_read github.list_gists github.actions_list github.list_secret_scanning_alerts github.list_code_scanning_alerts github.list_releases github.list_starred_repositories github.list_tags github.list_pull_requests +q081,Assign Copilot to an issue,verb_product,github.assign_copilot_to_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,435,1.0,1,822,1.0,github.assign_copilot_to_issue linear.linear_assign_issue jira.jira_assign_issue github.request_copilot_review linear.linear_create_comment jira.jira_update_issue github.create_issue jira.jira_get_issue jira.jira_transition_issue jira.jira_add_comment +q082,Get secret scanning alerts,resource,github.list_secret_scanning_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,295,1.0,1,470,1.0,1,756,1.0,github.list_secret_scanning_alerts github.get_secret_scanning_alert github.list_code_scanning_alerts github.get_code_scanning_alert github.list_dependabot_alerts slack.get_reminder github.get_hook pagerduty.pagerduty_get_incident gitlab.get_snippet vercel.vercel_get_deployment +q083,Create or update a file on GitHub,verb_product,github.create_or_update_file,1.0,1.0,1.0,1.0000,0.7602,1,1,389,1.0,1,536,1.0,1,1012,1.0,github.create_or_update_file github.projects_write github.issue_write github.update_gist slack.update_file github.create_gist github.label_write filesystem.write_file github.update_issue_comment github.get_file_contents +q084,List collaborators on a repo,resource,github.list_repository_collaborators,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,410,1.0,1,804,1.0,github.list_repository_collaborators github.actions_list github.list_commits github.list_labels github.list_releases github.list_discussions github.list_tags github.list_branches github.list_dependabot_alerts github.list_issues +q085,Reply in a Slack thread,verb_product,slack.slack_reply_to_thread,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,355,1.0,1,665,1.0,slack.slack_reply_to_thread slack.slack_get_thread_replies slack.get_reminder slack.get_canvase slack.delete_reminder slack.update_reminder slack.get_pin slack.get_file slack.update_pin slack.update_file +q086,Add an emoji reaction on Slack,verb_product,slack.slack_add_reaction,1.0,1.0,1.0,1.0000,1.0000,1,1,270,1.0,1,414,1.0,1,764,1.0,slack.slack_add_reaction discord.discord_add_reaction slack.slack_post_message slack.get_reminder slack.slack_reply_to_thread slack.get_canvase slack.slack_get_thread_replies slack.slack_search_messages slack.slack_create_channel slack.update_canvase +q087,Search Slack messages,verb_product,slack.slack_search_messages,1.0,1.0,1.0,1.0000,1.0000,1,1,213,1.0,1,316,1.0,1,636,1.0,slack.slack_search_messages slack.slack_get_channel_history github.search_orgs github.search_pull_requests github.search_users gmail.gmail_search_emails slack.get_reminder hubspot.hubspot_search_contacts slack.slack_list_channels slack.slack_post_message +q088,Create a Slack channel,verb_product,slack.slack_create_channel,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,339,1.0,1,751,1.0,slack.slack_create_channel slack.slack_join_channel slack.slack_leave_channel slack.slack_post_message slack.slack_set_channel_topic slack.slack_reply_to_thread slack.slack_get_channel_history slack.slack_add_reaction slack.slack_get_thread_replies slack.slack_upload_file +q089,Transition a Jira issue,verb_product,jira.jira_transition_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,222,1.0,1,342,1.0,1,706,1.0,jira.jira_transition_issue jira.jira_update_issue jira.jira_assign_issue jira.jira_get_issue jira.jira_add_comment jira.jira_create_issue jira.jira_add_worklog jira.jira_search_issues jira.get_sprint jira.jira_list_projects +q090,Search Jira with JQL,identifier,jira.jira_search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,179,1.0,1,286,1.0,1,549,1.0,jira.jira_search_issues jira.jira_list_projects jira.jira_get_issue github.search_pull_requests confluence.confluence_search hubspot.hubspot_search_contacts github.search_users jira.get_filter github.search_orgs jira.get_webhook +q091,Add a worklog to Jira,verb_product,jira.jira_add_worklog,1.0,1.0,1.0,1.0000,1.0000,1,1,214,1.0,1,332,1.0,1,689,1.0,jira.jira_add_worklog jira.jira_add_comment jira.jira_list_projects jira.jira_get_issue jira.jira_update_issue jira.jira_transition_issue jira.jira_create_issue jira.get_sprint jira.get_webhook jira.jira_assign_issue +q092,List Linear teams,verb_product,linear.linear_list_teams,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,339,1.0,1,673,1.0,linear.linear_list_teams linear.linear_list_projects linear.linear_list_issues linear.list_cycles linear.linear_create_project linear.list_labels linear.list_users linear.linear_get_issue linear.linear_search_issues linear.list_attachments +q093,Comment on a Linear issue,verb_product,linear.linear_create_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,195,1.0,1,334,1.0,1,794,1.0,linear.linear_create_comment linear.get_comment linear.linear_get_issue linear.update_comment linear.linear_assign_issue linear.delete_comment linear.linear_update_issue linear.linear_create_issue jira.jira_add_comment linear.create_comment +q094,Move a local file,paraphrase,filesystem.move_file,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,352,1.0,1,673,1.0,filesystem.move_file filesystem.read_text_file filesystem.read_file filesystem.read_media_file filesystem.write_file slack.get_file google_drive.gdrive_update_file slack.update_file figma.figma_get_file gmail.gmail_trash_email +q095,Recursively search files by pattern,paraphrase,filesystem.search_files,1.0,1.0,1.0,1.0000,1.0000,1,1,192,1.0,1,338,1.0,1,623,1.0,filesystem.search_files google_drive.gdrive_search redis.redis_keys linear.linear_search_issues github.search_repositories memory.search_nodes confluence.confluence_search github.search_orgs github.search_commits notion.notion_search +q096,Edit a file in place with old and new string,paraphrase,filesystem.edit_file,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,413,1.0,1,841,1.0,filesystem.edit_file filesystem.move_file filesystem.write_file slack.update_file slack.create_file slack.get_file slack.delete_file google_drive.gdrive_update_file filesystem.read_text_file github.create_or_update_file +q097,Read the whole knowledge graph,paraphrase,memory.read_graph,1.0,1.0,1.0,1.0000,1.0000,1,1,154,1.0,1,258,1.0,1,546,1.0,memory.read_graph memory.delete_relations memory.open_nodes memory.add_observations memory.create_entities memory.search_nodes memory.create_relations memory.delete_entities github.mark_all_notifications_read github.pull_request_read +q098,Search memory nodes,verb_product,memory.search_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,178,1.0,1,311,1.0,1,570,1.0,memory.search_nodes memory.open_nodes github.search_orgs figma.figma_get_file_nodes hubspot.hubspot_search_contacts memory.read_graph github.search_users filesystem.search_files github.search_pull_requests confluence.confluence_search +q099,Show unstaged git diff,verb_product,git.git_diff_unstaged,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,313,1.0,1,588,1.0,git.git_diff_unstaged git.git_diff git.git_diff_staged git.git_show git.git_log git.git_status git.git_reset git.git_commit git.git_add git.git_branch +q100,Stage files with git add,identifier,git.git_add,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,411,1.0,1,789,1.0,git.git_add github.update_gist github.create_gist salesforce.salesforce_list_opportunities github.push_files github.add_comment_to_pending_review git.git_diff_staged git.git_commit git.git_reset git.git_show +q101,Create a Shopify product,verb_product,shopify.shopify_create_product,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,365,1.0,1,702,1.0,shopify.shopify_create_product shopify.shopify_get_product shopify.shopify_update_product shopify.shopify_create_order shopify.shopify_get_order stripe.stripe_create_product shopify.shopify_list_products shopify.shopify_list_customers shopify.shopify_list_orders stripe.stripe_create_price +q102,Cancel a Stripe subscription,verb_product,stripe.stripe_cancel_subscription,1.0,1.0,1.0,1.0000,1.0000,1,1,167,1.0,1,301,1.0,1,610,1.0,stripe.stripe_cancel_subscription stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_list_subscriptions stripe.stripe_create_refund stripe.stripe_list_customers stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_get_payment_intent stripe.stripe_create_product +q103,Create a payment intent,resource,stripe.stripe_create_payment_intent,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,350,1.0,1,739,1.0,stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent stripe.stripe_create_refund stripe.stripe_create_invoice stripe.stripe_create_customer stripe.stripe_create_product stripe.stripe_create_price shopify.shopify_create_order hubspot.hubspot_create_deal gmail.gmail_create_draft +q104,Acknowledge a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,405,1.0,1,702,1.0,pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls sentry.sentry_list_events slack.get_bookmark slack.get_reminder notion.notion_get_page +q105,Create a Datadog monitor,verb_product,datadog.datadog_create_monitor,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,459,1.0,1,760,1.0,datadog.datadog_create_monitor datadog.datadog_get_monitor datadog.datadog_list_monitors datadog.datadog_list_logs datadog.datadog_query_metrics memory.create_entities hubspot.hubspot_create_deal stripe.stripe_create_invoice memory.add_observations vercel.vercel_create_project +q106,Resolve a Sentry issue,paraphrase,sentry.sentry_get_issue,0.0,1.0,1.0,0.5000,0.6309,2,1,274,1.0,1,412,1.0,1,777,1.0,sentry.sentry_get_issue sentry.sentry_update_issue sentry.sentry_list_issues sentry.sentry_list_events pagerduty.pagerduty_resolve_incident sentry.sentry_list_projects jira.jira_get_issue jira.jira_assign_issue github.assign_copilot_to_issue bitbucket.bitbucket_create_issue +q107,Scale a Kubernetes deployment,verb_product,kubernetes.k8s_scale_deployment,1.0,1.0,1.0,1.0000,1.0000,1,1,241,1.0,1,353,1.0,1,726,1.0,kubernetes.k8s_scale_deployment kubernetes.k8s_list_deployments kubernetes.k8s_apply kubernetes.k8s_list_services vercel.vercel_get_deployment kubernetes.k8s_get_pod kubernetes.k8s_pod_logs kubernetes.k8s_get_events kubernetes.k8s_list_pods kubernetes.k8s_delete +q108,Apply a Kubernetes manifest,verb_product,kubernetes.k8s_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,195,1.0,1,334,1.0,1,727,1.0,kubernetes.k8s_apply kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_get_pod terraform.tf_apply kubernetes.k8s_get_events kubernetes.k8s_list_pods kubernetes.k8s_scale_deployment kubernetes.k8s_delete kubernetes.k8s_pod_logs +q109,Stop a docker container,verb_product,docker.docker_stop,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,345,1.0,1,664,1.0,docker.docker_stop docker.docker_rm docker.docker_logs docker.docker_run docker.docker_exec docker.docker_images docker.docker_ps docker.docker_build kubernetes.k8s_pod_logs github.list_environments +q110,Build a docker image,verb_product,docker.docker_build,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,303,1.0,1,642,1.0,docker.docker_build docker.docker_run docker.docker_images docker.docker_stop docker.docker_ps docker.docker_logs docker.docker_exec docker.docker_rm filesystem.read_media_file github.create_environment +q111,Click an element in Playwright,verb_product,playwright.browser_click,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,326,1.0,1,569,1.0,playwright.browser_click playwright.browser_type puppeteer.puppeteer_click playwright.browser_take_screenshot playwright.browser_tab_list playwright.browser_navigate playwright.browser_press_key playwright.browser_tab_close playwright.browser_snapshot clickup.clickup_list_tasks +q112,Type text into a browser input,paraphrase,playwright.browser_type,1.0,1.0,1.0,1.0000,1.0000,1,1,235,1.0,1,336,1.0,1,586,1.0,playwright.browser_type puppeteer.puppeteer_fill playwright.browser_wait_for playwright.browser_navigate playwright.browser_tab_new playwright.browser_tab_list playwright.browser_fill_form playwright.browser_press_key playwright.browser_click playwright.browser_evaluate +q113,List GitLab projects,verb_product,gitlab.gitlab_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,288,1.0,1,640,1.0,gitlab.gitlab_list_projects gitlab.gitlab_list_issues gitlab.gitlab_list_merge_requests vercel.vercel_list_projects jira.jira_list_projects gitlab.gitlab_create_issue github.projects_list gitlab.gitlab_create_pipeline gitlab.gitlab_get_issue sentry.sentry_list_projects +q114,Create a Bitbucket pull request,verb_product,bitbucket.bitbucket_create_pr,1.0,1.0,1.0,1.0000,1.0000,1,1,313,1.0,1,606,1.0,1,1139,1.0,bitbucket.bitbucket_create_pr bitbucket.bitbucket_list_prs bitbucket.bitbucket_create_issue github.pull_request_review_write github.create_pull_request github.pull_request_read github.merge_pull_request github.update_pull_request_branch github.request_copilot_review github.update_pull_request +q115,Create a Gmail draft,verb_product,gmail.gmail_create_draft,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,338,1.0,1,789,1.0,gmail.gmail_create_draft gmail.gmail_list_labels gmail.gmail_get_email gmail.gmail_modify_email gmail.gmail_trash_email gmail.gmail_send_email gmail.gmail_search_emails stripe.stripe_create_customer github.create_pull_request shopify.shopify_create_order +q116,Share a Google Drive file,verb_product,google_drive.gdrive_share_file,1.0,1.0,1.0,1.0000,1.0000,1,1,306,1.0,1,417,1.0,1,705,1.0,google_drive.gdrive_share_file google_drive.gdrive_update_file google_drive.gdrive_create_file google_drive.gdrive_get_file google_drive.gdrive_search google_drive.gdrive_list_folder slack.get_file filesystem.write_file figma.figma_get_file filesystem.get_file_info +q117,List Google Calendar events,verb_product,google_calendar.gcal_list_events,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,427,1.0,1,673,1.0,google_calendar.gcal_list_events google_calendar.gcal_list_calendars google_calendar.gcal_delete_event google_calendar.gcal_create_event google_calendar.gcal_update_event kubernetes.k8s_get_events sentry.sentry_list_events clickup.clickup_list_tasks gmail.gmail_list_labels playwright.browser_tab_list +q118,Delete a calendar event,resource,google_calendar.gcal_delete_event,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,444,1.0,1,690,1.0,google_calendar.gcal_delete_event google_calendar.gcal_create_event github.delete_milestone google_calendar.gcal_update_event google_calendar.gcal_list_events gitlab.delete_milestone redis.redis_delete google_calendar.gcal_list_calendars slack.delete_reminder linear.delete_cycle +q119,List Vercel projects,verb_product,vercel.vercel_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,151,1.0,1,260,1.0,1,539,1.0,vercel.vercel_list_projects vercel.vercel_list_deployments jira.jira_list_projects vercel.vercel_create_project sentry.sentry_list_projects linear.linear_list_projects vercel.vercel_get_deployment gitlab.gitlab_list_projects github.projects_list kubernetes.k8s_list_deployments +q120,Select rows from Supabase,verb_product,supabase.supabase_select,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,354,1.0,1,736,1.0,supabase.supabase_select supabase.supabase_insert supabase.supabase_update supabase.supabase_list_tables supabase.supabase_invoke_fn datadog.datadog_list_logs datadog.datadog_query_metrics figma.figma_get_file_nodes memory.delete_observations sqlite.sqlite_query +q121,Post a Figma comment,verb_product,figma.figma_post_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,308,1.0,1,683,1.0,figma.figma_post_comment figma.figma_get_file figma.figma_get_comments linear.get_comment notion.get_comment figma.figma_get_images linear.create_comment linear.update_comment linear.delete_comment figma.figma_get_file_nodes +q122,List Okta users,verb_product,okta.okta_list_users,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,271,1.0,1,606,1.0,okta.okta_list_users okta.okta_create_user okta.okta_get_user okta.okta_deactivate_user notion.notion_list_users notion.list_users_tokens slack.slack_list_users slack.slack_get_users linear.list_users github.search_users +q123,Range query Prometheus,verb_product,prometheus.prom_query_range,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,379,1.0,1,680,1.0,prometheus.prom_query_range prometheus.prom_query prometheus.prom_list_metrics datadog.datadog_query_metrics memory.search_nodes datadog.datadog_list_monitors salesforce.salesforce_query postgres.query confluence.confluence_search shopify.shopify_list_products +q124,Create a Grafana annotation,verb_product,grafana.grafana_create_annotation,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,375,1.0,1,788,1.0,grafana.grafana_create_annotation grafana.grafana_search_dashboards grafana.grafana_get_dashboard trello.trello_create_board github.create_artifact gitlab.create_snippet trello.trello_create_card stripe.stripe_create_product linear.linear_create_project salesforce.salesforce_create_record +q125,Apply terraform,identifier,terraform.tf_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,303,1.0,1,653,1.0,terraform.tf_apply terraform.tf_plan kubernetes.k8s_apply terraform.tf_state_list github.update_environment cloudflare.cf_create_dns cloudflare.cf_purge_cache cloudflare.cf_list_zones playwright.browser_fill_form hubspot.hubspot_create_deal +q126,List Asana tasks,verb_product,asana.asana_list_tasks,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,368,1.0,1,635,1.0,asana.asana_list_tasks asana.asana_get_task asana.asana_create_task clickup.clickup_list_tasks asana.asana_create_project asana.asana_update_task okta.okta_list_users jira.jira_list_projects aws_s3.s3_list_buckets salesforce.salesforce_list_opportunities +q127,Comment on a Trello card,verb_product,trello.trello_add_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,368,1.0,1,721,1.0,trello.trello_add_comment trello.trello_create_card trello.trello_list_cards trello.trello_create_board linear.get_comment figma.figma_post_comment notion.get_comment linear.update_comment linear.create_comment notion.update_comment +q128,Update a ClickUp task,verb_product,clickup.clickup_update_task,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,399,1.0,1,732,1.0,clickup.clickup_update_task clickup.clickup_get_task clickup.clickup_list_tasks clickup.clickup_create_task asana.asana_update_task puppeteer.puppeteer_click playwright.browser_click shopify.shopify_update_product gitlab.update_snippet github.update_milestone +q129,Search Confluence,verb_product,confluence.confluence_search,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,371,1.0,1,636,1.0,confluence.confluence_search confluence.confluence_get_page confluence.confluence_update_page confluence.confluence_create_page grafana.grafana_search_dashboards github.search_orgs github.search_users hubspot.hubspot_search_contacts github.search_pull_requests github.search_commits +q130,Create a HubSpot deal,verb_product,hubspot.hubspot_create_deal,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,772,1.0,hubspot.hubspot_create_deal hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_list_contacts hubspot.hubspot_search_contacts stripe.stripe_create_refund github.create_pull_request stripe.stripe_create_invoice slack.slack_create_channel shopify.shopify_create_order +q131,List Zendesk tickets,verb_product,zendesk.zendesk_list_tickets,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,367,1.0,1,565,1.0,zendesk.zendesk_list_tickets zendesk.zendesk_search zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_create_ticket google_calendar.gcal_list_calendars salesforce.salesforce_list_opportunities vercel.vercel_list_projects jira.jira_list_projects playwright.browser_tab_list +q132,List Discord channels,verb_product,discord.discord_list_channels,1.0,1.0,1.0,1.0000,1.0000,1,1,166,1.0,1,300,1.0,1,562,1.0,discord.discord_list_channels discord.discord_list_members discord.discord_get_channel slack.slack_list_channels discord.discord_send_message discord.discord_add_reaction okta.okta_list_users gmail.gmail_list_labels google_calendar.gcal_list_calendars twilio.twilio_list_messages +q133,Set a Redis key,verb_product,redis.redis_set,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,280,1.0,1,642,1.0,redis.redis_set redis.redis_get redis.redis_delete redis.redis_keys playwright.browser_press_key jira.jira_update_issue aws_s3.s3_put_object jira.jira_transition_issue slack.slack_set_channel_topic slack.update_pin +q134,Insert MongoDB documents,verb_product,mongodb.mongo_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,256,1.0,1,399,1.0,1,759,1.0,mongodb.mongo_insert mongodb.mongo_update mongodb.mongo_delete mongodb.mongo_find supabase.supabase_insert mongodb.mongo_list_collections aws_s3.s3_put_object salesforce.salesforce_create_record elasticsearch.es_index_doc elasticsearch.es_get_doc +q135,Run a MySQL query,verb_product,mysql.mysql_query,1.0,1.0,1.0,1.0000,1.0000,1,1,179,1.0,1,294,1.0,1,662,1.0,mysql.mysql_query postgres.query sqlite.sqlite_query postgres.explain_query salesforce.salesforce_query mysql.mysql_list_tables prometheus.prom_query datadog.datadog_query_metrics prometheus.prom_query_range github.update_check_run +q136,Index a document in Elasticsearch,verb_product,elasticsearch.es_index_doc,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,367,1.0,1,637,1.0,elasticsearch.es_index_doc elasticsearch.es_get_doc elasticsearch.es_search playwright.browser_tab_close mongodb.mongo_find github.get_milestone confluence.confluence_search github.delete_milestone google_drive.gdrive_search github.get_hook +q137,Get an S3 object,verb_product,aws_s3.s3_get_object,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,366,1.0,1,640,1.0,aws_s3.s3_get_object aws_s3.s3_delete_object aws_s3.s3_put_object aws_s3.s3_list_objects redis.redis_get asana.asana_get_task okta.okta_get_user gitlab.get_snippet shopify.shopify_get_product shopify.shopify_get_order +q138,List Cloudflare zones,verb_product,cloudflare.cf_list_zones,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,326,1.0,1,558,1.0,cloudflare.cf_list_zones cloudflare.cf_list_dns cloudflare.cf_purge_cache cloudflare.cf_create_dns aws_s3.s3_list_buckets kubernetes.k8s_list_services kubernetes.k8s_list_deployments linear.linear_list_teams salesforce.salesforce_list_opportunities google_calendar.gcal_list_calendars +q139,Create a Stripe invoice,verb_product,stripe.stripe_create_invoice,1.0,1.0,1.0,1.0000,1.0000,1,1,185,1.0,1,340,1.0,1,662,1.0,stripe.stripe_create_invoice stripe.stripe_create_customer stripe.stripe_list_invoices stripe.stripe_create_product stripe.stripe_create_payment_intent stripe.stripe_create_refund stripe.stripe_create_price stripe.stripe_get_customer stripe.stripe_get_payment_intent stripe.stripe_list_subscriptions +q140,List GitHub Actions workflows,verb_product,github.actions_list,1.0,1.0,1.0,1.0000,1.0000,1,1,293,1.0,1,438,1.0,1,845,1.0,github.actions_list github.actions_get github.actions_run_trigger github.actions_get_job_logs github.list_gists github.list_notifications github.list_issues github.list_code_scanning_alerts github.list_releases github.list_tags +q141,Get GitHub Actions job logs,verb_product,github.actions_get_job_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,450,1.0,1,804,1.0,github.actions_get_job_logs github.actions_get github.actions_list github.actions_run_trigger git.git_log github.get_notification_details github.get_code_scanning_alert github.get_secret_scanning_alert gitlab.get_job github.search_pull_requests +q142,Create a GitHub repository,verb_product,github.create_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,333,1.0,1,556,1.0,1,915,1.0,github.create_repository github.create_branch github.create_issue github.create_or_update_file github.create_gist github.star_repository github.unstar_repository github.fork_repository github.delete_repository github.list_releases +q143,Checkout a git branch,verb_product,git.git_checkout,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,373,1.0,1,759,1.0,git.git_checkout git.git_branch git.git_create_branch github.create_branch git.git_show github.update_pull_request_branch git.git_status git.git_diff github.list_commits github.list_branches +q144,Show git log,identifier,git.git_log,1.0,1.0,1.0,1.0000,1.0000,1,1,194,1.0,1,312,1.0,1,600,1.0,git.git_log git.git_show git.git_status git.git_diff_unstaged git.git_diff git.git_branch git.git_commit git.git_diff_staged git.git_checkout git.git_add +q145,List postgres schemas,verb_product,postgres.list_schemas,1.0,1.0,1.0,1.0000,1.0000,1,1,169,1.0,1,248,1.0,1,491,1.0,postgres.list_schemas postgres.list_tables postgres.describe_table aws_s3.s3_list_buckets vercel.vercel_list_projects jira.jira_list_projects sentry.sentry_list_projects mysql.mysql_list_tables postgres.explain_query kubernetes.k8s_list_deployments +q146,Open nodes in the knowledge graph,verb_product,memory.open_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,163,1.0,1,262,1.0,1,531,1.0,memory.open_nodes memory.search_nodes memory.read_graph memory.create_entities memory.create_relations memory.delete_relations memory.add_observations memory.delete_entities figma.figma_get_file_nodes playwright.browser_tab_list +q147,Get file metadata from the filesystem,paraphrase,filesystem.get_file_info,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,334,1.0,1,713,1.0,filesystem.get_file_info filesystem.read_file google_drive.gdrive_get_file filesystem.read_media_file filesystem.read_text_file figma.figma_get_file_nodes github.get_file_contents figma.figma_get_images figma.figma_get_file filesystem.list_directory +q148,Write a new local file,paraphrase,filesystem.write_file,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,384,1.0,1,775,1.0,filesystem.write_file filesystem.edit_file filesystem.read_text_file filesystem.read_file filesystem.read_media_file slack.create_file playwright.browser_tab_new filesystem.create_directory github.issue_write slack.get_file +q149,Search GitHub issues,verb_product,github.search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,363,1.0,1,702,1.0,github.search_issues github.search_users github.search_orgs github.list_issues github.search_pull_requests github.search_commits github.search_repositories github.search_code linear.linear_search_issues jira.jira_search_issues +q150,Get the authenticated GitHub user,paraphrase,github.get_teams,0.0,1.0,1.0,0.5000,0.6309,2,1,179,1.0,1,301,1.0,1,656,1.0,github.get_teams github.get_me github.list_notifications github.get_latest_release github.list_gists slack.slack_get_user_profile github.projects_get github.get_label github.get_notification_details github.get_secret_scanning_alert diff --git a/experiments/mcp-tool-search/results/hybrid_rrf_bm25_embed.json b/experiments/mcp-tool-search/results/hybrid_rrf_bm25_embed.json new file mode 100644 index 00000000..8c95e490 --- /dev/null +++ b/experiments/mcp-tool-search/results/hybrid_rrf_bm25_embed.json @@ -0,0 +1,7013 @@ +{ + "summary": { + "n": 150, + "index_s": 1.9607068370000889, + "rss_mb": 0.0, + "p50_ms": 2.387484999985645, + "p95_ms": 3.310145450012669, + "recall@1": 0.9333333333333333, + "recall@1_lo": 0.8933333333333333, + "recall@1_hi": 0.9733333333333334, + "recall@3": 0.9866666666666667, + "recall@3_lo": 0.9666666666666667, + "recall@3_hi": 1.0, + "recall@5": 0.9933333333333333, + "recall@5_lo": 0.98, + "recall@5_hi": 1.0, + "mrr@10": 0.9594444444444444, + "mrr@10_lo": 0.9333333333333333, + "mrr@10_hi": 0.9833333333333333, + "ndcg@10": 0.9671705986683319, + "ndcg@10_lo": 0.9453740714219765, + "ndcg@10_hi": 0.9864568018198906 + }, + "meta": { + "name": "hybrid_rrf_bm25_embed", + "needs_model": true, + "needs_api": false, + "notes": "RRF BM25 + bge-small" + }, + "rows": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "github.issue_write", + "gitlab.gitlab_create_issue", + "github.add_issue_comment", + "github.issue_read", + "github.assign_copilot_to_issue", + "github.create_repository", + "github.create_gist", + "github.update_issue_comment", + "bitbucket.bitbucket_create_issue" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 380, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 584, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1042, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "kind": "verb_product", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_create_issue", + "jira.create_sprint", + "jira.create_board", + "jira.jira_create_sprint", + "jira.create_version", + "jira.create_filter", + "jira.create_webhook", + "zendesk.zendesk_create_ticket", + "jira.get_sprint", + "zendesk.zendesk_get_ticket" + ], + "hit1": "jira.jira_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 330, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 535, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 929, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "kind": "verb_product", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "ranking": [ + "slack.slack_post_message", + "slack.slack_add_reaction", + "slack.slack_create_channel", + "slack.get_reminder", + "discord.discord_send_message", + "slack.slack_upload_file", + "slack.slack_leave_channel", + "slack.slack_join_channel", + "slack.slack_search_messages", + "slack.slack_reply_to_thread" + ], + "hit1": "slack.slack_post_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 374, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 719, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "kind": "resource", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "ranking": [ + "github.list_pull_requests", + "github.search_pull_requests", + "bitbucket.bitbucket_list_prs", + "github.create_pull_request", + "github.pull_request_read", + "gitlab.gitlab_list_merge_requests", + "github.pull_request_review_write", + "github.update_pull_request", + "playwright.browser_tab_list", + "github.merge_pull_request" + ], + "hit1": "github.list_pull_requests", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 477, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 976, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "ranking": [ + "github.create_pull_request", + "github.pull_request_read", + "github.pull_request_review_write", + "github.search_pull_requests", + "github.merge_pull_request", + "github.request_copilot_review", + "github.list_pull_requests", + "github.update_pull_request", + "github.update_pull_request_branch", + "github.add_comment_to_pending_review" + ], + "hit1": "github.create_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 406, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 564, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1144, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "kind": "verb_product", + "gold": [ + "postgres.query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "mysql.mysql_query", + "sqlite.sqlite_query", + "postgres.explain_query", + "salesforce.salesforce_query", + "postgres.list_schemas", + "postgres.describe_table", + "postgres.list_tables", + "prometheus.prom_query", + "prometheus.prom_query_range" + ], + "hit1": "postgres.query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 596, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "kind": "paraphrase", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "ranking": [ + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "figma.figma_get_file_nodes", + "github.get_file_contents", + "figma.figma_get_file", + "gitlab.gitlab_get_file", + "figma.figma_get_images", + "filesystem.get_file_info", + "slack.get_file" + ], + "hit1": "filesystem.read_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 406, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 747, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "kind": "paraphrase", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "ranking": [ + "google_drive.gdrive_list_folder", + "filesystem.list_directory", + "filesystem.list_directory_with_sizes", + "filesystem.search_files", + "filesystem.directory_tree", + "git.git_add", + "filesystem.move_file", + "github.create_gist", + "google_drive.gdrive_search", + "slack.list_files" + ], + "hit1": "google_drive.gdrive_list_folder", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6862856989769305, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 191, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 648, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "ranking": [ + "memory.read_graph", + "memory.create_relations", + "memory.add_observations", + "memory.create_entities", + "memory.delete_relations", + "memory.open_nodes", + "memory.delete_entities", + "memory.search_nodes", + "postgres.list_schemas", + "sequential_thinking.sequentialthinking" + ], + "hit1": "memory.read_graph", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.543791241910304, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 164, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 261, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 568, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "kind": "verb_product", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "ranking": [ + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "confluence.confluence_search", + "grafana.grafana_search_dashboards", + "github.search_orgs", + "memory.search_nodes", + "hubspot.hubspot_search_contacts", + "slack.slack_search_messages", + "github.search_users", + "github.search_pull_requests" + ], + "hit1": "brave_search.brave_web_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 351, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 637, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "kind": "verb_product", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "ranking": [ + "fetch.fetch", + "playwright.browser_navigate", + "puppeteer.puppeteer_navigate", + "filesystem.read_file", + "github.get_file_contents", + "playwright.browser_take_screenshot", + "docker.docker_logs", + "playwright.browser_tab_new", + "filesystem.list_directory", + "twilio.twilio_make_call" + ], + "hit1": "fetch.fetch", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 372, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 700, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "kind": "identifier", + "gold": [ + "git.git_status" + ], + "alts": [], + "ranking": [ + "git.git_status", + "git.git_show", + "git.git_diff_unstaged", + "git.git_log", + "git.git_diff", + "git.git_branch", + "git.git_diff_staged", + "github.list_milestones", + "git.git_commit", + "git.git_checkout" + ], + "hit1": "git.git_status", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 616, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "kind": "paraphrase", + "gold": [ + "git.git_commit" + ], + "alts": [], + "ranking": [ + "git.git_commit", + "git.git_reset", + "git.git_diff_staged", + "git.git_show", + "github.get_commit", + "github.update_milestone", + "filesystem.move_file", + "git.git_log", + "github.push_files", + "slack.update_reminder" + ], + "hit1": "git.git_commit", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_comment", + "linear.linear_create_issue", + "linear.linear_get_issue", + "linear.linear_update_issue", + "linear.linear_assign_issue", + "linear.linear_create_project", + "linear.create_label", + "linear.create_cycle", + "linear.create_attachment", + "linear.create_comment" + ], + "hit1": "linear.linear_create_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 275, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 925, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "kind": "paraphrase", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_project", + "linear.linear_list_projects", + "asana.asana_create_task", + "linear.linear_create_issue", + "linear.get_cycle", + "linear.get_label", + "clickup.clickup_update_task", + "linear.get_attachment", + "linear.create_label", + "linear.list_labels" + ], + "hit1": "linear.linear_create_project", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 515, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 440, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 801, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "ranking": [ + "github.add_issue_comment", + "github.update_issue_comment", + "jira.jira_add_comment", + "github.assign_copilot_to_issue", + "github.issue_write", + "figma.figma_post_comment", + "github.issue_read", + "github.add_comment_to_pending_review", + "github.get_discussion_comments", + "linear.linear_create_comment" + ], + "hit1": "github.add_issue_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 302, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 542, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1028, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "kind": "ambiguous", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "ranking": [ + "github.merge_pull_request", + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_get_merge_request", + "github.update_pull_request_branch", + "gitlab.gitlab_create_merge_request", + "github.pull_request_read", + "github.create_pull_request", + "github.update_pull_request", + "gitlab.gitlab_list_merge_requests", + "github.pull_request_review_write" + ], + "hit1": "github.merge_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 495, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1093, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "kind": "verb_product", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "ranking": [ + "github.actions_run_trigger", + "github.actions_get", + "github.actions_list", + "github.actions_get_job_logs", + "circleci.circleci_trigger_pipeline", + "github.search_commits", + "github.dismiss_notification", + "github.update_gist", + "github.mark_all_notifications_read", + "github.search_pull_requests" + ], + "hit1": "github.actions_run_trigger", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 734, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "kind": "resource", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "ranking": [ + "github.list_dependabot_alerts", + "github.get_dependabot_alert", + "github.list_secret_scanning_alerts", + "github.list_code_scanning_alerts", + "vercel.vercel_get_deployment", + "slack.get_reminder", + "github.get_milestone", + "github.get_hook", + "gitlab.get_release", + "pagerduty.pagerduty_get_incident" + ], + "hit1": "github.list_dependabot_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "kind": "verb_product", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_issues", + "sentry.sentry_list_projects", + "sentry.sentry_list_events", + "sentry.sentry_get_issue", + "linear.linear_list_issues", + "sentry.sentry_update_issue", + "gitlab.gitlab_list_issues", + "github.list_issues", + "stripe.stripe_list_customers", + "hubspot.hubspot_list_contacts" + ], + "hit1": "sentry.sentry_list_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 393, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 772, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_customer", + "stripe.stripe_create_invoice", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_customer", + "stripe.stripe_list_invoices", + "stripe.stripe_create_product", + "stripe.stripe_list_subscriptions", + "stripe.stripe_create_price", + "stripe.stripe_create_refund", + "stripe.stripe_list_customers" + ], + "hit1": "stripe.stripe_create_customer", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "kind": "paraphrase", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_refund", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_invoice", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_customer", + "stripe.stripe_list_invoices", + "stripe.stripe_cancel_subscription", + "stripe.stripe_create_customer", + "stripe.stripe_create_product", + "stripe.stripe_list_customers" + ], + "hit1": "stripe.stripe_create_refund", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 332, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "kind": "verb_product", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "ranking": [ + "gmail.gmail_send_email", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "gmail.gmail_modify_email", + "gmail.gmail_create_draft", + "gmail.gmail_list_labels", + "twilio.twilio_send_sms", + "gmail.gmail_search_emails", + "discord.discord_send_message", + "google_drive.gdrive_share_file" + ], + "hit1": "gmail.gmail_send_email", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 743, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "kind": "verb_product", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "ranking": [ + "gmail.gmail_search_emails", + "gmail.gmail_list_labels", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "gmail.gmail_create_draft", + "gmail.gmail_modify_email", + "gmail.gmail_send_email", + "hubspot.hubspot_search_contacts", + "github.search_users", + "slack.slack_search_messages" + ], + "hit1": "gmail.gmail_search_emails", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 678, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "kind": "paraphrase", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_update_file", + "google_drive.gdrive_share_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_search", + "slack.slack_upload_file", + "google_drive.gdrive_list_folder", + "aws_s3.s3_put_object", + "filesystem.edit_file", + "filesystem.write_file" + ], + "hit1": "google_drive.gdrive_update_file", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 307, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 806, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_create_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "trello.trello_create_board", + "grafana.grafana_create_annotation", + "shopify.shopify_create_order", + "gmail.gmail_create_draft", + "github.create_milestone" + ], + "hit1": "google_calendar.gcal_create_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 824, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_list_pods", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_apply", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_delete", + "aws_s3.s3_list_buckets" + ], + "hit1": "kubernetes.k8s_list_pods", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_get_pod", + "docker.docker_logs", + "github.actions_get_job_logs", + "datadog.datadog_list_logs", + "git.git_log", + "slack.slack_get_channel_history", + "memory.delete_observations", + "redis.redis_get", + "github.get_runner" + ], + "hit1": "kubernetes.k8s_pod_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "ranking": [ + "docker.docker_run", + "docker.docker_exec", + "docker.docker_stop", + "docker.docker_ps", + "docker.docker_logs", + "docker.docker_rm", + "docker.docker_images", + "docker.docker_build", + "github.get_check_run", + "terraform.tf_plan" + ], + "hit1": "docker.docker_run", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 587, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "kind": "verb_product", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "ranking": [ + "docker.docker_logs", + "kubernetes.k8s_pod_logs", + "docker.docker_stop", + "docker.docker_run", + "docker.docker_images", + "docker.docker_exec", + "git.git_log", + "docker.docker_ps", + "docker.docker_rm", + "datadog.datadog_list_logs" + ], + "hit1": "docker.docker_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 683, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "kind": "ambiguous", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "ranking": [ + "playwright.browser_navigate", + "puppeteer.puppeteer_navigate", + "playwright.browser_tab_new", + "fetch.fetch", + "playwright.browser_press_key", + "playwright.browser_click", + "playwright.browser_evaluate", + "twilio.twilio_make_call", + "playwright.browser_snapshot", + "playwright.browser_take_screenshot" + ], + "hit1": "playwright.browser_navigate", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 309, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 624, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "kind": "ambiguous", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "ranking": [ + "playwright.browser_take_screenshot", + "puppeteer.puppeteer_screenshot", + "playwright.browser_snapshot", + "playwright.browser_evaluate", + "playwright.browser_press_key", + "playwright.browser_click", + "notion.notion_get_page", + "confluence.confluence_get_page", + "confluence.confluence_create_page", + "playwright.browser_navigate" + ], + "hit1": "playwright.browser_take_screenshot", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 629, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_create_merge_request", + "gitlab.gitlab_get_merge_request", + "github.merge_pull_request", + "gitlab.gitlab_list_merge_requests", + "gitlab.gitlab_create_pipeline", + "gitlab.gitlab_create_issue", + "github.create_pull_request", + "gitlab.gitlab_list_projects", + "gitlab.gitlab_get_file" + ], + "hit1": "gitlab.gitlab_merge_merge_request", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 929, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "kind": "verb_product", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "ranking": [ + "notion.notion_search", + "notion.notion_get_page", + "notion.notion_list_users", + "notion.notion_query_database", + "notion.notion_update_page", + "notion.notion_get_block", + "notion.notion_create_database", + "notion.notion_get_comments", + "notion.notion_create_page", + "github.search_orgs" + ], + "hit1": "notion.notion_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "kind": "verb_product", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "ranking": [ + "notion.notion_create_page", + "notion.notion_get_page", + "notion.notion_update_page", + "notion.notion_create_database", + "notion.notion_get_comments", + "notion.create_data_source", + "notion.create_comment", + "notion.create_users_token", + "notion.notion_search", + "notion.notion_list_users" + ], + "hit1": "notion.notion_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "kind": "verb_product", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "ranking": [ + "notion.notion_query_database", + "notion.notion_create_database", + "notion.notion_search", + "notion.notion_create_page", + "notion.notion_list_users", + "notion.notion_get_page", + "notion.notion_get_block", + "notion.notion_get_comments", + "notion.get_data_source", + "notion.get_comment" + ], + "hit1": "notion.notion_query_database", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 656, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_search", + "jira.jira_create_issue", + "vercel.vercel_create_project", + "gmail.gmail_create_draft", + "hubspot.hubspot_create_deal", + "stripe.stripe_create_refund" + ], + "hit1": "zendesk.zendesk_create_ticket", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 782, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_create_deal", + "hubspot.hubspot_search_contacts", + "hubspot.hubspot_list_contacts", + "slack.slack_create_channel", + "github.create_pull_request", + "github.pull_request_review_write", + "shopify.shopify_create_order", + "stripe.stripe_create_customer" + ], + "hit1": "hubspot.hubspot_create_contact", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 866, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "kind": "verb_product", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_convert_lead", + "salesforce.salesforce_create_lead", + "salesforce.salesforce_create_record", + "salesforce.salesforce_list_opportunities", + "salesforce.salesforce_get_record", + "salesforce.salesforce_query", + "salesforce.salesforce_update_record", + "hubspot.hubspot_create_deal", + "linear.linear_create_project", + "github.create_hook" + ], + "hit1": "salesforce.salesforce_convert_lead", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 798, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "kind": "identifier", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_query", + "mysql.mysql_query", + "sqlite.sqlite_query", + "postgres.query", + "prometheus.prom_query", + "prometheus.prom_query_range", + "postgres.explain_query", + "datadog.datadog_query_metrics", + "notion.notion_query_database", + "datadog.datadog_list_monitors" + ], + "hit1": "salesforce.salesforce_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "kind": "verb_product", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "ranking": [ + "shopify.shopify_list_orders", + "shopify.shopify_list_customers", + "shopify.shopify_get_order", + "shopify.shopify_list_products", + "shopify.shopify_create_order", + "shopify.shopify_get_product", + "shopify.shopify_update_product", + "shopify.shopify_create_product", + "clickup.clickup_list_tasks", + "aws_s3.s3_list_buckets" + ], + "hit1": "shopify.shopify_list_orders", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "shopify.shopify_create_order", + "gmail.gmail_create_draft", + "confluence.confluence_create_page", + "datadog.datadog_create_monitor" + ], + "hit1": "pagerduty.pagerduty_create_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 821, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "kind": "paraphrase", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_list_oncalls", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "twilio.twilio_make_call", + "playwright.browser_press_key", + "playwright.browser_click", + "gitlab.get_snippet" + ], + "hit1": "pagerduty.pagerduty_list_oncalls", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "kind": "verb_product", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "ranking": [ + "datadog.datadog_query_metrics", + "datadog.datadog_list_monitors", + "datadog.datadog_create_monitor", + "datadog.datadog_list_logs", + "datadog.datadog_get_monitor", + "prometheus.prom_list_metrics", + "memory.search_nodes", + "prometheus.prom_query", + "prometheus.prom_query_range", + "slack.slack_search_messages" + ], + "hit1": "datadog.datadog_query_metrics", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 785, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "kind": "verb_product", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "ranking": [ + "datadog.datadog_list_logs", + "datadog.datadog_list_monitors", + "datadog.datadog_get_monitor", + "datadog.datadog_create_monitor", + "datadog.datadog_query_metrics", + "docker.docker_logs", + "github.search_users", + "github.search_orgs", + "github.actions_get_job_logs", + "slack.slack_search_messages" + ], + "hit1": "datadog.datadog_list_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 785, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_list_buckets", + "jira.jira_list_projects", + "gmail.gmail_list_labels", + "kubernetes.k8s_list_services", + "bitbucket.bitbucket_list_repos", + "salesforce.salesforce_list_opportunities", + "cloudflare.cf_list_zones", + "clickup.clickup_list_tasks", + "okta.okta_list_users", + "kubernetes.k8s_list_deployments" + ], + "hit1": "aws_s3.s3_list_buckets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 128, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 233, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 482, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "kind": "paraphrase", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_put_object", + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "slack.slack_upload_file", + "twilio.twilio_send_sms", + "aws_s3.s3_list_objects", + "twilio.twilio_make_call", + "filesystem.write_file", + "gmail.gmail_create_draft", + "github.assign_copilot_to_issue" + ], + "hit1": "aws_s3.s3_put_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 420, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 835, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "cloudflare.cf_list_dns", + "cloudflare.cf_create_dns", + "slack.delete_bookmark", + "slack.delete_canvase", + "memory.delete_observations", + "redis.redis_delete", + "aws_s3.s3_delete_object", + "linear.delete_attachment" + ], + "hit1": "cloudflare.cf_purge_cache", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 618, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "salesforce.salesforce_create_record", + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "salesforce.salesforce_get_record", + "salesforce.salesforce_update_record", + "datadog.datadog_create_monitor", + "hubspot.hubspot_create_deal", + "confluence.confluence_create_page" + ], + "hit1": "cloudflare.cf_create_dns", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 819, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_deployments", + "vercel.vercel_list_projects", + "vercel.vercel_get_deployment", + "kubernetes.k8s_list_deployments", + "vercel.vercel_create_project", + "kubernetes.k8s_list_services", + "jira.jira_list_projects", + "sentry.sentry_list_projects", + "okta.okta_list_users", + "datadog.datadog_list_monitors" + ], + "hit1": "vercel.vercel_list_deployments", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 542, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "ranking": [ + "supabase.supabase_insert", + "supabase.supabase_select", + "mongodb.mongo_insert", + "supabase.supabase_update", + "supabase.supabase_list_tables", + "supabase.supabase_invoke_fn", + "playwright.browser_type", + "linear.update_cycle", + "linear.update_attachment", + "linear.update_user" + ], + "hit1": "supabase.supabase_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 701, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "kind": "paraphrase", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "ranking": [ + "figma.figma_get_images", + "figma.figma_get_file_nodes", + "figma.figma_get_file", + "docker.docker_images", + "figma.figma_get_comments", + "figma.figma_post_comment", + "filesystem.read_media_file", + "github.get_file_contents", + "memory.delete_observations", + "linear.update_attachment" + ], + "hit1": "figma.figma_get_images", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 685, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "kind": "verb_product", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "ranking": [ + "twilio.twilio_send_sms", + "twilio.twilio_list_messages", + "twilio.twilio_make_call", + "gmail.gmail_send_email", + "discord.discord_send_message", + "slack.slack_add_reaction", + "aws_s3.s3_put_object", + "playwright.browser_type", + "slack.slack_reply_to_thread", + "puppeteer.puppeteer_click" + ], + "hit1": "twilio.twilio_send_sms", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 806, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "kind": "verb_product", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "ranking": [ + "okta.okta_create_user", + "okta.okta_get_user", + "okta.okta_deactivate_user", + "okta.okta_list_users", + "linear.create_user", + "asana.asana_create_project", + "asana.asana_create_task", + "linear.get_user", + "linear.update_user", + "slack.slack_get_user_profile" + ], + "hit1": "okta.okta_create_user", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 676, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query", + "prometheus.prom_query_range", + "prometheus.prom_list_metrics", + "datadog.datadog_query_metrics", + "memory.search_nodes", + "salesforce.salesforce_query", + "postgres.query", + "sqlite.sqlite_query", + "datadog.datadog_list_monitors", + "confluence.confluence_search" + ], + "hit1": "prometheus.prom_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 661, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "kind": "verb_product", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "ranking": [ + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "grafana.grafana_create_annotation", + "github.search_orgs", + "github.search_users", + "confluence.confluence_search", + "github.search_pull_requests", + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "github.search_issues" + ], + "hit1": "grafana.grafana_search_dashboards", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 677, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "kind": "verb_product", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "ranking": [ + "circleci.circleci_trigger_pipeline", + "circleci.circleci_get_pipeline", + "circleci.circleci_list_pipelines", + "github.actions_run_trigger", + "gitlab.gitlab_create_pipeline", + "gitlab.update_pipeline", + "hubspot.hubspot_create_deal", + "gitlab.get_pipeline", + "gitlab.delete_pipeline", + "gitlab.create_pipeline" + ], + "hit1": "circleci.circleci_trigger_pipeline", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "kind": "identifier", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "ranking": [ + "terraform.tf_plan", + "terraform.tf_apply", + "terraform.tf_state_list", + "github.update_check_run", + "docker.docker_run", + "github.get_check_run", + "github.create_check_run", + "github.actions_run_trigger", + "docker.docker_exec", + "github.actions_get" + ], + "hit1": "terraform.tf_plan", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "kind": "verb_product", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "ranking": [ + "asana.asana_create_task", + "asana.asana_get_task", + "asana.asana_create_project", + "asana.asana_update_task", + "asana.asana_list_tasks", + "clickup.clickup_create_task", + "okta.okta_create_user", + "github.create_issue", + "notion.notion_create_database", + "clickup.clickup_get_task" + ], + "hit1": "asana.asana_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 820, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "ranking": [ + "trello.trello_create_card", + "trello.trello_add_comment", + "trello.trello_create_board", + "trello.trello_list_cards", + "stripe.stripe_create_invoice", + "stripe.stripe_create_customer", + "stripe.stripe_create_product", + "grafana.grafana_create_annotation", + "stripe.stripe_create_refund", + "stripe.stripe_create_price" + ], + "hit1": "trello.trello_create_card", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 734, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_create_task", + "clickup.clickup_get_task", + "clickup.clickup_list_tasks", + "clickup.clickup_update_task", + "puppeteer.puppeteer_click", + "asana.asana_create_task", + "playwright.browser_click", + "gmail.gmail_create_draft", + "asana.asana_get_task", + "shopify.shopify_create_order" + ], + "hit1": "clickup.clickup_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 734, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "kind": "verb_product", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "ranking": [ + "confluence.confluence_create_page", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "confluence.confluence_search", + "notion.notion_create_page", + "github.search_code", + "notion.notion_get_page", + "playwright.browser_snapshot", + "shopify.shopify_create_product", + "github.create_artifact" + ], + "hit1": "confluence.confluence_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "kind": "verb_product", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "discord.discord_add_reaction", + "discord.discord_get_channel", + "slack.slack_post_message", + "discord.discord_list_channels", + "twilio.twilio_send_sms", + "gmail.gmail_trash_email", + "discord.discord_list_members", + "gmail.gmail_send_email", + "slack.slack_add_reaction" + ], + "hit1": "discord.discord_send_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_search", + "confluence.confluence_search", + "elasticsearch.es_get_doc", + "github.search_orgs", + "github.search_users", + "elasticsearch.es_index_doc", + "hubspot.hubspot_search_contacts", + "grafana.grafana_search_dashboards", + "brave_search.brave_web_search", + "github.search_pull_requests" + ], + "hit1": "elasticsearch.es_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 310, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 653, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "redis.redis_set", + "redis.redis_delete", + "redis.redis_keys", + "playwright.browser_press_key", + "figma.figma_get_file", + "jira.jira_get_issue", + "aws_s3.s3_get_object", + "github.get_milestone", + "gitlab.get_pipeline" + ], + "hit1": "redis.redis_get", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 566, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_find", + "mongodb.mongo_delete", + "mongodb.mongo_insert", + "mongodb.mongo_list_collections", + "mongodb.mongo_update", + "elasticsearch.es_get_doc", + "filesystem.search_files", + "confluence.confluence_search", + "datadog.datadog_list_logs", + "hubspot.hubspot_search_contacts" + ], + "hit1": "mongodb.mongo_find", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "kind": "verb_product", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "ranking": [ + "sqlite.sqlite_list_tables", + "mysql.mysql_list_tables", + "sqlite.sqlite_describe_table", + "supabase.supabase_list_tables", + "postgres.list_tables", + "okta.okta_list_users", + "trello.trello_list_cards", + "datadog.datadog_list_monitors", + "jira.jira_list_projects", + "clickup.clickup_list_tasks" + ], + "hit1": "sqlite.sqlite_list_tables", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 240, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 517, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "kind": "verb_product", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "ranking": [ + "postgres.describe_table", + "sqlite.sqlite_describe_table", + "postgres.explain_query", + "postgres.list_schemas", + "supabase.supabase_select", + "supabase.supabase_insert", + "postgres.list_tables", + "postgres.query", + "supabase.supabase_update", + "mysql.mysql_list_tables" + ], + "hit1": "postgres.describe_table", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 590, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "github.issue_write", + "gitlab.gitlab_create_issue", + "bitbucket.bitbucket_create_issue", + "github.create_environment", + "github.assign_copilot_to_issue", + "github.create_repository", + "github.add_issue_comment", + "github.create_package", + "github.create_artifact" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 378, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 564, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1048, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "ranking": [ + "gmail.gmail_create_draft", + "stripe.stripe_create_product", + "okta.okta_create_user", + "stripe.stripe_create_invoice", + "salesforce.salesforce_create_record", + "trello.trello_create_board", + "notion.notion_create_database", + "vercel.vercel_create_project", + "filesystem.create_directory", + "github.create_gist" + ], + "hit1": "gmail.gmail_create_draft", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 39, + "k3_calls": 13, + "k3_tokens": 3322, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 8, + "k5_tokens": 3368, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3296, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q071", + "query": "list-directory", + "kind": "identifier", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "ranking": [ + "filesystem.list_directory", + "filesystem.list_directory_with_sizes", + "clickup.clickup_list_tasks", + "okta.okta_list_users", + "jira.jira_list_projects", + "gmail.gmail_list_labels", + "aws_s3.s3_list_buckets", + "sentry.sentry_list_projects", + "vercel.vercel_list_projects", + "trello.trello_list_cards" + ], + "hit1": "filesystem.list_directory", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 269, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 508, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "kind": "paraphrase", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "ranking": [ + "sequential_thinking.sequentialthinking", + "memory.create_relations", + "memory.add_observations", + "memory.search_nodes", + "memory.create_entities", + "memory.delete_relations", + "notion.update_comment", + "notion.notion_search", + "memory.open_nodes", + "notion.notion_append_block" + ], + "hit1": "sequential_thinking.sequentialthinking", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 650, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "kind": "paraphrase", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "ranking": [ + "time.get_current_time", + "time.convert_time", + "google_calendar.gcal_list_events", + "playwright.browser_wait_for", + "grafana.grafana_create_annotation", + "jira.jira_add_worklog", + "jira.get_sprint", + "jira.get_webhook", + "jira.get_board", + "gitlab.get_milestone" + ], + "hit1": "time.get_current_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 696, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "kind": "verb_product", + "gold": [ + "time.convert_time" + ], + "alts": [], + "ranking": [ + "time.convert_time", + "time.get_current_time", + "playwright.browser_wait_for", + "grafana.grafana_create_annotation", + "google_calendar.gcal_list_events", + "salesforce.salesforce_convert_lead", + "datadog.datadog_query_metrics", + "memory.create_relations", + "cloudflare.cf_list_zones", + "git.git_diff" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "kind": "verb_product", + "gold": [ + "github.star_repository" + ], + "alts": [], + "ranking": [ + "github.star_repository", + "github.unstar_repository", + "github.search_issues", + "github.get_label", + "github.list_releases", + "github.get_file_blame", + "github.actions_list", + "github.fork_repository", + "github.list_tags", + "github.get_tag" + ], + "hit1": "github.star_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 400, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 801, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "kind": "resource", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "ranking": [ + "github.fork_repository", + "github.unstar_repository", + "github.star_repository", + "github.delete_repository", + "github.create_repository", + "github.get_repository_tree", + "github.list_releases", + "github.list_dependabot_alerts", + "github.list_pull_requests", + "github.create_or_update_file" + ], + "hit1": "github.fork_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 875, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "kind": "verb_product", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "ranking": [ + "github.get_latest_release", + "github.get_release_by_tag", + "gitlab.get_release", + "github.get_gist", + "github.get_notification_details", + "github.get_teams", + "github.get_me", + "github.get_code_scanning_alert", + "github.get_commit", + "github.get_secret_scanning_alert" + ], + "hit1": "github.get_latest_release", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "kind": "resource", + "gold": [ + "github.create_gist" + ], + "alts": [], + "ranking": [ + "github.create_gist", + "github.get_gist", + "github.update_gist", + "github.create_milestone", + "gitlab.create_snippet", + "github.create_hook", + "github.create_repository", + "gmail.gmail_create_draft", + "notion.notion_create_database", + "gitlab.create_label" + ], + "hit1": "github.create_gist", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 877, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "kind": "verb_product", + "gold": [ + "github.search_code" + ], + "alts": [], + "ranking": [ + "github.search_code", + "github.search_orgs", + "github.search_users", + "github.get_code_scanning_alert", + "github.list_code_scanning_alerts", + "github.search_pull_requests", + "github.search_commits", + "github.search_repositories", + "github.search_issues", + "gitlab.gitlab_list_projects" + ], + "hit1": "github.search_code", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 719, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "kind": "verb_product", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "ranking": [ + "github.list_notifications", + "github.mark_all_notifications_read", + "github.list_gists", + "github.actions_list", + "github.list_secret_scanning_alerts", + "github.list_code_scanning_alerts", + "github.list_releases", + "github.list_starred_repositories", + "github.list_tags", + "github.list_pull_requests" + ], + "hit1": "github.list_notifications", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 174, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "kind": "verb_product", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "ranking": [ + "github.assign_copilot_to_issue", + "linear.linear_assign_issue", + "jira.jira_assign_issue", + "github.request_copilot_review", + "linear.linear_create_comment", + "jira.jira_update_issue", + "github.create_issue", + "jira.jira_get_issue", + "jira.jira_transition_issue", + "jira.jira_add_comment" + ], + "hit1": "github.assign_copilot_to_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 435, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 822, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "kind": "resource", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "ranking": [ + "github.list_secret_scanning_alerts", + "github.get_secret_scanning_alert", + "github.list_code_scanning_alerts", + "github.get_code_scanning_alert", + "github.list_dependabot_alerts", + "slack.get_reminder", + "github.get_hook", + "pagerduty.pagerduty_get_incident", + "gitlab.get_snippet", + "vercel.vercel_get_deployment" + ], + "hit1": "github.list_secret_scanning_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 756, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "ranking": [ + "github.create_or_update_file", + "github.projects_write", + "github.issue_write", + "github.update_gist", + "slack.update_file", + "github.create_gist", + "github.label_write", + "filesystem.write_file", + "github.update_issue_comment", + "github.get_file_contents" + ], + "hit1": "github.create_or_update_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 389, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 536, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1012, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "kind": "resource", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "ranking": [ + "github.list_repository_collaborators", + "github.actions_list", + "github.list_commits", + "github.list_labels", + "github.list_releases", + "github.list_discussions", + "github.list_tags", + "github.list_branches", + "github.list_dependabot_alerts", + "github.list_issues" + ], + "hit1": "github.list_repository_collaborators", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 410, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 804, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "kind": "verb_product", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "ranking": [ + "slack.slack_reply_to_thread", + "slack.slack_get_thread_replies", + "slack.get_reminder", + "slack.get_canvase", + "slack.delete_reminder", + "slack.update_reminder", + "slack.get_pin", + "slack.get_file", + "slack.update_pin", + "slack.update_file" + ], + "hit1": "slack.slack_reply_to_thread", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "kind": "verb_product", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "ranking": [ + "slack.slack_add_reaction", + "discord.discord_add_reaction", + "slack.slack_post_message", + "slack.get_reminder", + "slack.slack_reply_to_thread", + "slack.get_canvase", + "slack.slack_get_thread_replies", + "slack.slack_search_messages", + "slack.slack_create_channel", + "slack.update_canvase" + ], + "hit1": "slack.slack_add_reaction", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 270, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 414, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "kind": "verb_product", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "ranking": [ + "slack.slack_search_messages", + "slack.slack_get_channel_history", + "github.search_orgs", + "github.search_pull_requests", + "github.search_users", + "gmail.gmail_search_emails", + "slack.get_reminder", + "hubspot.hubspot_search_contacts", + "slack.slack_list_channels", + "slack.slack_post_message" + ], + "hit1": "slack.slack_search_messages", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 636, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "kind": "verb_product", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "ranking": [ + "slack.slack_create_channel", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_post_message", + "slack.slack_set_channel_topic", + "slack.slack_reply_to_thread", + "slack.slack_get_channel_history", + "slack.slack_add_reaction", + "slack.slack_get_thread_replies", + "slack.slack_upload_file" + ], + "hit1": "slack.slack_create_channel", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "kind": "verb_product", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_transition_issue", + "jira.jira_update_issue", + "jira.jira_assign_issue", + "jira.jira_get_issue", + "jira.jira_add_comment", + "jira.jira_create_issue", + "jira.jira_add_worklog", + "jira.jira_search_issues", + "jira.get_sprint", + "jira.jira_list_projects" + ], + "hit1": "jira.jira_transition_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 222, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "kind": "identifier", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "ranking": [ + "jira.jira_search_issues", + "jira.jira_list_projects", + "jira.jira_get_issue", + "github.search_pull_requests", + "confluence.confluence_search", + "hubspot.hubspot_search_contacts", + "github.search_users", + "jira.get_filter", + "github.search_orgs", + "jira.get_webhook" + ], + "hit1": "jira.jira_search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 286, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 549, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "kind": "verb_product", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "ranking": [ + "jira.jira_add_worklog", + "jira.jira_add_comment", + "jira.jira_list_projects", + "jira.jira_get_issue", + "jira.jira_update_issue", + "jira.jira_transition_issue", + "jira.jira_create_issue", + "jira.get_sprint", + "jira.get_webhook", + "jira.jira_assign_issue" + ], + "hit1": "jira.jira_add_worklog", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 332, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 689, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "kind": "verb_product", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "ranking": [ + "linear.linear_list_teams", + "linear.linear_list_projects", + "linear.linear_list_issues", + "linear.list_cycles", + "linear.linear_create_project", + "linear.list_labels", + "linear.list_users", + "linear.linear_get_issue", + "linear.linear_search_issues", + "linear.list_attachments" + ], + "hit1": "linear.linear_list_teams", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "ranking": [ + "linear.linear_create_comment", + "linear.get_comment", + "linear.linear_get_issue", + "linear.update_comment", + "linear.linear_assign_issue", + "linear.delete_comment", + "linear.linear_update_issue", + "linear.linear_create_issue", + "jira.jira_add_comment", + "linear.create_comment" + ], + "hit1": "linear.linear_create_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 794, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "kind": "paraphrase", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "ranking": [ + "filesystem.move_file", + "filesystem.read_text_file", + "filesystem.read_file", + "filesystem.read_media_file", + "filesystem.write_file", + "slack.get_file", + "google_drive.gdrive_update_file", + "slack.update_file", + "figma.figma_get_file", + "gmail.gmail_trash_email" + ], + "hit1": "filesystem.move_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "kind": "paraphrase", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "ranking": [ + "filesystem.search_files", + "google_drive.gdrive_search", + "redis.redis_keys", + "linear.linear_search_issues", + "github.search_repositories", + "memory.search_nodes", + "confluence.confluence_search", + "github.search_orgs", + "github.search_commits", + "notion.notion_search" + ], + "hit1": "filesystem.search_files", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 623, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "kind": "paraphrase", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "ranking": [ + "filesystem.edit_file", + "filesystem.move_file", + "filesystem.write_file", + "slack.update_file", + "slack.create_file", + "slack.get_file", + "slack.delete_file", + "google_drive.gdrive_update_file", + "filesystem.read_text_file", + "github.create_or_update_file" + ], + "hit1": "filesystem.edit_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 413, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 841, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "ranking": [ + "memory.read_graph", + "memory.delete_relations", + "memory.open_nodes", + "memory.add_observations", + "memory.create_entities", + "memory.search_nodes", + "memory.create_relations", + "memory.delete_entities", + "github.mark_all_notifications_read", + "github.pull_request_read" + ], + "hit1": "memory.read_graph", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 154, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 258, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 546, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "kind": "verb_product", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "memory.open_nodes", + "github.search_orgs", + "figma.figma_get_file_nodes", + "hubspot.hubspot_search_contacts", + "memory.read_graph", + "github.search_users", + "filesystem.search_files", + "github.search_pull_requests", + "confluence.confluence_search" + ], + "hit1": "memory.search_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 311, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 570, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "kind": "verb_product", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "ranking": [ + "git.git_diff_unstaged", + "git.git_diff", + "git.git_diff_staged", + "git.git_show", + "git.git_log", + "git.git_status", + "git.git_reset", + "git.git_commit", + "git.git_add", + "git.git_branch" + ], + "hit1": "git.git_diff_unstaged", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 313, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 588, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "kind": "identifier", + "gold": [ + "git.git_add" + ], + "alts": [], + "ranking": [ + "git.git_add", + "github.update_gist", + "github.create_gist", + "salesforce.salesforce_list_opportunities", + "github.push_files", + "github.add_comment_to_pending_review", + "git.git_diff_staged", + "git.git_commit", + "git.git_reset", + "git.git_show" + ], + "hit1": "git.git_add", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 411, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 789, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "kind": "verb_product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "ranking": [ + "shopify.shopify_create_product", + "shopify.shopify_get_product", + "shopify.shopify_update_product", + "shopify.shopify_create_order", + "shopify.shopify_get_order", + "stripe.stripe_create_product", + "shopify.shopify_list_products", + "shopify.shopify_list_customers", + "shopify.shopify_list_orders", + "stripe.stripe_create_price" + ], + "hit1": "shopify.shopify_create_product", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 365, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "kind": "verb_product", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "ranking": [ + "stripe.stripe_cancel_subscription", + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_list_subscriptions", + "stripe.stripe_create_refund", + "stripe.stripe_list_customers", + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_product" + ], + "hit1": "stripe.stripe_cancel_subscription", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 167, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 610, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "kind": "resource", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_refund", + "stripe.stripe_create_invoice", + "stripe.stripe_create_customer", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "shopify.shopify_create_order", + "hubspot.hubspot_create_deal", + "gmail.gmail_create_draft" + ], + "hit1": "stripe.stripe_create_payment_intent", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 739, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "sentry.sentry_list_events", + "slack.get_bookmark", + "slack.get_reminder", + "notion.notion_get_page" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "kind": "verb_product", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "ranking": [ + "datadog.datadog_create_monitor", + "datadog.datadog_get_monitor", + "datadog.datadog_list_monitors", + "datadog.datadog_list_logs", + "datadog.datadog_query_metrics", + "memory.create_entities", + "hubspot.hubspot_create_deal", + "stripe.stripe_create_invoice", + "memory.add_observations", + "vercel.vercel_create_project" + ], + "hit1": "datadog.datadog_create_monitor", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 760, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "kind": "paraphrase", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "ranking": [ + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "sentry.sentry_list_issues", + "sentry.sentry_list_events", + "pagerduty.pagerduty_resolve_incident", + "sentry.sentry_list_projects", + "jira.jira_get_issue", + "jira.jira_assign_issue", + "github.assign_copilot_to_issue", + "bitbucket.bitbucket_create_issue" + ], + "hit1": "sentry.sentry_get_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 274, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_apply", + "kubernetes.k8s_list_services", + "vercel.vercel_get_deployment", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_get_events", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_delete" + ], + "hit1": "kubernetes.k8s_scale_deployment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 241, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_apply", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_pod", + "terraform.tf_apply", + "kubernetes.k8s_get_events", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_delete", + "kubernetes.k8s_pod_logs" + ], + "hit1": "kubernetes.k8s_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "ranking": [ + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_run", + "docker.docker_exec", + "docker.docker_images", + "docker.docker_ps", + "docker.docker_build", + "kubernetes.k8s_pod_logs", + "github.list_environments" + ], + "hit1": "docker.docker_stop", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 664, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "kind": "verb_product", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "ranking": [ + "docker.docker_build", + "docker.docker_run", + "docker.docker_images", + "docker.docker_stop", + "docker.docker_ps", + "docker.docker_logs", + "docker.docker_exec", + "docker.docker_rm", + "filesystem.read_media_file", + "github.create_environment" + ], + "hit1": "docker.docker_build", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 642, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "kind": "verb_product", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "ranking": [ + "playwright.browser_click", + "playwright.browser_type", + "puppeteer.puppeteer_click", + "playwright.browser_take_screenshot", + "playwright.browser_tab_list", + "playwright.browser_navigate", + "playwright.browser_press_key", + "playwright.browser_tab_close", + "playwright.browser_snapshot", + "clickup.clickup_list_tasks" + ], + "hit1": "playwright.browser_click", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 326, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 569, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "kind": "paraphrase", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "ranking": [ + "playwright.browser_type", + "puppeteer.puppeteer_fill", + "playwright.browser_wait_for", + "playwright.browser_navigate", + "playwright.browser_tab_new", + "playwright.browser_tab_list", + "playwright.browser_fill_form", + "playwright.browser_press_key", + "playwright.browser_click", + "playwright.browser_evaluate" + ], + "hit1": "playwright.browser_type", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 336, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 586, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_list_projects", + "gitlab.gitlab_list_issues", + "gitlab.gitlab_list_merge_requests", + "vercel.vercel_list_projects", + "jira.jira_list_projects", + "gitlab.gitlab_create_issue", + "github.projects_list", + "gitlab.gitlab_create_pipeline", + "gitlab.gitlab_get_issue", + "sentry.sentry_list_projects" + ], + "hit1": "gitlab.gitlab_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 288, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 640, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "kind": "verb_product", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "ranking": [ + "bitbucket.bitbucket_create_pr", + "bitbucket.bitbucket_list_prs", + "bitbucket.bitbucket_create_issue", + "github.pull_request_review_write", + "github.create_pull_request", + "github.pull_request_read", + "github.merge_pull_request", + "github.update_pull_request_branch", + "github.request_copilot_review", + "github.update_pull_request" + ], + "hit1": "bitbucket.bitbucket_create_pr", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 606, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1139, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "kind": "verb_product", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "ranking": [ + "gmail.gmail_create_draft", + "gmail.gmail_list_labels", + "gmail.gmail_get_email", + "gmail.gmail_modify_email", + "gmail.gmail_trash_email", + "gmail.gmail_send_email", + "gmail.gmail_search_emails", + "stripe.stripe_create_customer", + "github.create_pull_request", + "shopify.shopify_create_order" + ], + "hit1": "gmail.gmail_create_draft", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 789, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "kind": "verb_product", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_share_file", + "google_drive.gdrive_update_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "slack.get_file", + "filesystem.write_file", + "figma.figma_get_file", + "filesystem.get_file_info" + ], + "hit1": "google_drive.gdrive_share_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 306, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "kind": "verb_product", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_update_event", + "kubernetes.k8s_get_events", + "sentry.sentry_list_events", + "clickup.clickup_list_tasks", + "gmail.gmail_list_labels", + "playwright.browser_tab_list" + ], + "hit1": "google_calendar.gcal_list_events", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_create_event", + "github.delete_milestone", + "google_calendar.gcal_update_event", + "google_calendar.gcal_list_events", + "gitlab.delete_milestone", + "redis.redis_delete", + "google_calendar.gcal_list_calendars", + "slack.delete_reminder", + "linear.delete_cycle" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 444, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 690, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_projects", + "vercel.vercel_list_deployments", + "jira.jira_list_projects", + "vercel.vercel_create_project", + "sentry.sentry_list_projects", + "linear.linear_list_projects", + "vercel.vercel_get_deployment", + "gitlab.gitlab_list_projects", + "github.projects_list", + "kubernetes.k8s_list_deployments" + ], + "hit1": "vercel.vercel_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 260, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 539, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "ranking": [ + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update", + "supabase.supabase_list_tables", + "supabase.supabase_invoke_fn", + "datadog.datadog_list_logs", + "datadog.datadog_query_metrics", + "figma.figma_get_file_nodes", + "memory.delete_observations", + "sqlite.sqlite_query" + ], + "hit1": "supabase.supabase_select", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 736, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "kind": "verb_product", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "figma.figma_get_file", + "figma.figma_get_comments", + "linear.get_comment", + "notion.get_comment", + "figma.figma_get_images", + "linear.create_comment", + "linear.update_comment", + "linear.delete_comment", + "figma.figma_get_file_nodes" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 308, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 683, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "kind": "verb_product", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "ranking": [ + "okta.okta_list_users", + "okta.okta_create_user", + "okta.okta_get_user", + "okta.okta_deactivate_user", + "notion.notion_list_users", + "notion.list_users_tokens", + "slack.slack_list_users", + "slack.slack_get_users", + "linear.list_users", + "github.search_users" + ], + "hit1": "okta.okta_list_users", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 271, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query_range", + "prometheus.prom_query", + "prometheus.prom_list_metrics", + "datadog.datadog_query_metrics", + "memory.search_nodes", + "datadog.datadog_list_monitors", + "salesforce.salesforce_query", + "postgres.query", + "confluence.confluence_search", + "shopify.shopify_list_products" + ], + "hit1": "prometheus.prom_query_range", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "kind": "verb_product", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "ranking": [ + "grafana.grafana_create_annotation", + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "trello.trello_create_board", + "github.create_artifact", + "gitlab.create_snippet", + "trello.trello_create_card", + "stripe.stripe_create_product", + "linear.linear_create_project", + "salesforce.salesforce_create_record" + ], + "hit1": "grafana.grafana_create_annotation", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 788, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "kind": "identifier", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "ranking": [ + "terraform.tf_apply", + "terraform.tf_plan", + "kubernetes.k8s_apply", + "terraform.tf_state_list", + "github.update_environment", + "cloudflare.cf_create_dns", + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "playwright.browser_fill_form", + "hubspot.hubspot_create_deal" + ], + "hit1": "terraform.tf_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 653, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "kind": "verb_product", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "ranking": [ + "asana.asana_list_tasks", + "asana.asana_get_task", + "asana.asana_create_task", + "clickup.clickup_list_tasks", + "asana.asana_create_project", + "asana.asana_update_task", + "okta.okta_list_users", + "jira.jira_list_projects", + "aws_s3.s3_list_buckets", + "salesforce.salesforce_list_opportunities" + ], + "hit1": "asana.asana_list_tasks", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 635, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "ranking": [ + "trello.trello_add_comment", + "trello.trello_create_card", + "trello.trello_list_cards", + "trello.trello_create_board", + "linear.get_comment", + "figma.figma_post_comment", + "notion.get_comment", + "linear.update_comment", + "linear.create_comment", + "notion.update_comment" + ], + "hit1": "trello.trello_add_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_update_task", + "clickup.clickup_get_task", + "clickup.clickup_list_tasks", + "clickup.clickup_create_task", + "asana.asana_update_task", + "puppeteer.puppeteer_click", + "playwright.browser_click", + "shopify.shopify_update_product", + "gitlab.update_snippet", + "github.update_milestone" + ], + "hit1": "clickup.clickup_update_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 732, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "kind": "verb_product", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "ranking": [ + "confluence.confluence_search", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "confluence.confluence_create_page", + "grafana.grafana_search_dashboards", + "github.search_orgs", + "github.search_users", + "hubspot.hubspot_search_contacts", + "github.search_pull_requests", + "github.search_commits" + ], + "hit1": "confluence.confluence_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 636, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_deal", + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_search_contacts", + "stripe.stripe_create_refund", + "github.create_pull_request", + "stripe.stripe_create_invoice", + "slack.slack_create_channel", + "shopify.shopify_create_order" + ], + "hit1": "hubspot.hubspot_create_deal", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 772, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_search", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_create_ticket", + "google_calendar.gcal_list_calendars", + "salesforce.salesforce_list_opportunities", + "vercel.vercel_list_projects", + "jira.jira_list_projects", + "playwright.browser_tab_list" + ], + "hit1": "zendesk.zendesk_list_tickets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 565, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "kind": "verb_product", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "ranking": [ + "discord.discord_list_channels", + "discord.discord_list_members", + "discord.discord_get_channel", + "slack.slack_list_channels", + "discord.discord_send_message", + "discord.discord_add_reaction", + "okta.okta_list_users", + "gmail.gmail_list_labels", + "google_calendar.gcal_list_calendars", + "twilio.twilio_list_messages" + ], + "hit1": "discord.discord_list_channels", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 166, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 562, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "ranking": [ + "redis.redis_set", + "redis.redis_get", + "redis.redis_delete", + "redis.redis_keys", + "playwright.browser_press_key", + "jira.jira_update_issue", + "aws_s3.s3_put_object", + "jira.jira_transition_issue", + "slack.slack_set_channel_topic", + "slack.update_pin" + ], + "hit1": "redis.redis_set", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 642, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert", + "mongodb.mongo_update", + "mongodb.mongo_delete", + "mongodb.mongo_find", + "supabase.supabase_insert", + "mongodb.mongo_list_collections", + "aws_s3.s3_put_object", + "salesforce.salesforce_create_record", + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "kind": "verb_product", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "ranking": [ + "mysql.mysql_query", + "postgres.query", + "sqlite.sqlite_query", + "postgres.explain_query", + "salesforce.salesforce_query", + "mysql.mysql_list_tables", + "prometheus.prom_query", + "datadog.datadog_query_metrics", + "prometheus.prom_query_range", + "github.update_check_run" + ], + "hit1": "mysql.mysql_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "elasticsearch.es_search", + "playwright.browser_tab_close", + "mongodb.mongo_find", + "github.get_milestone", + "confluence.confluence_search", + "github.delete_milestone", + "google_drive.gdrive_search", + "github.get_hook" + ], + "hit1": "elasticsearch.es_index_doc", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 637, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_put_object", + "aws_s3.s3_list_objects", + "redis.redis_get", + "asana.asana_get_task", + "okta.okta_get_user", + "gitlab.get_snippet", + "shopify.shopify_get_product", + "shopify.shopify_get_order" + ], + "hit1": "aws_s3.s3_get_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 640, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones", + "cloudflare.cf_list_dns", + "cloudflare.cf_purge_cache", + "cloudflare.cf_create_dns", + "aws_s3.s3_list_buckets", + "kubernetes.k8s_list_services", + "kubernetes.k8s_list_deployments", + "linear.linear_list_teams", + "salesforce.salesforce_list_opportunities", + "google_calendar.gcal_list_calendars" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 326, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 558, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_create_customer", + "stripe.stripe_list_invoices", + "stripe.stripe_create_product", + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_refund", + "stripe.stripe_create_price", + "stripe.stripe_get_customer", + "stripe.stripe_get_payment_intent", + "stripe.stripe_list_subscriptions" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "kind": "verb_product", + "gold": [ + "github.actions_list" + ], + "alts": [], + "ranking": [ + "github.actions_list", + "github.actions_get", + "github.actions_run_trigger", + "github.actions_get_job_logs", + "github.list_gists", + "github.list_notifications", + "github.list_issues", + "github.list_code_scanning_alerts", + "github.list_releases", + "github.list_tags" + ], + "hit1": "github.actions_list", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 438, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 845, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "kind": "verb_product", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "ranking": [ + "github.actions_get_job_logs", + "github.actions_get", + "github.actions_list", + "github.actions_run_trigger", + "git.git_log", + "github.get_notification_details", + "github.get_code_scanning_alert", + "github.get_secret_scanning_alert", + "gitlab.get_job", + "github.search_pull_requests" + ], + "hit1": "github.actions_get_job_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 450, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 804, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "kind": "verb_product", + "gold": [ + "github.create_repository" + ], + "alts": [], + "ranking": [ + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.create_or_update_file", + "github.create_gist", + "github.star_repository", + "github.unstar_repository", + "github.fork_repository", + "github.delete_repository", + "github.list_releases" + ], + "hit1": "github.create_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 556, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 915, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "kind": "verb_product", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "ranking": [ + "git.git_checkout", + "git.git_branch", + "git.git_create_branch", + "github.create_branch", + "git.git_show", + "github.update_pull_request_branch", + "git.git_status", + "git.git_diff", + "github.list_commits", + "github.list_branches" + ], + "hit1": "git.git_checkout", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "kind": "identifier", + "gold": [ + "git.git_log" + ], + "alts": [], + "ranking": [ + "git.git_log", + "git.git_show", + "git.git_status", + "git.git_diff_unstaged", + "git.git_diff", + "git.git_branch", + "git.git_commit", + "git.git_diff_staged", + "git.git_checkout", + "git.git_add" + ], + "hit1": "git.git_log", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "kind": "verb_product", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "ranking": [ + "postgres.list_schemas", + "postgres.list_tables", + "postgres.describe_table", + "aws_s3.s3_list_buckets", + "vercel.vercel_list_projects", + "jira.jira_list_projects", + "sentry.sentry_list_projects", + "mysql.mysql_list_tables", + "postgres.explain_query", + "kubernetes.k8s_list_deployments" + ], + "hit1": "postgres.list_schemas", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 169, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 248, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 491, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "kind": "verb_product", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "ranking": [ + "memory.open_nodes", + "memory.search_nodes", + "memory.read_graph", + "memory.create_entities", + "memory.create_relations", + "memory.delete_relations", + "memory.add_observations", + "memory.delete_entities", + "figma.figma_get_file_nodes", + "playwright.browser_tab_list" + ], + "hit1": "memory.open_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 531, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "kind": "paraphrase", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "ranking": [ + "filesystem.get_file_info", + "filesystem.read_file", + "google_drive.gdrive_get_file", + "filesystem.read_media_file", + "filesystem.read_text_file", + "figma.figma_get_file_nodes", + "github.get_file_contents", + "figma.figma_get_images", + "figma.figma_get_file", + "filesystem.list_directory" + ], + "hit1": "filesystem.get_file_info", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "kind": "paraphrase", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "ranking": [ + "filesystem.write_file", + "filesystem.edit_file", + "filesystem.read_text_file", + "filesystem.read_file", + "filesystem.read_media_file", + "slack.create_file", + "playwright.browser_tab_new", + "filesystem.create_directory", + "github.issue_write", + "slack.get_file" + ], + "hit1": "filesystem.write_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 775, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "kind": "verb_product", + "gold": [ + "github.search_issues" + ], + "alts": [], + "ranking": [ + "github.search_issues", + "github.search_users", + "github.search_orgs", + "github.list_issues", + "github.search_pull_requests", + "github.search_commits", + "github.search_repositories", + "github.search_code", + "linear.linear_search_issues", + "jira.jira_search_issues" + ], + "hit1": "github.search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "kind": "paraphrase", + "gold": [ + "github.get_me" + ], + "alts": [], + "ranking": [ + "github.get_teams", + "github.get_me", + "github.list_notifications", + "github.get_latest_release", + "github.list_gists", + "slack.slack_get_user_profile", + "github.projects_get", + "github.get_label", + "github.get_notification_details", + "github.get_secret_scanning_alert" + ], + "hit1": "github.get_teams", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 656, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "cost": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.0866666666666667, + "p95_calls": 1.0, + "success_within_1": 0.9866666666666667, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 250.80666666666667, + "p95_tokens": 322.3499999999997, + "total_tokens": 37621, + "tokens_per_successful_query": 250.80666666666667, + "total_tokens_successful": 37621 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.0466666666666666, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 389.6066666666667, + "p95_tokens": 535.55, + "total_tokens": 58441, + "tokens_per_successful_query": 389.6066666666667, + "total_tokens_successful": 58441 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.02, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 736.7133333333334, + "p95_tokens": 995.7999999999994, + "total_tokens": 110507, + "tokens_per_successful_query": 736.7133333333334, + "total_tokens_successful": 110507 + } + } +} diff --git a/experiments/mcp-tool-search/results/hybrid_rrf_bm25_trigram.csv b/experiments/mcp-tool-search/results/hybrid_rrf_bm25_trigram.csv new file mode 100644 index 00000000..8a13842f --- /dev/null +++ b/experiments/mcp-tool-search/results/hybrid_rrf_bm25_trigram.csv @@ -0,0 +1,151 @@ +id,query,kind,hit1,recall@1,recall@3,recall@5,mrr@10,ndcg@10,hit_rank,k3_calls,k3_tokens,k3_success1,k5_calls,k5_tokens,k5_success1,k10_calls,k10_tokens,k10_success1,ranking +q001,Create a GitHub issue,verb_product,github.create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,380,1.0,1,557,1.0,1,1038,1.0,github.create_issue github.issue_write gitlab.gitlab_create_issue github.issue_read github.create_gist github.update_issue_comment github.create_repository github.create_branch bitbucket.bitbucket_create_issue github.add_issue_comment +q002,Create a Jira ticket,verb_product,jira.jira_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,318,1.0,1,513,1.0,1,1003,1.0,jira.jira_create_issue zendesk.zendesk_create_ticket jira.create_board jira.create_sprint jira.create_filter jira.create_version jira.create_webhook zendesk.zendesk_update_ticket jira.create_attachment jira.jira_create_sprint +q003,Send a Slack message,verb_product,slack.slack_post_message,1.0,1.0,1.0,1.0000,1.0000,1,1,247,1.0,1,368,1.0,1,673,1.0,slack.slack_post_message discord.discord_send_message slack.slack_add_reaction slack.slack_search_messages slack.slack_join_channel slack.slack_leave_channel slack.slack_create_channel slack.slack_get_user_profile slack.slack_get_channel_history slack.get_pin +q004,List open pull requests,resource,github.list_pull_requests,1.0,1.0,1.0,1.0000,0.9502,1,1,237,1.0,1,460,1.0,1,887,1.0,github.list_pull_requests bitbucket.bitbucket_list_prs github.search_pull_requests github.create_pull_request gitlab.gitlab_list_merge_requests github.pull_request_read github.merge_pull_request playwright.browser_tab_list github.request_copilot_review github.update_pull_request_branch +q005,Open a pull request on GitHub,verb_product,github.create_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,364,1.0,1,522,1.0,1,1144,1.0,github.create_pull_request github.request_copilot_review github.pull_request_read github.merge_pull_request github.search_pull_requests github.update_pull_request_branch github.pull_request_review_write github.list_pull_requests github.update_pull_request github.add_comment_to_pending_review +q006,Run a SQL query against postgres,verb_product,postgres.query,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,295,1.0,1,590,1.0,postgres.query mysql.mysql_query sqlite.sqlite_query postgres.explain_query salesforce.salesforce_query postgres.list_schemas postgres.list_tables postgres.describe_table docker.docker_run prometheus.prom_query +q007,Read a local file from disk,paraphrase,filesystem.read_file,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,390,1.0,1,806,1.0,filesystem.read_file filesystem.read_text_file filesystem.read_media_file gitlab.gitlab_get_file figma.figma_get_file_nodes github.delete_file figma.figma_get_images github.get_file_contents figma.figma_get_file slack.get_file +q008,What files are in this folder,paraphrase,google_drive.gdrive_list_folder,0.0,0.0,0.0,0.1111,0.1923,9,3,679,0.0,2,712,0.0,1,690,1.0,google_drive.gdrive_list_folder slack.list_files google_drive.gdrive_search cloudflare.cf_purge_cache github.push_files git.git_add filesystem.search_files github.get_milestone filesystem.list_directory github.delete_milestone +q009,Remember this fact in the knowledge graph,paraphrase,memory.read_graph,0.0,1.0,1.0,0.5000,0.5106,2,1,154,1.0,1,255,1.0,1,507,1.0,memory.read_graph memory.create_entities memory.delete_relations memory.open_nodes memory.create_relations memory.add_observations memory.search_nodes memory.delete_entities time.get_current_time postgres.list_schemas +q010,Search the web with Brave,verb_product,brave_search.brave_web_search,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,410,1.0,1,688,1.0,brave_search.brave_web_search brave_search.brave_local_search jira.jira_search_issues github.update_pull_request_branch memory.search_nodes github.search_users github.search_orgs github.search_commits confluence.confluence_search slack.slack_search_messages +q011,Fetch the contents of a URL,verb_product,fetch.fetch,1.0,1.0,1.0,1.0000,1.0000,1,1,222,1.0,1,335,1.0,1,660,1.0,fetch.fetch filesystem.read_file git.git_add playwright.browser_navigate docker.docker_logs github.get_me github.get_file_contents git.git_status slack.slack_set_channel_topic playwright.browser_snapshot +q012,Show git status,identifier,git.git_status,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,315,1.0,1,667,1.0,git.git_status git.git_diff_staged shopify.shopify_list_orders git.git_diff_unstaged github.list_hooks git.git_show gitlab.list_jobs gitlab.list_snippets git.git_log github.list_packages +q013,Commit staged changes,paraphrase,git.git_commit,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,305,1.0,1,725,1.0,git.git_commit git.git_reset git.git_diff_staged git.git_show git.git_log github.get_commit github.push_files linear.list_comments notion.list_comments github.list_packages +q014,Create a Linear issue,verb_product,linear.linear_create_comment,0.0,1.0,1.0,0.5000,0.6309,2,1,333,1.0,1,463,1.0,1,921,1.0,linear.linear_create_comment linear.linear_create_issue linear.linear_update_issue linear.linear_assign_issue linear.linear_get_issue linear.linear_create_project linear.create_user linear.create_label linear.create_cycle bitbucket.bitbucket_create_issue +q015,Make a task in my project tracker Linear,paraphrase,linear.linear_create_project,0.0,0.0,0.0,0.0000,0.0000,146,49,12289,0.0,30,12029,0.0,15,11669,0.0,linear.linear_create_project linear.delete_user linear.get_user linear.delete_cycle asana.asana_create_task linear.get_cycle linear.delete_label linear.get_label linear.delete_comment linear.linear_list_projects +q016,Post a comment on a GitHub issue,verb_product,github.update_issue_comment,0.0,1.0,1.0,0.3333,0.5000,3,1,307,1.0,1,442,1.0,1,912,1.0,github.update_issue_comment figma.figma_post_comment github.add_issue_comment jira.jira_add_comment linear.linear_create_comment github.issue_read github.assign_copilot_to_issue github.get_discussion_comments trello.trello_add_comment github.list_commits +q017,Merge the pull request,ambiguous,github.merge_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,299,1.0,1,468,1.0,1,957,1.0,github.merge_pull_request gitlab.gitlab_merge_merge_request github.update_pull_request_branch gitlab.gitlab_get_merge_request github.pull_request_read github.request_copilot_review github.search_pull_requests gitlab.gitlab_create_merge_request github.update_pull_request bitbucket.bitbucket_list_prs +q018,Trigger a GitHub Actions workflow,verb_product,github.actions_run_trigger,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,455,1.0,1,797,1.0,github.actions_run_trigger github.actions_get github.actions_list github.actions_get_job_logs github.star_repository github.fork_repository github.unstar_repository github.search_orgs github.delete_repository github.list_tags +q019,Get Dependabot alerts,resource,github.list_dependabot_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,427,1.0,1,691,1.0,github.list_dependabot_alerts github.get_dependabot_alert github.list_secret_scanning_alerts github.list_code_scanning_alerts vercel.vercel_get_deployment github.get_hook zendesk.zendesk_get_ticket gitlab.get_job redis.redis_get gitlab.get_wiki +q020,List Sentry issues,verb_product,sentry.sentry_list_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,380,1.0,1,796,1.0,sentry.sentry_list_issues sentry.sentry_list_projects sentry.sentry_list_events gitlab.gitlab_list_issues sentry.sentry_get_issue linear.linear_list_issues linear.linear_search_issues sentry.sentry_update_issue github.list_issues github.search_issues +q021,Create a Stripe customer,verb_product,stripe.stripe_create_customer,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,322,1.0,1,669,1.0,stripe.stripe_create_customer stripe.stripe_create_invoice stripe.stripe_get_customer stripe.stripe_create_payment_intent stripe.stripe_list_invoices stripe.stripe_create_price stripe.stripe_create_product stripe.stripe_list_subscriptions stripe.stripe_list_customers stripe.stripe_create_refund +q022,Refund a Stripe payment,paraphrase,stripe.stripe_create_refund,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,332,1.0,1,647,1.0,stripe.stripe_create_refund stripe.stripe_get_payment_intent stripe.stripe_create_payment_intent stripe.stripe_get_customer stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_create_customer stripe.stripe_create_product stripe.stripe_create_price stripe.stripe_cancel_subscription +q023,Send an email with Gmail,verb_product,gmail.gmail_send_email,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,365,1.0,1,713,1.0,gmail.gmail_send_email gmail.gmail_get_email twilio.twilio_send_sms puppeteer.puppeteer_click gmail.gmail_trash_email gmail.gmail_search_emails gmail.gmail_list_labels puppeteer.puppeteer_fill gmail.gmail_modify_email gmail.gmail_create_draft +q024,Search my Gmail inbox,verb_product,gmail.gmail_search_emails,1.0,1.0,1.0,1.0000,1.0000,1,1,174,1.0,1,294,1.0,1,615,1.0,gmail.gmail_search_emails gmail.gmail_list_labels github.search_users confluence.confluence_search elasticsearch.es_search slack.slack_search_messages gmail.gmail_get_email github.search_orgs gmail.gmail_send_email github.search_commits +q025,Upload a file to Google Drive,paraphrase,google_drive.gdrive_update_file,0.0,0.0,1.0,0.2000,0.3869,5,2,523,0.0,1,418,1.0,1,736,1.0,google_drive.gdrive_update_file google_drive.gdrive_share_file google_drive.gdrive_search google_drive.gdrive_get_file google_drive.gdrive_create_file slack.slack_upload_file google_drive.gdrive_list_folder aws_s3.s3_put_object figma.figma_get_file google_calendar.gcal_list_calendars +q026,Create a calendar event,resource,google_calendar.gcal_create_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,446,1.0,1,840,1.0,google_calendar.gcal_create_event google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_list_events trello.trello_create_board google_calendar.gcal_list_calendars stripe.stripe_create_refund vercel.vercel_create_project github.pull_request_review_write notion.notion_create_database +q027,List Kubernetes pods,verb_product,kubernetes.k8s_list_pods,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,314,1.0,1,699,1.0,kubernetes.k8s_list_pods kubernetes.k8s_list_services kubernetes.k8s_list_deployments kubernetes.k8s_get_events kubernetes.k8s_get_pod kubernetes.k8s_apply kubernetes.k8s_pod_logs kubernetes.k8s_delete kubernetes.k8s_scale_deployment linear.linear_list_teams +q028,Get logs from a k8s pod,verb_product,kubernetes.k8s_pod_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,262,1.0,1,441,1.0,1,830,1.0,kubernetes.k8s_pod_logs kubernetes.k8s_get_pod docker.docker_logs github.actions_get_job_logs github.get_tag figma.figma_get_file_nodes elasticsearch.es_get_doc datadog.datadog_list_logs aws_s3.s3_get_object slack.slack_get_channel_history +q029,Run a docker container,verb_product,docker.docker_run,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,345,1.0,1,647,1.0,docker.docker_run docker.docker_stop docker.docker_exec docker.docker_rm docker.docker_logs docker.docker_ps docker.docker_images docker.docker_build kubernetes.k8s_pod_logs salesforce.salesforce_query +q030,Show docker container logs,verb_product,docker.docker_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,345,1.0,1,649,1.0,docker.docker_logs docker.docker_stop docker.docker_rm docker.docker_exec docker.docker_run docker.docker_ps kubernetes.k8s_pod_logs git.git_log docker.docker_images docker.docker_build +q031,Navigate the browser to a URL,ambiguous,playwright.browser_navigate,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,282,1.0,1,536,1.0,playwright.browser_navigate puppeteer.puppeteer_navigate playwright.browser_tab_new playwright.browser_press_key playwright.browser_evaluate playwright.browser_wait_for playwright.browser_click playwright.browser_tab_close playwright.browser_tab_list playwright.browser_snapshot +q032,Take a screenshot of the page,ambiguous,playwright.browser_take_screenshot,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,314,1.0,1,612,1.0,playwright.browser_take_screenshot puppeteer.puppeteer_screenshot playwright.browser_snapshot playwright.browser_evaluate playwright.browser_press_key playwright.browser_click git.git_status filesystem.read_file notion.notion_update_page confluence.confluence_get_page +q033,Create a GitLab merge request,verb_product,gitlab.gitlab_merge_merge_request,0.0,1.0,1.0,0.5000,0.6309,2,1,297,1.0,1,475,1.0,1,1054,1.0,gitlab.gitlab_merge_merge_request gitlab.gitlab_create_merge_request gitlab.gitlab_get_merge_request github.merge_pull_request gitlab.gitlab_list_merge_requests gitlab.gitlab_create_pipeline gitlab.gitlab_create_issue bitbucket.bitbucket_create_pr github.pull_request_review_write github.create_pull_request +q034,Search Notion pages,verb_product,notion.notion_search,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,319,1.0,1,614,1.0,notion.notion_search notion.notion_get_page notion.notion_update_page notion.notion_get_comments notion.notion_get_block github.search_orgs notion.notion_create_database slack.slack_search_messages notion.notion_list_users github.search_users +q035,Create a Notion page,verb_product,notion.notion_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,230,1.0,1,371,1.0,1,832,1.0,notion.notion_create_page notion.notion_get_page notion.notion_update_page notion.notion_create_database notion.notion_get_comments notion.create_comment confluence.confluence_create_page notion.create_users_token notion.create_data_source notion.update_comment +q036,Query a Notion database,verb_product,notion.notion_query_database,1.0,1.0,1.0,1.0000,1.0000,1,1,261,1.0,1,401,1.0,1,688,1.0,notion.notion_query_database notion.notion_create_database notion.notion_search notion.notion_create_page notion.notion_get_block notion.notion_get_page notion.notion_update_page notion.delete_data_source notion.get_data_source notion.delete_comment +q037,Create a Zendesk ticket,verb_product,zendesk.zendesk_create_ticket,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,368,1.0,1,780,1.0,zendesk.zendesk_create_ticket zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_list_tickets zendesk.zendesk_search jira.jira_create_issue trello.trello_create_board bitbucket.bitbucket_create_issue vercel.vercel_create_project confluence.confluence_create_page +q038,Create a HubSpot contact,verb_product,hubspot.hubspot_create_contact,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,364,1.0,1,709,1.0,hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_list_contacts hubspot.hubspot_create_deal hubspot.hubspot_search_contacts github.create_gist vercel.vercel_create_project stripe.stripe_create_product trello.trello_create_board github.projects_write +q039,Create a Salesforce lead,verb_product,salesforce.salesforce_convert_lead,0.0,1.0,1.0,0.5000,0.6309,2,1,252,1.0,1,402,1.0,1,710,1.0,salesforce.salesforce_convert_lead salesforce.salesforce_create_lead salesforce.salesforce_create_record salesforce.salesforce_update_record salesforce.salesforce_query salesforce.salesforce_delete_record salesforce.salesforce_list_opportunities salesforce.salesforce_get_record stripe.stripe_create_invoice vercel.vercel_create_project +q040,Run a SOQL query,identifier,salesforce.salesforce_query,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,291,1.0,1,592,1.0,salesforce.salesforce_query sqlite.sqlite_query mysql.mysql_query postgres.query prometheus.prom_query postgres.explain_query notion.notion_query_database memory.search_nodes github.search_users confluence.confluence_search +q041,List Shopify orders,verb_product,shopify.shopify_list_orders,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,320,1.0,1,638,1.0,shopify.shopify_list_orders shopify.shopify_list_customers shopify.shopify_list_products shopify.shopify_get_order shopify.shopify_get_product shopify.shopify_create_order shopify.shopify_update_product shopify.shopify_create_product okta.okta_list_users linear.linear_list_teams +q042,Create a PagerDuty incident,verb_product,pagerduty.pagerduty_create_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,403,1.0,1,721,1.0,pagerduty.pagerduty_create_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents stripe.stripe_create_invoice stripe.stripe_create_refund notion.notion_create_database trello.trello_create_board vercel.vercel_create_project +q043,Who is on call in PagerDuty,paraphrase,pagerduty.pagerduty_list_oncalls,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,363,1.0,1,727,1.0,pagerduty.pagerduty_list_oncalls pagerduty.pagerduty_get_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_create_incident notion.notion_get_comments playwright.browser_press_key playwright.browser_evaluate github.list_commits +q044,Query Datadog metrics,verb_product,datadog.datadog_query_metrics,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,458,1.0,1,799,1.0,datadog.datadog_query_metrics datadog.datadog_list_monitors datadog.datadog_create_monitor datadog.datadog_list_logs datadog.datadog_get_monitor prometheus.prom_list_metrics prometheus.prom_query notion.notion_query_database prometheus.prom_query_range postgres.explain_query +q045,Search Datadog logs,verb_product,datadog.datadog_list_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,395,1.0,1,742,1.0,datadog.datadog_list_logs datadog.datadog_list_monitors datadog.datadog_get_monitor github.search_orgs datadog.datadog_query_metrics grafana.grafana_search_dashboards github.search_users confluence.confluence_search datadog.datadog_create_monitor notion.notion_search +q046,List S3 buckets,verb_product,aws_s3.s3_list_buckets,1.0,1.0,1.0,1.0000,1.0000,1,1,129,1.0,1,229,1.0,1,476,1.0,aws_s3.s3_list_buckets linear.linear_list_teams jira.jira_list_projects linear.linear_list_projects zendesk.zendesk_list_tickets bitbucket.bitbucket_list_repos vercel.vercel_list_projects sentry.sentry_list_projects aws_s3.s3_list_objects gmail.gmail_list_labels +q047,Upload an object to S3,paraphrase,aws_s3.s3_put_object,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,426,1.0,1,850,1.0,aws_s3.s3_put_object aws_s3.s3_delete_object aws_s3.s3_get_object slack.slack_upload_file gmail.gmail_send_email twilio.twilio_send_sms aws_s3.s3_list_objects github.assign_copilot_to_issue slack.slack_add_reaction asana.asana_create_project +q048,Purge Cloudflare cache,verb_product,cloudflare.cf_purge_cache,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,335,1.0,1,548,1.0,cloudflare.cf_purge_cache cloudflare.cf_list_zones cloudflare.cf_list_dns cloudflare.cf_create_dns playwright.browser_tab_close playwright.browser_snapshot postgres.list_tables memory.read_graph postgres.list_schemas puppeteer.puppeteer_click +q049,Create a Cloudflare DNS record,verb_product,cloudflare.cf_create_dns,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,356,1.0,1,703,1.0,cloudflare.cf_create_dns cloudflare.cf_list_dns salesforce.salesforce_create_record cloudflare.cf_list_zones cloudflare.cf_purge_cache salesforce.salesforce_update_record trello.trello_create_board salesforce.salesforce_delete_record vercel.vercel_create_project stripe.stripe_create_product +q050,List Vercel deployments,verb_product,vercel.vercel_list_deployments,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,291,1.0,1,497,1.0,vercel.vercel_list_deployments vercel.vercel_list_projects vercel.vercel_get_deployment kubernetes.k8s_list_deployments vercel.vercel_create_project linear.linear_list_teams linear.linear_list_projects jira.jira_list_projects aws_s3.s3_list_buckets sentry.sentry_list_projects +q051,Insert a row into Supabase,verb_product,supabase.supabase_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,203,1.0,1,352,1.0,1,672,1.0,supabase.supabase_insert mongodb.mongo_insert supabase.supabase_list_tables supabase.supabase_select supabase.supabase_update supabase.supabase_invoke_fn playwright.browser_type gitlab.get_release slack.get_canvase gitlab.delete_release +q052,Export images from Figma,paraphrase,figma.figma_get_images,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,328,1.0,1,702,1.0,figma.figma_get_images figma.figma_get_file_nodes docker.docker_images figma.figma_get_file figma.figma_get_comments figma.figma_post_comment supabase.supabase_select docker.docker_logs mongodb.mongo_delete kubernetes.k8s_pod_logs +q053,Send an SMS with Twilio,verb_product,twilio.twilio_send_sms,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,377,1.0,1,740,1.0,twilio.twilio_send_sms twilio.twilio_list_messages gmail.gmail_send_email puppeteer.puppeteer_click puppeteer.puppeteer_fill twilio.twilio_make_call filesystem.list_directory_with_sizes discord.discord_send_message jira.jira_search_issues puppeteer.puppeteer_screenshot +q054,Create an Okta user,verb_product,okta.okta_create_user,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,329,1.0,1,753,1.0,okta.okta_create_user okta.okta_get_user okta.okta_deactivate_user okta.okta_list_users linear.create_user asana.asana_create_project linear.update_user linear.delete_user asana.asana_create_task github.create_issue +q055,Instant query Prometheus,verb_product,prometheus.prom_query,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,334,1.0,1,666,1.0,prometheus.prom_query prometheus.prom_query_range prometheus.prom_list_metrics mysql.mysql_query salesforce.salesforce_query sqlite.sqlite_query postgres.explain_query datadog.datadog_query_metrics postgres.query shopify.shopify_list_products +q056,Search Grafana dashboards,verb_product,grafana.grafana_search_dashboards,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,288,1.0,1,670,1.0,grafana.grafana_search_dashboards grafana.grafana_get_dashboard github.search_users github.search_orgs confluence.confluence_search grafana.grafana_create_annotation slack.slack_search_messages github.search_pull_requests github.search_issues brave_search.brave_web_search +q057,Trigger a CircleCI pipeline,verb_product,circleci.circleci_trigger_pipeline,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,325,1.0,1,755,1.0,circleci.circleci_trigger_pipeline circleci.circleci_get_pipeline gitlab.gitlab_create_pipeline circleci.circleci_list_pipelines gitlab.get_pipeline gitlab.delete_pipeline github.actions_run_trigger gitlab.update_pipeline hubspot.hubspot_create_deal gitlab.create_pipeline +q058,Run terraform plan,identifier,terraform.tf_plan,1.0,1.0,1.0,1.0000,1.0000,1,1,172,1.0,1,287,1.0,1,593,1.0,terraform.tf_plan terraform.tf_apply terraform.tf_state_list salesforce.salesforce_query postgres.explain_query github.get_check_run github.delete_check_run mysql.mysql_query sqlite.sqlite_query docker.docker_run +q059,Create an Asana task,verb_product,asana.asana_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,429,1.0,1,841,1.0,asana.asana_create_task asana.asana_get_task asana.asana_update_task asana.asana_create_project clickup.clickup_create_task asana.asana_list_tasks okta.okta_create_user clickup.clickup_update_task github.create_issue filesystem.write_file +q060,Create a Trello card,verb_product,trello.trello_create_card,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,374,1.0,1,699,1.0,trello.trello_create_card trello.trello_add_comment trello.trello_create_board trello.trello_list_cards vercel.vercel_create_project stripe.stripe_create_invoice salesforce.salesforce_create_record stripe.stripe_create_customer slack.slack_create_channel stripe.stripe_create_product +q061,Create a ClickUp task,verb_product,clickup.clickup_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,415,1.0,1,724,1.0,clickup.clickup_create_task clickup.clickup_get_task clickup.clickup_update_task clickup.clickup_list_tasks asana.asana_create_task trello.trello_create_board vercel.vercel_create_project stripe.stripe_create_invoice github.create_gist stripe.stripe_create_customer +q062,Create a Confluence page,verb_product,confluence.confluence_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,401,1.0,1,713,1.0,confluence.confluence_create_page confluence.confluence_get_page confluence.confluence_update_page confluence.confluence_search notion.notion_create_page notion.notion_update_page stripe.stripe_create_invoice trello.trello_create_board stripe.stripe_create_price vercel.vercel_create_project +q063,Send a Discord message,verb_product,discord.discord_send_message,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,327,1.0,1,681,1.0,discord.discord_send_message discord.discord_add_reaction discord.discord_get_channel discord.discord_list_members gmail.gmail_get_email slack.slack_post_message discord.discord_list_channels gmail.gmail_trash_email git.git_commit datadog.datadog_create_monitor +q064,Search Elasticsearch,verb_product,elasticsearch.es_search,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,343,1.0,1,633,1.0,elasticsearch.es_search elasticsearch.es_get_doc elasticsearch.es_index_doc github.search_users confluence.confluence_search slack.slack_search_messages github.search_orgs zendesk.zendesk_search grafana.grafana_search_dashboards github.search_commits +q065,Get a Redis key,verb_product,redis.redis_get,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,281,1.0,1,567,1.0,redis.redis_get redis.redis_set redis.redis_delete redis.redis_keys jira.jira_get_issue figma.figma_get_file aws_s3.s3_get_object github.get_runner gitlab.get_release figma.figma_get_comments +q066,Find MongoDB documents,verb_product,mongodb.mongo_find,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,381,1.0,1,742,1.0,mongodb.mongo_find mongodb.mongo_insert mongodb.mongo_update mongodb.mongo_delete mongodb.mongo_list_collections elasticsearch.es_index_doc elasticsearch.es_get_doc figma.figma_get_comments github.get_discussion_comments filesystem.read_file +q067,List SQLite tables,verb_product,sqlite.sqlite_list_tables,1.0,1.0,1.0,1.0000,1.0000,1,1,151,1.0,1,243,1.0,1,480,1.0,sqlite.sqlite_list_tables sqlite.sqlite_describe_table mysql.mysql_list_tables sqlite.sqlite_query supabase.supabase_list_tables postgres.list_tables linear.linear_list_teams clickup.clickup_list_tasks salesforce.salesforce_list_opportunities stripe.stripe_list_invoices +q068,Describe a postgres table,verb_product,postgres.describe_table,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,274,1.0,1,640,1.0,postgres.describe_table sqlite.sqlite_describe_table postgres.explain_query postgres.list_schemas postgres.list_tables postgres.query supabase.supabase_insert supabase.supabase_update supabase.supabase_select slack.create_pin +q069,github_create_issue,identifier,github.create_issue,1.0,1.0,1.0,1.0000,0.9239,1,1,333,1.0,1,568,1.0,1,1120,1.0,github.create_issue gitlab.gitlab_create_issue bitbucket.bitbucket_create_issue github.issue_write github.issue_read github.update_issue_comment linear.linear_create_issue github.add_issue_comment jira.jira_create_issue github.create_hook +q070,createIssue,identifier,bitbucket.bitbucket_create_issue,0.0,1.0,1.0,0.3333,0.5805,3,1,332,1.0,1,579,1.0,1,963,1.0,bitbucket.bitbucket_create_issue gitlab.gitlab_create_issue github.create_issue linear.linear_create_issue jira.jira_create_issue linear.linear_create_comment github.issue_write trello.trello_create_board jira.jira_update_issue vercel.vercel_create_project +q071,list-directory,identifier,filesystem.list_directory,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,348,1.0,1,580,1.0,filesystem.list_directory filesystem.create_directory filesystem.list_directory_with_sizes filesystem.directory_tree github.get_file_contents filesystem.get_file_info linear.linear_list_teams jira.jira_list_projects linear.linear_list_projects datadog.datadog_list_monitors +q072,sequential thinking next thought,paraphrase,sequential_thinking.sequentialthinking,1.0,1.0,1.0,1.0000,1.0000,1,1,278,1.0,1,434,1.0,1,862,1.0,sequential_thinking.sequentialthinking filesystem.edit_file confluence.confluence_search playwright.browser_wait_for brave_search.brave_web_search playwright.browser_type brave_search.brave_local_search github.add_comment_to_pending_review github.list_notifications memory.add_observations +q073,What time is it in Tokyo,paraphrase,time.get_current_time,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,391,1.0,1,709,1.0,time.get_current_time time.convert_time playwright.browser_wait_for grafana.grafana_create_annotation google_calendar.gcal_list_events jira.jira_add_worklog notion.list_users_tokens slack.slack_create_channel sqlite.sqlite_list_tables filesystem.list_allowed_directories +q074,Convert time between timezones,verb_product,time.convert_time,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,319,1.0,1,687,1.0,time.convert_time time.get_current_time salesforce.salesforce_convert_lead git.git_diff memory.create_relations playwright.browser_wait_for grafana.grafana_create_annotation google_calendar.gcal_list_events jira.jira_add_worklog cloudflare.cf_list_zones +q075,Star a GitHub repo,verb_product,github.star_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,226,1.0,1,381,1.0,1,759,1.0,github.star_repository github.unstar_repository github.delete_repository github.list_tags github.get_tag github.get_label github.fork_repository github.list_labels github.list_branches github.list_releases +q076,Fork a repository,resource,github.fork_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,235,1.0,1,378,1.0,1,750,1.0,github.fork_repository github.delete_repository github.star_repository github.unstar_repository github.list_releases github.list_dependabot_alerts github.create_repository github.list_discussions github.list_branches github.list_tags +q077,Get the latest GitHub release,verb_product,github.get_latest_release,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,383,1.0,1,724,1.0,github.get_latest_release github.get_release_by_tag gitlab.get_release github.get_teams github.get_repository_tree github.get_tag github.get_gist github.get_label gitlab.delete_release github.projects_get +q078,Create a gist,resource,github.create_gist,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,340,1.0,1,737,1.0,github.create_gist trello.trello_create_board github.update_gist vercel.vercel_create_project stripe.stripe_create_invoice gmail.gmail_create_draft trello.trello_create_card stripe.stripe_create_customer grafana.grafana_create_annotation slack.slack_create_channel +q079,Search GitHub code,verb_product,github.search_code,1.0,1.0,1.0,1.0000,1.0000,1,1,191,1.0,1,341,1.0,1,725,1.0,github.search_code github.search_users github.search_commits github.search_orgs github.list_code_scanning_alerts github.get_code_scanning_alert github.search_pull_requests github.search_issues github.search_repositories hubspot.hubspot_search_contacts +q080,List GitHub notifications,verb_product,github.list_notifications,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,311,1.0,1,662,1.0,github.list_notifications github.mark_all_notifications_read github.actions_list github.list_tags github.list_gists github.projects_list github.list_discussions github.list_releases github.list_labels github.list_branches +q081,Assign Copilot to an issue,verb_product,github.assign_copilot_to_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,404,1.0,1,864,1.0,github.assign_copilot_to_issue linear.linear_assign_issue jira.jira_assign_issue jira.jira_transition_issue jira.jira_add_comment github.add_issue_comment linear.linear_create_comment github.create_issue jira.jira_update_issue jira.jira_add_worklog +q082,Get secret scanning alerts,resource,github.list_secret_scanning_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,295,1.0,1,470,1.0,1,733,1.0,github.list_secret_scanning_alerts github.get_secret_scanning_alert github.list_code_scanning_alerts github.get_code_scanning_alert github.list_dependabot_alerts github.get_hook gitlab.get_job gitlab.get_wiki github.get_runner github.get_package +q083,Create or update a file on GitHub,verb_product,github.create_or_update_file,1.0,1.0,1.0,1.0000,0.7602,1,1,369,1.0,1,585,1.0,1,1047,1.0,github.create_or_update_file github.projects_write github.label_write github.issue_write github.update_gist github.update_issue_comment github.create_gist slack.update_file filesystem.write_file github.update_pull_request +q084,List collaborators on a repo,resource,github.list_repository_collaborators,1.0,1.0,1.0,1.0000,1.0000,1,1,278,1.0,1,443,1.0,1,827,1.0,github.list_repository_collaborators github.list_commits trello.trello_list_cards github.get_discussion_comments github.list_branches github.list_labels github.list_discussions github.list_releases github.request_copilot_review github.list_tags +q085,Reply in a Slack thread,verb_product,slack.slack_reply_to_thread,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,355,1.0,1,627,1.0,slack.slack_reply_to_thread slack.slack_get_thread_replies slack.get_reminder slack.delete_reminder slack.get_canvase slack.get_pin slack.get_bookmark slack.get_file slack.delete_canvase slack.delete_pin +q086,Add an emoji reaction on Slack,verb_product,slack.slack_add_reaction,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,412,1.0,1,686,1.0,slack.slack_add_reaction discord.discord_add_reaction gmail.gmail_modify_email playwright.browser_click slack.get_reminder slack.delete_reminder slack.get_canvase slack.get_file slack.get_pin slack.get_bookmark +q087,Search Slack messages,verb_product,slack.slack_search_messages,1.0,1.0,1.0,1.0000,1.0000,1,1,232,1.0,1,338,1.0,1,654,1.0,slack.slack_search_messages gmail.gmail_search_emails slack.slack_get_channel_history twilio.twilio_list_messages confluence.confluence_search brave_search.brave_local_search github.search_users notion.notion_search elasticsearch.es_search google_drive.gdrive_search +q088,Create a Slack channel,verb_product,slack.slack_create_channel,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,339,1.0,1,758,1.0,slack.slack_create_channel slack.slack_join_channel slack.slack_leave_channel slack.slack_set_channel_topic slack.slack_post_message slack.slack_get_thread_replies slack.slack_reply_to_thread slack.slack_get_channel_history slack.create_pin slack.slack_upload_file +q089,Transition a Jira issue,verb_product,jira.jira_transition_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,342,1.0,1,777,1.0,jira.jira_transition_issue jira.jira_get_issue jira.jira_assign_issue jira.jira_update_issue jira.jira_add_comment jira.jira_create_issue jira.jira_add_worklog sentry.sentry_get_issue sentry.sentry_update_issue jira.jira_search_issues +q090,Search Jira with JQL,identifier,jira.jira_search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,195,1.0,1,302,1.0,1,635,1.0,jira.jira_search_issues github.search_users github.search_orgs confluence.confluence_search github.search_commits slack.slack_search_messages grafana.grafana_search_dashboards github.search_pull_requests github.search_issues elasticsearch.es_search +q091,Add a worklog to Jira,verb_product,jira.jira_add_worklog,1.0,1.0,1.0,1.0000,1.0000,1,1,247,1.0,1,404,1.0,1,756,1.0,jira.jira_add_worklog jira.jira_add_comment linear.linear_create_comment jira.jira_transition_issue discord.discord_add_reaction github.add_issue_comment jira.get_webhook jira.get_board slack.slack_add_reaction jira.delete_webhook +q092,List Linear teams,verb_product,linear.linear_list_teams,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,330,1.0,1,690,1.0,linear.linear_list_teams linear.linear_list_projects linear.linear_list_issues linear.list_labels linear.list_cycles linear.list_users linear.list_comments linear.list_attachments linear.linear_create_project linear.linear_search_issues +q093,Comment on a Linear issue,verb_product,linear.linear_create_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,360,1.0,1,767,1.0,linear.linear_create_comment linear.get_comment linear.delete_comment github.update_issue_comment linear.update_comment linear.linear_get_issue linear.linear_assign_issue linear.linear_update_issue jira.jira_add_comment linear.create_comment +q094,Move a local file,paraphrase,filesystem.read_text_file,0.0,1.0,1.0,0.3333,0.5000,3,1,224,1.0,1,338,1.0,1,675,1.0,filesystem.read_text_file filesystem.read_file filesystem.move_file filesystem.read_media_file gmail.gmail_trash_email slack.delete_file google_drive.gdrive_share_file figma.figma_get_file google_drive.gdrive_update_file slack.get_file +q095,Recursively search files by pattern,paraphrase,filesystem.search_files,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,338,1.0,1,655,1.0,filesystem.search_files google_drive.gdrive_search linear.linear_search_issues redis.redis_keys github.search_repositories slack.list_files github.search_users confluence.confluence_search notion.notion_search slack.slack_search_messages +q096,Edit a file in place with old and new string,paraphrase,filesystem.edit_file,1.0,1.0,1.0,1.0000,1.0000,1,1,230,1.0,1,353,1.0,1,729,1.0,filesystem.edit_file filesystem.list_directory_with_sizes slack.get_file slack.delete_file filesystem.write_file slack.update_file slack.create_file figma.figma_get_file github.get_file_blame filesystem.move_file +q097,Read the whole knowledge graph,paraphrase,memory.read_graph,1.0,1.0,1.0,1.0000,1.0000,1,1,152,1.0,1,253,1.0,1,561,1.0,memory.read_graph memory.delete_relations memory.create_entities memory.create_relations memory.open_nodes memory.add_observations memory.search_nodes memory.delete_entities filesystem.read_file filesystem.read_text_file +q098,Search memory nodes,verb_product,memory.search_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,176,1.0,1,299,1.0,1,618,1.0,memory.search_nodes memory.open_nodes github.search_users slack.slack_search_messages confluence.confluence_search notion.notion_search zendesk.zendesk_search linear.linear_search_issues github.search_orgs github.search_issues +q099,Show unstaged git diff,verb_product,git.git_diff_unstaged,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,313,1.0,1,613,1.0,git.git_diff_unstaged git.git_diff_staged git.git_diff git.git_show git.git_log git.git_status git.git_reset git.git_commit git.git_add github.unstar_repository +q100,Stage files with git add,identifier,git.git_add,1.0,1.0,1.0,1.0000,1.0000,1,1,220,1.0,1,387,1.0,1,762,1.0,git.git_add filesystem.list_directory_with_sizes slack.list_files salesforce.salesforce_list_opportunities github.push_files github.create_gist github.update_gist github.add_issue_comment git.git_reset git.git_commit +q101,Create a Shopify product,verb_product,shopify.shopify_create_product,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,377,1.0,1,690,1.0,shopify.shopify_create_product shopify.shopify_get_product shopify.shopify_update_product shopify.shopify_create_order stripe.stripe_create_product shopify.shopify_list_products shopify.shopify_get_order stripe.stripe_create_price vercel.vercel_create_project stripe.stripe_create_invoice +q102,Cancel a Stripe subscription,verb_product,stripe.stripe_cancel_subscription,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,301,1.0,1,663,1.0,stripe.stripe_cancel_subscription stripe.stripe_create_invoice stripe.stripe_list_subscriptions stripe.stripe_create_product stripe.stripe_get_customer stripe.stripe_create_customer stripe.stripe_get_payment_intent stripe.stripe_create_refund stripe.stripe_create_price stripe.stripe_create_payment_intent +q103,Create a payment intent,resource,stripe.stripe_create_payment_intent,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,339,1.0,1,737,1.0,stripe.stripe_create_payment_intent stripe.stripe_create_refund stripe.stripe_get_payment_intent stripe.stripe_create_invoice trello.trello_create_board notion.notion_create_database vercel.vercel_create_project github.create_environment stripe.stripe_create_customer confluence.confluence_create_page +q104,Acknowledge a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,405,1.0,1,665,1.0,pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls memory.read_graph github.delete_package memory.create_entities github.update_package +q105,Create a Datadog monitor,verb_product,datadog.datadog_create_monitor,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,374,1.0,1,711,1.0,datadog.datadog_create_monitor datadog.datadog_get_monitor datadog.datadog_list_monitors trello.trello_create_board vercel.vercel_create_project stripe.stripe_create_invoice notion.notion_create_database stripe.stripe_create_customer filesystem.create_directory github.create_repository +q106,Resolve a Sentry issue,paraphrase,sentry.sentry_get_issue,0.0,1.0,1.0,0.5000,0.6309,2,1,251,1.0,1,412,1.0,1,729,1.0,sentry.sentry_get_issue sentry.sentry_update_issue sentry.sentry_list_events pagerduty.pagerduty_resolve_incident sentry.sentry_list_issues jira.jira_assign_issue jira.jira_get_issue sentry.sentry_list_projects bitbucket.bitbucket_create_issue linear.linear_get_issue +q107,Scale a Kubernetes deployment,verb_product,kubernetes.k8s_scale_deployment,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,389,1.0,1,726,1.0,kubernetes.k8s_scale_deployment vercel.vercel_get_deployment kubernetes.k8s_list_deployments kubernetes.k8s_apply kubernetes.k8s_delete kubernetes.k8s_get_pod kubernetes.k8s_get_events kubernetes.k8s_list_services kubernetes.k8s_list_pods kubernetes.k8s_pod_logs +q108,Apply a Kubernetes manifest,verb_product,kubernetes.k8s_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,330,1.0,1,727,1.0,kubernetes.k8s_apply kubernetes.k8s_get_pod kubernetes.k8s_list_services kubernetes.k8s_list_deployments kubernetes.k8s_list_pods kubernetes.k8s_delete kubernetes.k8s_get_events terraform.tf_apply kubernetes.k8s_scale_deployment kubernetes.k8s_pod_logs +q109,Stop a docker container,verb_product,docker.docker_stop,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,345,1.0,1,655,1.0,docker.docker_stop docker.docker_rm docker.docker_logs docker.docker_run docker.docker_exec docker.docker_ps docker.docker_images docker.docker_build kubernetes.k8s_pod_logs stripe.stripe_create_customer +q110,Build a docker image,verb_product,docker.docker_build,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,317,1.0,1,598,1.0,docker.docker_build docker.docker_run docker.docker_images docker.docker_stop docker.docker_rm docker.docker_ps docker.docker_logs docker.docker_exec filesystem.read_media_file jira.delete_filter +q111,Click an element in Playwright,verb_product,playwright.browser_click,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,329,1.0,1,564,1.0,playwright.browser_click puppeteer.puppeteer_click playwright.browser_type playwright.browser_take_screenshot playwright.browser_snapshot playwright.browser_tab_close playwright.browser_tab_list playwright.browser_navigate playwright.browser_tab_new playwright.browser_fill_form +q112,Type text into a browser input,paraphrase,playwright.browser_type,1.0,1.0,1.0,1.0000,0.8801,1,1,217,1.0,1,333,1.0,1,609,1.0,playwright.browser_type playwright.browser_wait_for playwright.browser_navigate playwright.browser_tab_close supabase.supabase_insert playwright.browser_tab_list playwright.browser_tab_new puppeteer.puppeteer_fill slack.slack_post_message playwright.browser_press_key +q113,List GitLab projects,verb_product,gitlab.gitlab_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,209,1.0,1,289,1.0,1,597,1.0,gitlab.gitlab_list_projects github.projects_list gitlab.gitlab_list_issues jira.jira_list_projects linear.linear_list_projects vercel.vercel_list_projects gitlab.gitlab_list_merge_requests sentry.sentry_list_projects gitlab.gitlab_get_merge_request github.projects_get +q114,Create a Bitbucket pull request,verb_product,bitbucket.bitbucket_create_pr,1.0,1.0,1.0,1.0000,1.0000,1,1,313,1.0,1,546,1.0,1,1085,1.0,bitbucket.bitbucket_create_pr bitbucket.bitbucket_list_prs bitbucket.bitbucket_create_issue github.pull_request_review_write github.pull_request_read github.create_pull_request github.merge_pull_request github.update_pull_request_branch github.list_pull_requests github.request_copilot_review +q115,Create a Gmail draft,verb_product,gmail.gmail_create_draft,1.0,1.0,1.0,1.0000,1.0000,1,1,225,1.0,1,340,1.0,1,739,1.0,gmail.gmail_create_draft stripe.stripe_create_customer gmail.gmail_trash_email trello.trello_create_board vercel.vercel_create_project github.create_pull_request shopify.shopify_create_order stripe.stripe_create_invoice gmail.gmail_get_email github.create_gist +q116,Share a Google Drive file,verb_product,google_drive.gdrive_share_file,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,418,1.0,1,724,1.0,google_drive.gdrive_share_file google_drive.gdrive_update_file google_drive.gdrive_get_file google_drive.gdrive_list_folder google_drive.gdrive_create_file google_drive.gdrive_search google_calendar.gcal_delete_event google_calendar.gcal_update_event slack.delete_file google_calendar.gcal_list_calendars +q117,List Google Calendar events,verb_product,google_calendar.gcal_list_events,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,427,1.0,1,697,1.0,google_calendar.gcal_list_events google_calendar.gcal_list_calendars google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event kubernetes.k8s_get_events sentry.sentry_list_events google_drive.gdrive_list_folder google_drive.gdrive_get_file linear.linear_list_teams +q118,Delete a calendar event,resource,google_calendar.gcal_delete_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,440,1.0,1,685,1.0,google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event google_calendar.gcal_list_events github.delete_environment linear.delete_comment github.delete_hook gitlab.delete_job linear.delete_cycle google_calendar.gcal_list_calendars +q119,List Vercel projects,verb_product,vercel.vercel_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,151,1.0,1,248,1.0,1,550,1.0,vercel.vercel_list_projects vercel.vercel_list_deployments jira.jira_list_projects linear.linear_list_projects sentry.sentry_list_projects vercel.vercel_create_project gitlab.gitlab_list_projects github.projects_list github.projects_get vercel.vercel_get_deployment +q120,Select rows from Supabase,verb_product,supabase.supabase_select,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,354,1.0,1,664,1.0,supabase.supabase_select supabase.supabase_insert supabase.supabase_update supabase.supabase_list_tables supabase.supabase_invoke_fn aws_s3.s3_get_object mongodb.mongo_delete memory.delete_observations memory.delete_relations docker.docker_logs +q121,Post a Figma comment,verb_product,figma.figma_post_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,201,1.0,1,308,1.0,1,649,1.0,figma.figma_post_comment figma.figma_get_comments linear.get_comment figma.figma_get_file notion.get_comment linear.delete_comment linear.update_comment notion.delete_comment notion.update_comment github.update_issue_comment +q122,List Okta users,verb_product,okta.okta_list_users,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,288,1.0,1,618,1.0,okta.okta_list_users okta.okta_create_user okta.okta_get_user linear.list_users notion.notion_list_users slack.slack_get_users okta.okta_deactivate_user slack.slack_list_users notion.list_users_tokens pagerduty.pagerduty_list_oncalls +q123,Range query Prometheus,verb_product,prometheus.prom_query_range,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,377,1.0,1,660,1.0,prometheus.prom_query_range prometheus.prom_query prometheus.prom_list_metrics datadog.datadog_query_metrics postgres.explain_query shopify.shopify_list_products mysql.mysql_query sqlite.sqlite_query github.search_users memory.search_nodes +q124,Create a Grafana annotation,verb_product,grafana.grafana_create_annotation,1.0,1.0,1.0,1.0000,1.0000,1,1,252,1.0,1,391,1.0,1,782,1.0,grafana.grafana_create_annotation github.create_gist stripe.stripe_create_product grafana.grafana_get_dashboard notion.notion_create_database asana.asana_create_project linear.linear_create_project gmail.gmail_create_draft github.create_artifact trello.trello_create_board +q125,Apply terraform,identifier,terraform.tf_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,327,1.0,1,644,1.0,terraform.tf_apply terraform.tf_plan kubernetes.k8s_apply terraform.tf_state_list slack.slack_reply_to_thread playwright.browser_fill_form filesystem.search_files linear.linear_list_issues docker.docker_rm linear.linear_search_issues +q126,List Asana tasks,verb_product,asana.asana_list_tasks,1.0,1.0,1.0,1.0000,1.0000,1,1,192,1.0,1,386,1.0,1,612,1.0,asana.asana_list_tasks clickup.clickup_list_tasks asana.asana_get_task asana.asana_update_task asana.asana_create_task linear.linear_list_teams aws_s3.s3_list_buckets linear.linear_list_projects asana.asana_create_project jira.jira_list_projects +q127,Comment on a Trello card,verb_product,trello.trello_add_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,390,1.0,1,703,1.0,trello.trello_add_comment trello.trello_create_card trello.trello_list_cards trello.trello_create_board figma.figma_post_comment github.update_issue_comment notion.get_comment linear.get_comment notion.delete_comment linear.delete_comment +q128,Update a ClickUp task,verb_product,clickup.clickup_update_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,399,1.0,1,736,1.0,clickup.clickup_update_task clickup.clickup_get_task clickup.clickup_create_task asana.asana_update_task clickup.clickup_list_tasks github.update_hook linear.update_cycle gitlab.update_job shopify.shopify_update_product linear.update_comment +q129,Search Confluence,verb_product,confluence.confluence_search,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,362,1.0,1,682,1.0,confluence.confluence_search confluence.confluence_get_page confluence.confluence_update_page confluence.confluence_create_page github.search_users github.search_commits hubspot.hubspot_search_contacts slack.slack_search_messages brave_search.brave_web_search github.search_orgs +q130,Create a HubSpot deal,verb_product,hubspot.hubspot_create_deal,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,413,1.0,1,696,1.0,hubspot.hubspot_create_deal hubspot.hubspot_create_contact hubspot.hubspot_get_contact github.create_gist github.create_repository trello.trello_create_board vercel.vercel_create_project stripe.stripe_create_product stripe.stripe_create_invoice hubspot.hubspot_list_contacts +q131,List Zendesk tickets,verb_product,zendesk.zendesk_list_tickets,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,367,1.0,1,575,1.0,zendesk.zendesk_list_tickets zendesk.zendesk_search zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_create_ticket aws_s3.s3_list_buckets linear.linear_list_teams clickup.clickup_list_tasks jira.jira_list_projects linear.linear_list_projects +q132,List Discord channels,verb_product,discord.discord_list_channels,1.0,1.0,1.0,1.0000,1.0000,1,1,166,1.0,1,300,1.0,1,577,1.0,discord.discord_list_channels discord.discord_list_members discord.discord_get_channel discord.discord_send_message slack.slack_list_channels discord.discord_add_reaction gmail.gmail_list_labels linear.linear_list_teams linear.linear_list_projects trello.trello_list_cards +q133,Set a Redis key,verb_product,redis.redis_set,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,281,1.0,1,600,1.0,redis.redis_set redis.redis_get redis.redis_delete redis.redis_keys jira.jira_get_issue figma.figma_get_file slack.slack_set_channel_topic aws_s3.s3_get_object aws_s3.s3_delete_object playwright.browser_press_key +q134,Insert MongoDB documents,verb_product,mongodb.mongo_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,381,1.0,1,751,1.0,mongodb.mongo_insert mongodb.mongo_find mongodb.mongo_delete mongodb.mongo_update mongodb.mongo_list_collections supabase.supabase_insert elasticsearch.es_index_doc elasticsearch.es_get_doc linear.list_comments notion.list_comments +q135,Run a MySQL query,verb_product,mysql.mysql_query,1.0,1.0,1.0,1.0000,1.0000,1,1,179,1.0,1,294,1.0,1,585,1.0,mysql.mysql_query postgres.query sqlite.sqlite_query postgres.explain_query salesforce.salesforce_query mysql.mysql_list_tables prometheus.prom_query github.search_users notion.notion_query_database memory.search_nodes +q136,Index a document in Elasticsearch,verb_product,elasticsearch.es_index_doc,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,344,1.0,1,597,1.0,elasticsearch.es_index_doc elasticsearch.es_get_doc elasticsearch.es_search github.get_environment github.delete_environment playwright.browser_tab_close jira.delete_attachment linear.get_comment linear.delete_comment jira.get_attachment +q137,Get an S3 object,verb_product,aws_s3.s3_get_object,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,375,1.0,1,678,1.0,aws_s3.s3_get_object aws_s3.s3_delete_object aws_s3.s3_put_object okta.okta_get_user aws_s3.s3_list_objects asana.asana_get_task gmail.gmail_send_email github.get_artifact github.get_hook gitlab.get_job +q138,List Cloudflare zones,verb_product,cloudflare.cf_list_zones,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,338,1.0,1,554,1.0,cloudflare.cf_list_zones cloudflare.cf_list_dns cloudflare.cf_purge_cache cloudflare.cf_create_dns clickup.clickup_list_tasks linear.linear_list_teams linear.linear_list_projects okta.okta_list_users kubernetes.k8s_list_services gmail.gmail_list_labels +q139,Create a Stripe invoice,verb_product,stripe.stripe_create_invoice,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,353,1.0,1,680,1.0,stripe.stripe_create_invoice stripe.stripe_create_customer stripe.stripe_create_product stripe.stripe_create_price stripe.stripe_create_refund stripe.stripe_create_payment_intent stripe.stripe_list_invoices stripe.stripe_get_customer slack.slack_create_channel salesforce.salesforce_create_record +q140,List GitHub Actions workflows,verb_product,github.actions_list,1.0,1.0,1.0,1.0000,1.0000,1,1,293,1.0,1,438,1.0,1,783,1.0,github.actions_list github.actions_get github.actions_run_trigger github.actions_get_job_logs github.list_gists github.list_discussions github.list_tags github.projects_list github.list_releases github.list_notifications +q141,Get GitHub Actions job logs,verb_product,github.actions_get_job_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,442,1.0,1,802,1.0,github.actions_get_job_logs github.actions_get github.actions_list github.actions_run_trigger gitlab.get_job github.get_gist github.projects_get github.get_label github.get_discussion github.get_latest_release +q142,Create a GitHub repository,verb_product,github.create_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,268,1.0,1,406,1.0,1,853,1.0,github.create_repository github.delete_repository github.create_branch github.star_repository github.unstar_repository github.fork_repository github.create_issue github.list_tags github.get_tag github.list_repository_collaborators +q143,Checkout a git branch,verb_product,git.git_checkout,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,407,1.0,1,926,1.0,git.git_checkout git.git_create_branch git.git_branch github.create_branch github.update_pull_request_branch github.delete_file github.list_commits gitlab.gitlab_create_merge_request github.push_files circleci.circleci_trigger_pipeline +q144,Show git log,identifier,git.git_log,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,299,1.0,1,603,1.0,git.git_log git.git_show git.git_diff_unstaged git.git_diff_staged git.git_status git.git_diff git.git_branch git.git_create_branch git.git_checkout git.git_reset +q145,List postgres schemas,verb_product,postgres.list_schemas,1.0,1.0,1.0,1.0000,1.0000,1,1,160,1.0,1,277,1.0,1,488,1.0,postgres.list_schemas postgres.list_tables postgres.explain_query postgres.query postgres.describe_table linear.linear_list_teams aws_s3.s3_list_buckets kubernetes.k8s_list_services mysql.mysql_list_tables cloudflare.cf_list_zones +q146,Open nodes in the knowledge graph,verb_product,memory.open_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,262,1.0,1,492,1.0,memory.open_nodes memory.search_nodes memory.create_entities memory.read_graph memory.create_relations memory.delete_relations memory.add_observations memory.delete_entities postgres.list_schemas sqlite.sqlite_list_tables +q147,Get file metadata from the filesystem,paraphrase,filesystem.get_file_info,1.0,1.0,1.0,1.0000,1.0000,1,1,220,1.0,1,334,1.0,1,709,1.0,filesystem.get_file_info filesystem.read_file filesystem.read_text_file filesystem.read_media_file google_drive.gdrive_get_file gitlab.gitlab_get_file figma.figma_get_file_nodes filesystem.list_allowed_directories github.get_file_contents filesystem.move_file +q148,Write a new local file,paraphrase,filesystem.write_file,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,384,1.0,1,699,1.0,filesystem.write_file filesystem.read_file filesystem.read_text_file filesystem.read_media_file filesystem.edit_file slack.delete_file playwright.browser_tab_new figma.figma_get_file google_drive.gdrive_update_file slack.update_file +q149,Search GitHub issues,verb_product,github.search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,228,1.0,1,335,1.0,1,702,1.0,github.search_issues github.search_users jira.jira_search_issues linear.linear_search_issues github.search_orgs github.search_pull_requests github.search_commits github.list_issues github.search_code github.search_repositories +q150,Get the authenticated GitHub user,paraphrase,github.get_teams,0.0,1.0,1.0,0.5000,0.6309,2,1,179,1.0,1,356,1.0,1,694,1.0,github.get_teams github.get_me github.list_notifications github.get_latest_release github.get_repository_tree github.projects_get github.get_gist github.get_tag github.get_label github.list_gists diff --git a/experiments/mcp-tool-search/results/hybrid_rrf_bm25_trigram.json b/experiments/mcp-tool-search/results/hybrid_rrf_bm25_trigram.json new file mode 100644 index 00000000..f7612004 --- /dev/null +++ b/experiments/mcp-tool-search/results/hybrid_rrf_bm25_trigram.json @@ -0,0 +1,7013 @@ +{ + "summary": { + "n": 150, + "index_s": 0.0060391910000134885, + "rss_mb": 0.01171875, + "p50_ms": 0.5474685000308455, + "p95_ms": 0.7852027499495757, + "recall@1": 0.92, + "recall@1_lo": 0.8733333333333333, + "recall@1_hi": 0.96, + "recall@3": 0.98, + "recall@3_lo": 0.9533333333333334, + "recall@3_hi": 1.0, + "recall@5": 0.9866666666666667, + "recall@5_lo": 0.9666666666666667, + "recall@5_hi": 1.0, + "mrr@10": 0.9487407407407407, + "mrr@10_lo": 0.9192592592592592, + "mrr@10_hi": 0.9768888888888889, + "ndcg@10": 0.9555953414387945, + "ndcg@10_lo": 0.9313536952974558, + "ndcg@10_hi": 0.9782977920782063 + }, + "meta": { + "name": "hybrid_rrf_bm25_trigram", + "needs_model": false, + "needs_api": false, + "notes": "RRF of BM25 + char trigram" + }, + "rows": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "github.issue_write", + "gitlab.gitlab_create_issue", + "github.issue_read", + "github.create_gist", + "github.update_issue_comment", + "github.create_repository", + "github.create_branch", + "bitbucket.bitbucket_create_issue", + "github.add_issue_comment" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 380, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 557, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1038, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "kind": "verb_product", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_create_issue", + "zendesk.zendesk_create_ticket", + "jira.create_board", + "jira.create_sprint", + "jira.create_filter", + "jira.create_version", + "jira.create_webhook", + "zendesk.zendesk_update_ticket", + "jira.create_attachment", + "jira.jira_create_sprint" + ], + "hit1": "jira.jira_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 318, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 513, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1003, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "kind": "verb_product", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "ranking": [ + "slack.slack_post_message", + "discord.discord_send_message", + "slack.slack_add_reaction", + "slack.slack_search_messages", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_create_channel", + "slack.slack_get_user_profile", + "slack.slack_get_channel_history", + "slack.get_pin" + ], + "hit1": "slack.slack_post_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "kind": "resource", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "ranking": [ + "github.list_pull_requests", + "bitbucket.bitbucket_list_prs", + "github.search_pull_requests", + "github.create_pull_request", + "gitlab.gitlab_list_merge_requests", + "github.pull_request_read", + "github.merge_pull_request", + "playwright.browser_tab_list", + "github.request_copilot_review", + "github.update_pull_request_branch" + ], + "hit1": "github.list_pull_requests", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 460, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 887, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "ranking": [ + "github.create_pull_request", + "github.request_copilot_review", + "github.pull_request_read", + "github.merge_pull_request", + "github.search_pull_requests", + "github.update_pull_request_branch", + "github.pull_request_review_write", + "github.list_pull_requests", + "github.update_pull_request", + "github.add_comment_to_pending_review" + ], + "hit1": "github.create_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 364, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 522, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1144, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "kind": "verb_product", + "gold": [ + "postgres.query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "mysql.mysql_query", + "sqlite.sqlite_query", + "postgres.explain_query", + "salesforce.salesforce_query", + "postgres.list_schemas", + "postgres.list_tables", + "postgres.describe_table", + "docker.docker_run", + "prometheus.prom_query" + ], + "hit1": "postgres.query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 590, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "kind": "paraphrase", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "ranking": [ + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "gitlab.gitlab_get_file", + "figma.figma_get_file_nodes", + "github.delete_file", + "figma.figma_get_images", + "github.get_file_contents", + "figma.figma_get_file", + "slack.get_file" + ], + "hit1": "filesystem.read_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 806, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "kind": "paraphrase", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "ranking": [ + "google_drive.gdrive_list_folder", + "slack.list_files", + "google_drive.gdrive_search", + "cloudflare.cf_purge_cache", + "github.push_files", + "git.git_add", + "filesystem.search_files", + "github.get_milestone", + "filesystem.list_directory", + "github.delete_milestone" + ], + "hit1": "google_drive.gdrive_list_folder", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.1111111111111111, + "ndcg@10": 0.1922943146971571, + "hit_rank": 9, + "k3_calls": 3, + "k3_tokens": 679, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 712, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 690, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "ranking": [ + "memory.read_graph", + "memory.create_entities", + "memory.delete_relations", + "memory.open_nodes", + "memory.create_relations", + "memory.add_observations", + "memory.search_nodes", + "memory.delete_entities", + "time.get_current_time", + "postgres.list_schemas" + ], + "hit1": "memory.read_graph", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.5105967295264829, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 154, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 255, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 507, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "kind": "verb_product", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "ranking": [ + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "jira.jira_search_issues", + "github.update_pull_request_branch", + "memory.search_nodes", + "github.search_users", + "github.search_orgs", + "github.search_commits", + "confluence.confluence_search", + "slack.slack_search_messages" + ], + "hit1": "brave_search.brave_web_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 410, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 688, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "kind": "verb_product", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "ranking": [ + "fetch.fetch", + "filesystem.read_file", + "git.git_add", + "playwright.browser_navigate", + "docker.docker_logs", + "github.get_me", + "github.get_file_contents", + "git.git_status", + "slack.slack_set_channel_topic", + "playwright.browser_snapshot" + ], + "hit1": "fetch.fetch", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 222, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "kind": "identifier", + "gold": [ + "git.git_status" + ], + "alts": [], + "ranking": [ + "git.git_status", + "git.git_diff_staged", + "shopify.shopify_list_orders", + "git.git_diff_unstaged", + "github.list_hooks", + "git.git_show", + "gitlab.list_jobs", + "gitlab.list_snippets", + "git.git_log", + "github.list_packages" + ], + "hit1": "git.git_status", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "kind": "paraphrase", + "gold": [ + "git.git_commit" + ], + "alts": [], + "ranking": [ + "git.git_commit", + "git.git_reset", + "git.git_diff_staged", + "git.git_show", + "git.git_log", + "github.get_commit", + "github.push_files", + "linear.list_comments", + "notion.list_comments", + "github.list_packages" + ], + "hit1": "git.git_commit", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 305, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_comment", + "linear.linear_create_issue", + "linear.linear_update_issue", + "linear.linear_assign_issue", + "linear.linear_get_issue", + "linear.linear_create_project", + "linear.create_user", + "linear.create_label", + "linear.create_cycle", + "bitbucket.bitbucket_create_issue" + ], + "hit1": "linear.linear_create_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 921, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "kind": "paraphrase", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_project", + "linear.delete_user", + "linear.get_user", + "linear.delete_cycle", + "asana.asana_create_task", + "linear.get_cycle", + "linear.delete_label", + "linear.get_label", + "linear.delete_comment", + "linear.linear_list_projects" + ], + "hit1": "linear.linear_create_project", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 146, + "k3_calls": 49, + "k3_tokens": 12289, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 30, + "k5_tokens": 12029, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11669, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "ranking": [ + "github.update_issue_comment", + "figma.figma_post_comment", + "github.add_issue_comment", + "jira.jira_add_comment", + "linear.linear_create_comment", + "github.issue_read", + "github.assign_copilot_to_issue", + "github.get_discussion_comments", + "trello.trello_add_comment", + "github.list_commits" + ], + "hit1": "github.update_issue_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 307, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 442, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 912, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "kind": "ambiguous", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "ranking": [ + "github.merge_pull_request", + "gitlab.gitlab_merge_merge_request", + "github.update_pull_request_branch", + "gitlab.gitlab_get_merge_request", + "github.pull_request_read", + "github.request_copilot_review", + "github.search_pull_requests", + "gitlab.gitlab_create_merge_request", + "github.update_pull_request", + "bitbucket.bitbucket_list_prs" + ], + "hit1": "github.merge_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 299, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 468, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 957, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "kind": "verb_product", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "ranking": [ + "github.actions_run_trigger", + "github.actions_get", + "github.actions_list", + "github.actions_get_job_logs", + "github.star_repository", + "github.fork_repository", + "github.unstar_repository", + "github.search_orgs", + "github.delete_repository", + "github.list_tags" + ], + "hit1": "github.actions_run_trigger", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "kind": "resource", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "ranking": [ + "github.list_dependabot_alerts", + "github.get_dependabot_alert", + "github.list_secret_scanning_alerts", + "github.list_code_scanning_alerts", + "vercel.vercel_get_deployment", + "github.get_hook", + "zendesk.zendesk_get_ticket", + "gitlab.get_job", + "redis.redis_get", + "gitlab.get_wiki" + ], + "hit1": "github.list_dependabot_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 691, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "kind": "verb_product", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_issues", + "sentry.sentry_list_projects", + "sentry.sentry_list_events", + "gitlab.gitlab_list_issues", + "sentry.sentry_get_issue", + "linear.linear_list_issues", + "linear.linear_search_issues", + "sentry.sentry_update_issue", + "github.list_issues", + "github.search_issues" + ], + "hit1": "sentry.sentry_list_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_customer", + "stripe.stripe_create_invoice", + "stripe.stripe_get_customer", + "stripe.stripe_create_payment_intent", + "stripe.stripe_list_invoices", + "stripe.stripe_create_price", + "stripe.stripe_create_product", + "stripe.stripe_list_subscriptions", + "stripe.stripe_list_customers", + "stripe.stripe_create_refund" + ], + "hit1": "stripe.stripe_create_customer", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "kind": "paraphrase", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_refund", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_customer", + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_create_customer", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "stripe.stripe_cancel_subscription" + ], + "hit1": "stripe.stripe_create_refund", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 332, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "kind": "verb_product", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "ranking": [ + "gmail.gmail_send_email", + "gmail.gmail_get_email", + "twilio.twilio_send_sms", + "puppeteer.puppeteer_click", + "gmail.gmail_trash_email", + "gmail.gmail_search_emails", + "gmail.gmail_list_labels", + "puppeteer.puppeteer_fill", + "gmail.gmail_modify_email", + "gmail.gmail_create_draft" + ], + "hit1": "gmail.gmail_send_email", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 365, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "kind": "verb_product", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "ranking": [ + "gmail.gmail_search_emails", + "gmail.gmail_list_labels", + "github.search_users", + "confluence.confluence_search", + "elasticsearch.es_search", + "slack.slack_search_messages", + "gmail.gmail_get_email", + "github.search_orgs", + "gmail.gmail_send_email", + "github.search_commits" + ], + "hit1": "gmail.gmail_search_emails", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 174, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 615, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "kind": "paraphrase", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_update_file", + "google_drive.gdrive_share_file", + "google_drive.gdrive_search", + "google_drive.gdrive_get_file", + "google_drive.gdrive_create_file", + "slack.slack_upload_file", + "google_drive.gdrive_list_folder", + "aws_s3.s3_put_object", + "figma.figma_get_file", + "google_calendar.gcal_list_calendars" + ], + "hit1": "google_drive.gdrive_update_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.2, + "ndcg@10": 0.38685280723454163, + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 523, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 736, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_create_event", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_list_events", + "trello.trello_create_board", + "google_calendar.gcal_list_calendars", + "stripe.stripe_create_refund", + "vercel.vercel_create_project", + "github.pull_request_review_write", + "notion.notion_create_database" + ], + "hit1": "google_calendar.gcal_create_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 446, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 840, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_list_pods", + "kubernetes.k8s_list_services", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_get_events", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_apply", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_delete", + "kubernetes.k8s_scale_deployment", + "linear.linear_list_teams" + ], + "hit1": "kubernetes.k8s_list_pods", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_get_pod", + "docker.docker_logs", + "github.actions_get_job_logs", + "github.get_tag", + "figma.figma_get_file_nodes", + "elasticsearch.es_get_doc", + "datadog.datadog_list_logs", + "aws_s3.s3_get_object", + "slack.slack_get_channel_history" + ], + "hit1": "kubernetes.k8s_pod_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 441, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 830, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "ranking": [ + "docker.docker_run", + "docker.docker_stop", + "docker.docker_exec", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_ps", + "docker.docker_images", + "docker.docker_build", + "kubernetes.k8s_pod_logs", + "salesforce.salesforce_query" + ], + "hit1": "docker.docker_run", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "kind": "verb_product", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "ranking": [ + "docker.docker_logs", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_exec", + "docker.docker_run", + "docker.docker_ps", + "kubernetes.k8s_pod_logs", + "git.git_log", + "docker.docker_images", + "docker.docker_build" + ], + "hit1": "docker.docker_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 649, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "kind": "ambiguous", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "ranking": [ + "playwright.browser_navigate", + "puppeteer.puppeteer_navigate", + "playwright.browser_tab_new", + "playwright.browser_press_key", + "playwright.browser_evaluate", + "playwright.browser_wait_for", + "playwright.browser_click", + "playwright.browser_tab_close", + "playwright.browser_tab_list", + "playwright.browser_snapshot" + ], + "hit1": "playwright.browser_navigate", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 282, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 536, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "kind": "ambiguous", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "ranking": [ + "playwright.browser_take_screenshot", + "puppeteer.puppeteer_screenshot", + "playwright.browser_snapshot", + "playwright.browser_evaluate", + "playwright.browser_press_key", + "playwright.browser_click", + "git.git_status", + "filesystem.read_file", + "notion.notion_update_page", + "confluence.confluence_get_page" + ], + "hit1": "playwright.browser_take_screenshot", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 612, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_create_merge_request", + "gitlab.gitlab_get_merge_request", + "github.merge_pull_request", + "gitlab.gitlab_list_merge_requests", + "gitlab.gitlab_create_pipeline", + "gitlab.gitlab_create_issue", + "bitbucket.bitbucket_create_pr", + "github.pull_request_review_write", + "github.create_pull_request" + ], + "hit1": "gitlab.gitlab_merge_merge_request", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1054, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "kind": "verb_product", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "ranking": [ + "notion.notion_search", + "notion.notion_get_page", + "notion.notion_update_page", + "notion.notion_get_comments", + "notion.notion_get_block", + "github.search_orgs", + "notion.notion_create_database", + "slack.slack_search_messages", + "notion.notion_list_users", + "github.search_users" + ], + "hit1": "notion.notion_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 614, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "kind": "verb_product", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "ranking": [ + "notion.notion_create_page", + "notion.notion_get_page", + "notion.notion_update_page", + "notion.notion_create_database", + "notion.notion_get_comments", + "notion.create_comment", + "confluence.confluence_create_page", + "notion.create_users_token", + "notion.create_data_source", + "notion.update_comment" + ], + "hit1": "notion.notion_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 832, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "kind": "verb_product", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "ranking": [ + "notion.notion_query_database", + "notion.notion_create_database", + "notion.notion_search", + "notion.notion_create_page", + "notion.notion_get_block", + "notion.notion_get_page", + "notion.notion_update_page", + "notion.delete_data_source", + "notion.get_data_source", + "notion.delete_comment" + ], + "hit1": "notion.notion_query_database", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 688, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_search", + "jira.jira_create_issue", + "trello.trello_create_board", + "bitbucket.bitbucket_create_issue", + "vercel.vercel_create_project", + "confluence.confluence_create_page" + ], + "hit1": "zendesk.zendesk_create_ticket", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 780, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_create_deal", + "hubspot.hubspot_search_contacts", + "github.create_gist", + "vercel.vercel_create_project", + "stripe.stripe_create_product", + "trello.trello_create_board", + "github.projects_write" + ], + "hit1": "hubspot.hubspot_create_contact", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "kind": "verb_product", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_convert_lead", + "salesforce.salesforce_create_lead", + "salesforce.salesforce_create_record", + "salesforce.salesforce_update_record", + "salesforce.salesforce_query", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_list_opportunities", + "salesforce.salesforce_get_record", + "stripe.stripe_create_invoice", + "vercel.vercel_create_project" + ], + "hit1": "salesforce.salesforce_convert_lead", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 710, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "kind": "identifier", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "postgres.query", + "prometheus.prom_query", + "postgres.explain_query", + "notion.notion_query_database", + "memory.search_nodes", + "github.search_users", + "confluence.confluence_search" + ], + "hit1": "salesforce.salesforce_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 592, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "kind": "verb_product", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "ranking": [ + "shopify.shopify_list_orders", + "shopify.shopify_list_customers", + "shopify.shopify_list_products", + "shopify.shopify_get_order", + "shopify.shopify_get_product", + "shopify.shopify_create_order", + "shopify.shopify_update_product", + "shopify.shopify_create_product", + "okta.okta_list_users", + "linear.linear_list_teams" + ], + "hit1": "shopify.shopify_list_orders", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 638, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "stripe.stripe_create_invoice", + "stripe.stripe_create_refund", + "notion.notion_create_database", + "trello.trello_create_board", + "vercel.vercel_create_project" + ], + "hit1": "pagerduty.pagerduty_create_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "kind": "paraphrase", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_list_oncalls", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_create_incident", + "notion.notion_get_comments", + "playwright.browser_press_key", + "playwright.browser_evaluate", + "github.list_commits" + ], + "hit1": "pagerduty.pagerduty_list_oncalls", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "kind": "verb_product", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "ranking": [ + "datadog.datadog_query_metrics", + "datadog.datadog_list_monitors", + "datadog.datadog_create_monitor", + "datadog.datadog_list_logs", + "datadog.datadog_get_monitor", + "prometheus.prom_list_metrics", + "prometheus.prom_query", + "notion.notion_query_database", + "prometheus.prom_query_range", + "postgres.explain_query" + ], + "hit1": "datadog.datadog_query_metrics", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 799, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "kind": "verb_product", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "ranking": [ + "datadog.datadog_list_logs", + "datadog.datadog_list_monitors", + "datadog.datadog_get_monitor", + "github.search_orgs", + "datadog.datadog_query_metrics", + "grafana.grafana_search_dashboards", + "github.search_users", + "confluence.confluence_search", + "datadog.datadog_create_monitor", + "notion.notion_search" + ], + "hit1": "datadog.datadog_list_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 395, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_list_buckets", + "linear.linear_list_teams", + "jira.jira_list_projects", + "linear.linear_list_projects", + "zendesk.zendesk_list_tickets", + "bitbucket.bitbucket_list_repos", + "vercel.vercel_list_projects", + "sentry.sentry_list_projects", + "aws_s3.s3_list_objects", + "gmail.gmail_list_labels" + ], + "hit1": "aws_s3.s3_list_buckets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 129, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 229, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 476, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "kind": "paraphrase", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_put_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_get_object", + "slack.slack_upload_file", + "gmail.gmail_send_email", + "twilio.twilio_send_sms", + "aws_s3.s3_list_objects", + "github.assign_copilot_to_issue", + "slack.slack_add_reaction", + "asana.asana_create_project" + ], + "hit1": "aws_s3.s3_put_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 426, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 850, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "cloudflare.cf_list_dns", + "cloudflare.cf_create_dns", + "playwright.browser_tab_close", + "playwright.browser_snapshot", + "postgres.list_tables", + "memory.read_graph", + "postgres.list_schemas", + "puppeteer.puppeteer_click" + ], + "hit1": "cloudflare.cf_purge_cache", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 548, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "salesforce.salesforce_create_record", + "cloudflare.cf_list_zones", + "cloudflare.cf_purge_cache", + "salesforce.salesforce_update_record", + "trello.trello_create_board", + "salesforce.salesforce_delete_record", + "vercel.vercel_create_project", + "stripe.stripe_create_product" + ], + "hit1": "cloudflare.cf_create_dns", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_deployments", + "vercel.vercel_list_projects", + "vercel.vercel_get_deployment", + "kubernetes.k8s_list_deployments", + "vercel.vercel_create_project", + "linear.linear_list_teams", + "linear.linear_list_projects", + "jira.jira_list_projects", + "aws_s3.s3_list_buckets", + "sentry.sentry_list_projects" + ], + "hit1": "vercel.vercel_list_deployments", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 497, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "ranking": [ + "supabase.supabase_insert", + "mongodb.mongo_insert", + "supabase.supabase_list_tables", + "supabase.supabase_select", + "supabase.supabase_update", + "supabase.supabase_invoke_fn", + "playwright.browser_type", + "gitlab.get_release", + "slack.get_canvase", + "gitlab.delete_release" + ], + "hit1": "supabase.supabase_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 203, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 672, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "kind": "paraphrase", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "ranking": [ + "figma.figma_get_images", + "figma.figma_get_file_nodes", + "docker.docker_images", + "figma.figma_get_file", + "figma.figma_get_comments", + "figma.figma_post_comment", + "supabase.supabase_select", + "docker.docker_logs", + "mongodb.mongo_delete", + "kubernetes.k8s_pod_logs" + ], + "hit1": "figma.figma_get_images", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "kind": "verb_product", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "ranking": [ + "twilio.twilio_send_sms", + "twilio.twilio_list_messages", + "gmail.gmail_send_email", + "puppeteer.puppeteer_click", + "puppeteer.puppeteer_fill", + "twilio.twilio_make_call", + "filesystem.list_directory_with_sizes", + "discord.discord_send_message", + "jira.jira_search_issues", + "puppeteer.puppeteer_screenshot" + ], + "hit1": "twilio.twilio_send_sms", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "kind": "verb_product", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "ranking": [ + "okta.okta_create_user", + "okta.okta_get_user", + "okta.okta_deactivate_user", + "okta.okta_list_users", + "linear.create_user", + "asana.asana_create_project", + "linear.update_user", + "linear.delete_user", + "asana.asana_create_task", + "github.create_issue" + ], + "hit1": "okta.okta_create_user", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query", + "prometheus.prom_query_range", + "prometheus.prom_list_metrics", + "mysql.mysql_query", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "postgres.explain_query", + "datadog.datadog_query_metrics", + "postgres.query", + "shopify.shopify_list_products" + ], + "hit1": "prometheus.prom_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 666, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "kind": "verb_product", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "ranking": [ + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "github.search_users", + "github.search_orgs", + "confluence.confluence_search", + "grafana.grafana_create_annotation", + "slack.slack_search_messages", + "github.search_pull_requests", + "github.search_issues", + "brave_search.brave_web_search" + ], + "hit1": "grafana.grafana_search_dashboards", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 288, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 670, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "kind": "verb_product", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "ranking": [ + "circleci.circleci_trigger_pipeline", + "circleci.circleci_get_pipeline", + "gitlab.gitlab_create_pipeline", + "circleci.circleci_list_pipelines", + "gitlab.get_pipeline", + "gitlab.delete_pipeline", + "github.actions_run_trigger", + "gitlab.update_pipeline", + "hubspot.hubspot_create_deal", + "gitlab.create_pipeline" + ], + "hit1": "circleci.circleci_trigger_pipeline", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 325, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "kind": "identifier", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "ranking": [ + "terraform.tf_plan", + "terraform.tf_apply", + "terraform.tf_state_list", + "salesforce.salesforce_query", + "postgres.explain_query", + "github.get_check_run", + "github.delete_check_run", + "mysql.mysql_query", + "sqlite.sqlite_query", + "docker.docker_run" + ], + "hit1": "terraform.tf_plan", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 287, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 593, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "kind": "verb_product", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "ranking": [ + "asana.asana_create_task", + "asana.asana_get_task", + "asana.asana_update_task", + "asana.asana_create_project", + "clickup.clickup_create_task", + "asana.asana_list_tasks", + "okta.okta_create_user", + "clickup.clickup_update_task", + "github.create_issue", + "filesystem.write_file" + ], + "hit1": "asana.asana_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 841, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "ranking": [ + "trello.trello_create_card", + "trello.trello_add_comment", + "trello.trello_create_board", + "trello.trello_list_cards", + "vercel.vercel_create_project", + "stripe.stripe_create_invoice", + "salesforce.salesforce_create_record", + "stripe.stripe_create_customer", + "slack.slack_create_channel", + "stripe.stripe_create_product" + ], + "hit1": "trello.trello_create_card", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 374, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_create_task", + "clickup.clickup_get_task", + "clickup.clickup_update_task", + "clickup.clickup_list_tasks", + "asana.asana_create_task", + "trello.trello_create_board", + "vercel.vercel_create_project", + "stripe.stripe_create_invoice", + "github.create_gist", + "stripe.stripe_create_customer" + ], + "hit1": "clickup.clickup_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 415, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "kind": "verb_product", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "ranking": [ + "confluence.confluence_create_page", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "confluence.confluence_search", + "notion.notion_create_page", + "notion.notion_update_page", + "stripe.stripe_create_invoice", + "trello.trello_create_board", + "stripe.stripe_create_price", + "vercel.vercel_create_project" + ], + "hit1": "confluence.confluence_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "kind": "verb_product", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "discord.discord_add_reaction", + "discord.discord_get_channel", + "discord.discord_list_members", + "gmail.gmail_get_email", + "slack.slack_post_message", + "discord.discord_list_channels", + "gmail.gmail_trash_email", + "git.git_commit", + "datadog.datadog_create_monitor" + ], + "hit1": "discord.discord_send_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 681, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_search", + "elasticsearch.es_get_doc", + "elasticsearch.es_index_doc", + "github.search_users", + "confluence.confluence_search", + "slack.slack_search_messages", + "github.search_orgs", + "zendesk.zendesk_search", + "grafana.grafana_search_dashboards", + "github.search_commits" + ], + "hit1": "elasticsearch.es_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 633, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "redis.redis_set", + "redis.redis_delete", + "redis.redis_keys", + "jira.jira_get_issue", + "figma.figma_get_file", + "aws_s3.s3_get_object", + "github.get_runner", + "gitlab.get_release", + "figma.figma_get_comments" + ], + "hit1": "redis.redis_get", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 567, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_find", + "mongodb.mongo_insert", + "mongodb.mongo_update", + "mongodb.mongo_delete", + "mongodb.mongo_list_collections", + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "figma.figma_get_comments", + "github.get_discussion_comments", + "filesystem.read_file" + ], + "hit1": "mongodb.mongo_find", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "kind": "verb_product", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "ranking": [ + "sqlite.sqlite_list_tables", + "sqlite.sqlite_describe_table", + "mysql.mysql_list_tables", + "sqlite.sqlite_query", + "supabase.supabase_list_tables", + "postgres.list_tables", + "linear.linear_list_teams", + "clickup.clickup_list_tasks", + "salesforce.salesforce_list_opportunities", + "stripe.stripe_list_invoices" + ], + "hit1": "sqlite.sqlite_list_tables", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 243, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 480, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "kind": "verb_product", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "ranking": [ + "postgres.describe_table", + "sqlite.sqlite_describe_table", + "postgres.explain_query", + "postgres.list_schemas", + "postgres.list_tables", + "postgres.query", + "supabase.supabase_insert", + "supabase.supabase_update", + "supabase.supabase_select", + "slack.create_pin" + ], + "hit1": "postgres.describe_table", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 274, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 640, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "gitlab.gitlab_create_issue", + "bitbucket.bitbucket_create_issue", + "github.issue_write", + "github.issue_read", + "github.update_issue_comment", + "linear.linear_create_issue", + "github.add_issue_comment", + "jira.jira_create_issue", + "github.create_hook" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9238850086262381, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 568, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1120, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "ranking": [ + "bitbucket.bitbucket_create_issue", + "gitlab.gitlab_create_issue", + "github.create_issue", + "linear.linear_create_issue", + "jira.jira_create_issue", + "linear.linear_create_comment", + "github.issue_write", + "trello.trello_create_board", + "jira.jira_update_issue", + "vercel.vercel_create_project" + ], + "hit1": "bitbucket.bitbucket_create_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5805078709398304, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 332, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 579, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 963, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "kind": "identifier", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "ranking": [ + "filesystem.list_directory", + "filesystem.create_directory", + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree", + "github.get_file_contents", + "filesystem.get_file_info", + "linear.linear_list_teams", + "jira.jira_list_projects", + "linear.linear_list_projects", + "datadog.datadog_list_monitors" + ], + "hit1": "filesystem.list_directory", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 580, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "kind": "paraphrase", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "ranking": [ + "sequential_thinking.sequentialthinking", + "filesystem.edit_file", + "confluence.confluence_search", + "playwright.browser_wait_for", + "brave_search.brave_web_search", + "playwright.browser_type", + "brave_search.brave_local_search", + "github.add_comment_to_pending_review", + "github.list_notifications", + "memory.add_observations" + ], + "hit1": "sequential_thinking.sequentialthinking", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 278, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 434, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 862, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "kind": "paraphrase", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "ranking": [ + "time.get_current_time", + "time.convert_time", + "playwright.browser_wait_for", + "grafana.grafana_create_annotation", + "google_calendar.gcal_list_events", + "jira.jira_add_worklog", + "notion.list_users_tokens", + "slack.slack_create_channel", + "sqlite.sqlite_list_tables", + "filesystem.list_allowed_directories" + ], + "hit1": "time.get_current_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "kind": "verb_product", + "gold": [ + "time.convert_time" + ], + "alts": [], + "ranking": [ + "time.convert_time", + "time.get_current_time", + "salesforce.salesforce_convert_lead", + "git.git_diff", + "memory.create_relations", + "playwright.browser_wait_for", + "grafana.grafana_create_annotation", + "google_calendar.gcal_list_events", + "jira.jira_add_worklog", + "cloudflare.cf_list_zones" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "kind": "verb_product", + "gold": [ + "github.star_repository" + ], + "alts": [], + "ranking": [ + "github.star_repository", + "github.unstar_repository", + "github.delete_repository", + "github.list_tags", + "github.get_tag", + "github.get_label", + "github.fork_repository", + "github.list_labels", + "github.list_branches", + "github.list_releases" + ], + "hit1": "github.star_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "kind": "resource", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "ranking": [ + "github.fork_repository", + "github.delete_repository", + "github.star_repository", + "github.unstar_repository", + "github.list_releases", + "github.list_dependabot_alerts", + "github.create_repository", + "github.list_discussions", + "github.list_branches", + "github.list_tags" + ], + "hit1": "github.fork_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 750, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "kind": "verb_product", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "ranking": [ + "github.get_latest_release", + "github.get_release_by_tag", + "gitlab.get_release", + "github.get_teams", + "github.get_repository_tree", + "github.get_tag", + "github.get_gist", + "github.get_label", + "gitlab.delete_release", + "github.projects_get" + ], + "hit1": "github.get_latest_release", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 383, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "kind": "resource", + "gold": [ + "github.create_gist" + ], + "alts": [], + "ranking": [ + "github.create_gist", + "trello.trello_create_board", + "github.update_gist", + "vercel.vercel_create_project", + "stripe.stripe_create_invoice", + "gmail.gmail_create_draft", + "trello.trello_create_card", + "stripe.stripe_create_customer", + "grafana.grafana_create_annotation", + "slack.slack_create_channel" + ], + "hit1": "github.create_gist", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "kind": "verb_product", + "gold": [ + "github.search_code" + ], + "alts": [], + "ranking": [ + "github.search_code", + "github.search_users", + "github.search_commits", + "github.search_orgs", + "github.list_code_scanning_alerts", + "github.get_code_scanning_alert", + "github.search_pull_requests", + "github.search_issues", + "github.search_repositories", + "hubspot.hubspot_search_contacts" + ], + "hit1": "github.search_code", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 191, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "kind": "verb_product", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "ranking": [ + "github.list_notifications", + "github.mark_all_notifications_read", + "github.actions_list", + "github.list_tags", + "github.list_gists", + "github.projects_list", + "github.list_discussions", + "github.list_releases", + "github.list_labels", + "github.list_branches" + ], + "hit1": "github.list_notifications", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 311, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "kind": "verb_product", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "ranking": [ + "github.assign_copilot_to_issue", + "linear.linear_assign_issue", + "jira.jira_assign_issue", + "jira.jira_transition_issue", + "jira.jira_add_comment", + "github.add_issue_comment", + "linear.linear_create_comment", + "github.create_issue", + "jira.jira_update_issue", + "jira.jira_add_worklog" + ], + "hit1": "github.assign_copilot_to_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 864, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "kind": "resource", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "ranking": [ + "github.list_secret_scanning_alerts", + "github.get_secret_scanning_alert", + "github.list_code_scanning_alerts", + "github.get_code_scanning_alert", + "github.list_dependabot_alerts", + "github.get_hook", + "gitlab.get_job", + "gitlab.get_wiki", + "github.get_runner", + "github.get_package" + ], + "hit1": "github.list_secret_scanning_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "ranking": [ + "github.create_or_update_file", + "github.projects_write", + "github.label_write", + "github.issue_write", + "github.update_gist", + "github.update_issue_comment", + "github.create_gist", + "slack.update_file", + "filesystem.write_file", + "github.update_pull_request" + ], + "hit1": "github.create_or_update_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 369, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 585, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1047, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "kind": "resource", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "ranking": [ + "github.list_repository_collaborators", + "github.list_commits", + "trello.trello_list_cards", + "github.get_discussion_comments", + "github.list_branches", + "github.list_labels", + "github.list_discussions", + "github.list_releases", + "github.request_copilot_review", + "github.list_tags" + ], + "hit1": "github.list_repository_collaborators", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 278, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 443, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 827, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "kind": "verb_product", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "ranking": [ + "slack.slack_reply_to_thread", + "slack.slack_get_thread_replies", + "slack.get_reminder", + "slack.delete_reminder", + "slack.get_canvase", + "slack.get_pin", + "slack.get_bookmark", + "slack.get_file", + "slack.delete_canvase", + "slack.delete_pin" + ], + "hit1": "slack.slack_reply_to_thread", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 627, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "kind": "verb_product", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "ranking": [ + "slack.slack_add_reaction", + "discord.discord_add_reaction", + "gmail.gmail_modify_email", + "playwright.browser_click", + "slack.get_reminder", + "slack.delete_reminder", + "slack.get_canvase", + "slack.get_file", + "slack.get_pin", + "slack.get_bookmark" + ], + "hit1": "slack.slack_add_reaction", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "kind": "verb_product", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "ranking": [ + "slack.slack_search_messages", + "gmail.gmail_search_emails", + "slack.slack_get_channel_history", + "twilio.twilio_list_messages", + "confluence.confluence_search", + "brave_search.brave_local_search", + "github.search_users", + "notion.notion_search", + "elasticsearch.es_search", + "google_drive.gdrive_search" + ], + "hit1": "slack.slack_search_messages", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 654, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "kind": "verb_product", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "ranking": [ + "slack.slack_create_channel", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_set_channel_topic", + "slack.slack_post_message", + "slack.slack_get_thread_replies", + "slack.slack_reply_to_thread", + "slack.slack_get_channel_history", + "slack.create_pin", + "slack.slack_upload_file" + ], + "hit1": "slack.slack_create_channel", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "kind": "verb_product", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_transition_issue", + "jira.jira_get_issue", + "jira.jira_assign_issue", + "jira.jira_update_issue", + "jira.jira_add_comment", + "jira.jira_create_issue", + "jira.jira_add_worklog", + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "jira.jira_search_issues" + ], + "hit1": "jira.jira_transition_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "kind": "identifier", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "ranking": [ + "jira.jira_search_issues", + "github.search_users", + "github.search_orgs", + "confluence.confluence_search", + "github.search_commits", + "slack.slack_search_messages", + "grafana.grafana_search_dashboards", + "github.search_pull_requests", + "github.search_issues", + "elasticsearch.es_search" + ], + "hit1": "jira.jira_search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 635, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "kind": "verb_product", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "ranking": [ + "jira.jira_add_worklog", + "jira.jira_add_comment", + "linear.linear_create_comment", + "jira.jira_transition_issue", + "discord.discord_add_reaction", + "github.add_issue_comment", + "jira.get_webhook", + "jira.get_board", + "slack.slack_add_reaction", + "jira.delete_webhook" + ], + "hit1": "jira.jira_add_worklog", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 756, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "kind": "verb_product", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "ranking": [ + "linear.linear_list_teams", + "linear.linear_list_projects", + "linear.linear_list_issues", + "linear.list_labels", + "linear.list_cycles", + "linear.list_users", + "linear.list_comments", + "linear.list_attachments", + "linear.linear_create_project", + "linear.linear_search_issues" + ], + "hit1": "linear.linear_list_teams", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 690, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "ranking": [ + "linear.linear_create_comment", + "linear.get_comment", + "linear.delete_comment", + "github.update_issue_comment", + "linear.update_comment", + "linear.linear_get_issue", + "linear.linear_assign_issue", + "linear.linear_update_issue", + "jira.jira_add_comment", + "linear.create_comment" + ], + "hit1": "linear.linear_create_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 767, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "kind": "paraphrase", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "ranking": [ + "filesystem.read_text_file", + "filesystem.read_file", + "filesystem.move_file", + "filesystem.read_media_file", + "gmail.gmail_trash_email", + "slack.delete_file", + "google_drive.gdrive_share_file", + "figma.figma_get_file", + "google_drive.gdrive_update_file", + "slack.get_file" + ], + "hit1": "filesystem.read_text_file", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "kind": "paraphrase", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "ranking": [ + "filesystem.search_files", + "google_drive.gdrive_search", + "linear.linear_search_issues", + "redis.redis_keys", + "github.search_repositories", + "slack.list_files", + "github.search_users", + "confluence.confluence_search", + "notion.notion_search", + "slack.slack_search_messages" + ], + "hit1": "filesystem.search_files", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 655, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "kind": "paraphrase", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "ranking": [ + "filesystem.edit_file", + "filesystem.list_directory_with_sizes", + "slack.get_file", + "slack.delete_file", + "filesystem.write_file", + "slack.update_file", + "slack.create_file", + "figma.figma_get_file", + "github.get_file_blame", + "filesystem.move_file" + ], + "hit1": "filesystem.edit_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "ranking": [ + "memory.read_graph", + "memory.delete_relations", + "memory.create_entities", + "memory.create_relations", + "memory.open_nodes", + "memory.add_observations", + "memory.search_nodes", + "memory.delete_entities", + "filesystem.read_file", + "filesystem.read_text_file" + ], + "hit1": "memory.read_graph", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 253, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 561, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "kind": "verb_product", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "memory.open_nodes", + "github.search_users", + "slack.slack_search_messages", + "confluence.confluence_search", + "notion.notion_search", + "zendesk.zendesk_search", + "linear.linear_search_issues", + "github.search_orgs", + "github.search_issues" + ], + "hit1": "memory.search_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 176, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 618, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "kind": "verb_product", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "ranking": [ + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_diff", + "git.git_show", + "git.git_log", + "git.git_status", + "git.git_reset", + "git.git_commit", + "git.git_add", + "github.unstar_repository" + ], + "hit1": "git.git_diff_unstaged", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 313, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 613, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "kind": "identifier", + "gold": [ + "git.git_add" + ], + "alts": [], + "ranking": [ + "git.git_add", + "filesystem.list_directory_with_sizes", + "slack.list_files", + "salesforce.salesforce_list_opportunities", + "github.push_files", + "github.create_gist", + "github.update_gist", + "github.add_issue_comment", + "git.git_reset", + "git.git_commit" + ], + "hit1": "git.git_add", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 220, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 387, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 762, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "kind": "verb_product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "ranking": [ + "shopify.shopify_create_product", + "shopify.shopify_get_product", + "shopify.shopify_update_product", + "shopify.shopify_create_order", + "stripe.stripe_create_product", + "shopify.shopify_list_products", + "shopify.shopify_get_order", + "stripe.stripe_create_price", + "vercel.vercel_create_project", + "stripe.stripe_create_invoice" + ], + "hit1": "shopify.shopify_create_product", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 690, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "kind": "verb_product", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "ranking": [ + "stripe.stripe_cancel_subscription", + "stripe.stripe_create_invoice", + "stripe.stripe_list_subscriptions", + "stripe.stripe_create_product", + "stripe.stripe_get_customer", + "stripe.stripe_create_customer", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_refund", + "stripe.stripe_create_price", + "stripe.stripe_create_payment_intent" + ], + "hit1": "stripe.stripe_cancel_subscription", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "kind": "resource", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_refund", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_invoice", + "trello.trello_create_board", + "notion.notion_create_database", + "vercel.vercel_create_project", + "github.create_environment", + "stripe.stripe_create_customer", + "confluence.confluence_create_page" + ], + "hit1": "stripe.stripe_create_payment_intent", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "memory.read_graph", + "github.delete_package", + "memory.create_entities", + "github.update_package" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "kind": "verb_product", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "ranking": [ + "datadog.datadog_create_monitor", + "datadog.datadog_get_monitor", + "datadog.datadog_list_monitors", + "trello.trello_create_board", + "vercel.vercel_create_project", + "stripe.stripe_create_invoice", + "notion.notion_create_database", + "stripe.stripe_create_customer", + "filesystem.create_directory", + "github.create_repository" + ], + "hit1": "datadog.datadog_create_monitor", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 374, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 711, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "kind": "paraphrase", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "ranking": [ + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "sentry.sentry_list_events", + "pagerduty.pagerduty_resolve_incident", + "sentry.sentry_list_issues", + "jira.jira_assign_issue", + "jira.jira_get_issue", + "sentry.sentry_list_projects", + "bitbucket.bitbucket_create_issue", + "linear.linear_get_issue" + ], + "hit1": "sentry.sentry_get_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_scale_deployment", + "vercel.vercel_get_deployment", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_apply", + "kubernetes.k8s_delete", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_get_events", + "kubernetes.k8s_list_services", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_pod_logs" + ], + "hit1": "kubernetes.k8s_scale_deployment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_apply", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_list_services", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_delete", + "kubernetes.k8s_get_events", + "terraform.tf_apply", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_pod_logs" + ], + "hit1": "kubernetes.k8s_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "ranking": [ + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_run", + "docker.docker_exec", + "docker.docker_ps", + "docker.docker_images", + "docker.docker_build", + "kubernetes.k8s_pod_logs", + "stripe.stripe_create_customer" + ], + "hit1": "docker.docker_stop", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 655, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "kind": "verb_product", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "ranking": [ + "docker.docker_build", + "docker.docker_run", + "docker.docker_images", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_ps", + "docker.docker_logs", + "docker.docker_exec", + "filesystem.read_media_file", + "jira.delete_filter" + ], + "hit1": "docker.docker_build", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 598, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "kind": "verb_product", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "ranking": [ + "playwright.browser_click", + "puppeteer.puppeteer_click", + "playwright.browser_type", + "playwright.browser_take_screenshot", + "playwright.browser_snapshot", + "playwright.browser_tab_close", + "playwright.browser_tab_list", + "playwright.browser_navigate", + "playwright.browser_tab_new", + "playwright.browser_fill_form" + ], + "hit1": "playwright.browser_click", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 564, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "kind": "paraphrase", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "ranking": [ + "playwright.browser_type", + "playwright.browser_wait_for", + "playwright.browser_navigate", + "playwright.browser_tab_close", + "supabase.supabase_insert", + "playwright.browser_tab_list", + "playwright.browser_tab_new", + "puppeteer.puppeteer_fill", + "slack.slack_post_message", + "playwright.browser_press_key" + ], + "hit1": "playwright.browser_type", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8800937667159342, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 609, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_list_projects", + "github.projects_list", + "gitlab.gitlab_list_issues", + "jira.jira_list_projects", + "linear.linear_list_projects", + "vercel.vercel_list_projects", + "gitlab.gitlab_list_merge_requests", + "sentry.sentry_list_projects", + "gitlab.gitlab_get_merge_request", + "github.projects_get" + ], + "hit1": "gitlab.gitlab_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 209, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 289, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "kind": "verb_product", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "ranking": [ + "bitbucket.bitbucket_create_pr", + "bitbucket.bitbucket_list_prs", + "bitbucket.bitbucket_create_issue", + "github.pull_request_review_write", + "github.pull_request_read", + "github.create_pull_request", + "github.merge_pull_request", + "github.update_pull_request_branch", + "github.list_pull_requests", + "github.request_copilot_review" + ], + "hit1": "bitbucket.bitbucket_create_pr", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 546, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1085, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "kind": "verb_product", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "ranking": [ + "gmail.gmail_create_draft", + "stripe.stripe_create_customer", + "gmail.gmail_trash_email", + "trello.trello_create_board", + "vercel.vercel_create_project", + "github.create_pull_request", + "shopify.shopify_create_order", + "stripe.stripe_create_invoice", + "gmail.gmail_get_email", + "github.create_gist" + ], + "hit1": "gmail.gmail_create_draft", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 225, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 739, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "kind": "verb_product", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_share_file", + "google_drive.gdrive_update_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_list_folder", + "google_drive.gdrive_create_file", + "google_drive.gdrive_search", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "slack.delete_file", + "google_calendar.gcal_list_calendars" + ], + "hit1": "google_drive.gdrive_share_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "kind": "verb_product", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "kubernetes.k8s_get_events", + "sentry.sentry_list_events", + "google_drive.gdrive_list_folder", + "google_drive.gdrive_get_file", + "linear.linear_list_teams" + ], + "hit1": "google_calendar.gcal_list_events", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 697, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_list_events", + "github.delete_environment", + "linear.delete_comment", + "github.delete_hook", + "gitlab.delete_job", + "linear.delete_cycle", + "google_calendar.gcal_list_calendars" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 440, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 685, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_projects", + "vercel.vercel_list_deployments", + "jira.jira_list_projects", + "linear.linear_list_projects", + "sentry.sentry_list_projects", + "vercel.vercel_create_project", + "gitlab.gitlab_list_projects", + "github.projects_list", + "github.projects_get", + "vercel.vercel_get_deployment" + ], + "hit1": "vercel.vercel_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 248, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "ranking": [ + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update", + "supabase.supabase_list_tables", + "supabase.supabase_invoke_fn", + "aws_s3.s3_get_object", + "mongodb.mongo_delete", + "memory.delete_observations", + "memory.delete_relations", + "docker.docker_logs" + ], + "hit1": "supabase.supabase_select", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 664, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "kind": "verb_product", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "figma.figma_get_comments", + "linear.get_comment", + "figma.figma_get_file", + "notion.get_comment", + "linear.delete_comment", + "linear.update_comment", + "notion.delete_comment", + "notion.update_comment", + "github.update_issue_comment" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 201, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 308, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 649, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "kind": "verb_product", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "ranking": [ + "okta.okta_list_users", + "okta.okta_create_user", + "okta.okta_get_user", + "linear.list_users", + "notion.notion_list_users", + "slack.slack_get_users", + "okta.okta_deactivate_user", + "slack.slack_list_users", + "notion.list_users_tokens", + "pagerduty.pagerduty_list_oncalls" + ], + "hit1": "okta.okta_list_users", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 288, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 618, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query_range", + "prometheus.prom_query", + "prometheus.prom_list_metrics", + "datadog.datadog_query_metrics", + "postgres.explain_query", + "shopify.shopify_list_products", + "mysql.mysql_query", + "sqlite.sqlite_query", + "github.search_users", + "memory.search_nodes" + ], + "hit1": "prometheus.prom_query_range", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "kind": "verb_product", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "ranking": [ + "grafana.grafana_create_annotation", + "github.create_gist", + "stripe.stripe_create_product", + "grafana.grafana_get_dashboard", + "notion.notion_create_database", + "asana.asana_create_project", + "linear.linear_create_project", + "gmail.gmail_create_draft", + "github.create_artifact", + "trello.trello_create_board" + ], + "hit1": "grafana.grafana_create_annotation", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 782, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "kind": "identifier", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "ranking": [ + "terraform.tf_apply", + "terraform.tf_plan", + "kubernetes.k8s_apply", + "terraform.tf_state_list", + "slack.slack_reply_to_thread", + "playwright.browser_fill_form", + "filesystem.search_files", + "linear.linear_list_issues", + "docker.docker_rm", + "linear.linear_search_issues" + ], + "hit1": "terraform.tf_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 644, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "kind": "verb_product", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "ranking": [ + "asana.asana_list_tasks", + "clickup.clickup_list_tasks", + "asana.asana_get_task", + "asana.asana_update_task", + "asana.asana_create_task", + "linear.linear_list_teams", + "aws_s3.s3_list_buckets", + "linear.linear_list_projects", + "asana.asana_create_project", + "jira.jira_list_projects" + ], + "hit1": "asana.asana_list_tasks", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 612, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "ranking": [ + "trello.trello_add_comment", + "trello.trello_create_card", + "trello.trello_list_cards", + "trello.trello_create_board", + "figma.figma_post_comment", + "github.update_issue_comment", + "notion.get_comment", + "linear.get_comment", + "notion.delete_comment", + "linear.delete_comment" + ], + "hit1": "trello.trello_add_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_update_task", + "clickup.clickup_get_task", + "clickup.clickup_create_task", + "asana.asana_update_task", + "clickup.clickup_list_tasks", + "github.update_hook", + "linear.update_cycle", + "gitlab.update_job", + "shopify.shopify_update_product", + "linear.update_comment" + ], + "hit1": "clickup.clickup_update_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 736, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "kind": "verb_product", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "ranking": [ + "confluence.confluence_search", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "confluence.confluence_create_page", + "github.search_users", + "github.search_commits", + "hubspot.hubspot_search_contacts", + "slack.slack_search_messages", + "brave_search.brave_web_search", + "github.search_orgs" + ], + "hit1": "confluence.confluence_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_deal", + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "github.create_gist", + "github.create_repository", + "trello.trello_create_board", + "vercel.vercel_create_project", + "stripe.stripe_create_product", + "stripe.stripe_create_invoice", + "hubspot.hubspot_list_contacts" + ], + "hit1": "hubspot.hubspot_create_deal", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 413, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 696, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_search", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_create_ticket", + "aws_s3.s3_list_buckets", + "linear.linear_list_teams", + "clickup.clickup_list_tasks", + "jira.jira_list_projects", + "linear.linear_list_projects" + ], + "hit1": "zendesk.zendesk_list_tickets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 575, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "kind": "verb_product", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "ranking": [ + "discord.discord_list_channels", + "discord.discord_list_members", + "discord.discord_get_channel", + "discord.discord_send_message", + "slack.slack_list_channels", + "discord.discord_add_reaction", + "gmail.gmail_list_labels", + "linear.linear_list_teams", + "linear.linear_list_projects", + "trello.trello_list_cards" + ], + "hit1": "discord.discord_list_channels", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 166, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 577, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "ranking": [ + "redis.redis_set", + "redis.redis_get", + "redis.redis_delete", + "redis.redis_keys", + "jira.jira_get_issue", + "figma.figma_get_file", + "slack.slack_set_channel_topic", + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "playwright.browser_press_key" + ], + "hit1": "redis.redis_set", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert", + "mongodb.mongo_find", + "mongodb.mongo_delete", + "mongodb.mongo_update", + "mongodb.mongo_list_collections", + "supabase.supabase_insert", + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "linear.list_comments", + "notion.list_comments" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "kind": "verb_product", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "ranking": [ + "mysql.mysql_query", + "postgres.query", + "sqlite.sqlite_query", + "postgres.explain_query", + "salesforce.salesforce_query", + "mysql.mysql_list_tables", + "prometheus.prom_query", + "github.search_users", + "notion.notion_query_database", + "memory.search_nodes" + ], + "hit1": "mysql.mysql_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 585, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "elasticsearch.es_search", + "github.get_environment", + "github.delete_environment", + "playwright.browser_tab_close", + "jira.delete_attachment", + "linear.get_comment", + "linear.delete_comment", + "jira.get_attachment" + ], + "hit1": "elasticsearch.es_index_doc", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_put_object", + "okta.okta_get_user", + "aws_s3.s3_list_objects", + "asana.asana_get_task", + "gmail.gmail_send_email", + "github.get_artifact", + "github.get_hook", + "gitlab.get_job" + ], + "hit1": "aws_s3.s3_get_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 678, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones", + "cloudflare.cf_list_dns", + "cloudflare.cf_purge_cache", + "cloudflare.cf_create_dns", + "clickup.clickup_list_tasks", + "linear.linear_list_teams", + "linear.linear_list_projects", + "okta.okta_list_users", + "kubernetes.k8s_list_services", + "gmail.gmail_list_labels" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 554, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_create_customer", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "stripe.stripe_create_refund", + "stripe.stripe_create_payment_intent", + "stripe.stripe_list_invoices", + "stripe.stripe_get_customer", + "slack.slack_create_channel", + "salesforce.salesforce_create_record" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "kind": "verb_product", + "gold": [ + "github.actions_list" + ], + "alts": [], + "ranking": [ + "github.actions_list", + "github.actions_get", + "github.actions_run_trigger", + "github.actions_get_job_logs", + "github.list_gists", + "github.list_discussions", + "github.list_tags", + "github.projects_list", + "github.list_releases", + "github.list_notifications" + ], + "hit1": "github.actions_list", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 438, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 783, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "kind": "verb_product", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "ranking": [ + "github.actions_get_job_logs", + "github.actions_get", + "github.actions_list", + "github.actions_run_trigger", + "gitlab.get_job", + "github.get_gist", + "github.projects_get", + "github.get_label", + "github.get_discussion", + "github.get_latest_release" + ], + "hit1": "github.actions_get_job_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 442, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 802, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "kind": "verb_product", + "gold": [ + "github.create_repository" + ], + "alts": [], + "ranking": [ + "github.create_repository", + "github.delete_repository", + "github.create_branch", + "github.star_repository", + "github.unstar_repository", + "github.fork_repository", + "github.create_issue", + "github.list_tags", + "github.get_tag", + "github.list_repository_collaborators" + ], + "hit1": "github.create_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 406, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 853, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "kind": "verb_product", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "ranking": [ + "git.git_checkout", + "git.git_create_branch", + "git.git_branch", + "github.create_branch", + "github.update_pull_request_branch", + "github.delete_file", + "github.list_commits", + "gitlab.gitlab_create_merge_request", + "github.push_files", + "circleci.circleci_trigger_pipeline" + ], + "hit1": "git.git_checkout", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 926, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "kind": "identifier", + "gold": [ + "git.git_log" + ], + "alts": [], + "ranking": [ + "git.git_log", + "git.git_show", + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_status", + "git.git_diff", + "git.git_branch", + "git.git_create_branch", + "git.git_checkout", + "git.git_reset" + ], + "hit1": "git.git_log", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 603, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "kind": "verb_product", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "ranking": [ + "postgres.list_schemas", + "postgres.list_tables", + "postgres.explain_query", + "postgres.query", + "postgres.describe_table", + "linear.linear_list_teams", + "aws_s3.s3_list_buckets", + "kubernetes.k8s_list_services", + "mysql.mysql_list_tables", + "cloudflare.cf_list_zones" + ], + "hit1": "postgres.list_schemas", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 160, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 488, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "kind": "verb_product", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "ranking": [ + "memory.open_nodes", + "memory.search_nodes", + "memory.create_entities", + "memory.read_graph", + "memory.create_relations", + "memory.delete_relations", + "memory.add_observations", + "memory.delete_entities", + "postgres.list_schemas", + "sqlite.sqlite_list_tables" + ], + "hit1": "memory.open_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 492, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "kind": "paraphrase", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "ranking": [ + "filesystem.get_file_info", + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "google_drive.gdrive_get_file", + "gitlab.gitlab_get_file", + "figma.figma_get_file_nodes", + "filesystem.list_allowed_directories", + "github.get_file_contents", + "filesystem.move_file" + ], + "hit1": "filesystem.get_file_info", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 220, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "kind": "paraphrase", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "ranking": [ + "filesystem.write_file", + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "filesystem.edit_file", + "slack.delete_file", + "playwright.browser_tab_new", + "figma.figma_get_file", + "google_drive.gdrive_update_file", + "slack.update_file" + ], + "hit1": "filesystem.write_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "kind": "verb_product", + "gold": [ + "github.search_issues" + ], + "alts": [], + "ranking": [ + "github.search_issues", + "github.search_users", + "jira.jira_search_issues", + "linear.linear_search_issues", + "github.search_orgs", + "github.search_pull_requests", + "github.search_commits", + "github.list_issues", + "github.search_code", + "github.search_repositories" + ], + "hit1": "github.search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 228, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "kind": "paraphrase", + "gold": [ + "github.get_me" + ], + "alts": [], + "ranking": [ + "github.get_teams", + "github.get_me", + "github.list_notifications", + "github.get_latest_release", + "github.get_repository_tree", + "github.projects_get", + "github.get_gist", + "github.get_tag", + "github.get_label", + "github.list_gists" + ], + "hit1": "github.get_teams", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "cost": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.34, + "p95_calls": 1.0, + "success_within_1": 0.98, + "success_within_2": 0.9866666666666667, + "success_within_3": 0.9933333333333333, + "mean_tokens": 314.38, + "p95_tokens": 332.54999999999995, + "total_tokens": 47157, + "tokens_per_successful_query": 314.38, + "total_tokens_successful": 47157 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.2, + "p95_calls": 1.0, + "success_within_1": 0.9866666666666667, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 446.9266666666667, + "p95_tokens": 517.9499999999998, + "total_tokens": 67039, + "tokens_per_successful_query": 446.9266666666667, + "total_tokens_successful": 67039 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.0933333333333333, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 784.1466666666666, + "p95_tokens": 984.9999999999993, + "total_tokens": 117622, + "tokens_per_successful_query": 784.1466666666666, + "total_tokens_successful": 117622 + } + } +} diff --git a/experiments/mcp-tool-search/results/hybrid_rrf_trigram_embed.csv b/experiments/mcp-tool-search/results/hybrid_rrf_trigram_embed.csv new file mode 100644 index 00000000..dbe33964 --- /dev/null +++ b/experiments/mcp-tool-search/results/hybrid_rrf_trigram_embed.csv @@ -0,0 +1,151 @@ +id,query,kind,hit1,recall@1,recall@3,recall@5,mrr@10,ndcg@10,hit_rank,k3_calls,k3_tokens,k3_success1,k5_calls,k5_tokens,k5_success1,k10_calls,k10_tokens,k10_success1,ranking +q001,Create a GitHub issue,verb_product,github.create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,380,1.0,1,538,1.0,1,1014,1.0,github.create_issue github.issue_write gitlab.gitlab_create_issue github.create_gist github.create_repository github.issue_read github.update_issue_comment github.projects_write github.search_issues github.list_issues +q002,Create a Jira ticket,verb_product,jira.jira_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,330,1.0,1,535,1.0,1,981,1.0,jira.jira_create_issue jira.create_board jira.create_sprint jira.create_version jira.jira_create_sprint jira.create_filter jira.create_webhook zendesk.zendesk_create_ticket jira.create_attachment jira.update_sprint +q003,Send a Slack message,verb_product,slack.slack_post_message,1.0,1.0,1.0,1.0000,1.0000,1,1,253,1.0,1,377,1.0,1,673,1.0,slack.slack_post_message slack.slack_add_reaction slack.slack_get_channel_history slack.slack_search_messages slack.get_reminder slack.slack_leave_channel slack.get_pin discord.discord_send_message slack.slack_join_channel slack.slack_create_channel +q004,List open pull requests,resource,bitbucket.bitbucket_list_prs,0.0,1.0,1.0,0.5000,0.6697,2,1,237,1.0,1,477,1.0,1,976,1.0,bitbucket.bitbucket_list_prs github.list_pull_requests github.search_pull_requests github.create_pull_request github.pull_request_read gitlab.gitlab_list_merge_requests github.pull_request_review_write github.update_pull_request github.merge_pull_request playwright.browser_tab_list +q005,Open a pull request on GitHub,verb_product,github.create_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,314,1.0,1,509,1.0,1,1144,1.0,github.create_pull_request github.search_pull_requests github.list_pull_requests github.pull_request_read github.merge_pull_request github.request_copilot_review github.pull_request_review_write github.update_pull_request_branch github.update_pull_request github.add_comment_to_pending_review +q006,Run a SQL query against postgres,verb_product,postgres.query,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,295,1.0,1,560,1.0,postgres.query mysql.mysql_query sqlite.sqlite_query salesforce.salesforce_query postgres.explain_query postgres.list_schemas prometheus.prom_query postgres.describe_table postgres.list_tables datadog.datadog_list_monitors +q007,Read a local file from disk,paraphrase,filesystem.read_text_file,1.0,1.0,1.0,1.0000,0.8597,1,1,223,1.0,1,384,1.0,1,714,1.0,filesystem.read_text_file filesystem.read_file filesystem.read_media_file github.get_file_contents figma.figma_get_file figma.figma_get_file_nodes gitlab.gitlab_get_file slack.get_file filesystem.get_file_info filesystem.directory_tree +q008,What files are in this folder,paraphrase,google_drive.gdrive_list_folder,0.0,0.0,1.0,0.2000,0.4356,5,2,387,0.0,1,290,1.0,1,587,1.0,google_drive.gdrive_list_folder git.git_add filesystem.list_allowed_directories slack.get_file filesystem.list_directory slack.list_files slack.delete_file google_drive.gdrive_search filesystem.list_directory_with_sizes filesystem.directory_tree +q009,Remember this fact in the knowledge graph,paraphrase,memory.read_graph,0.0,0.0,1.0,0.2000,0.4295,5,2,334,0.0,1,263,1.0,1,601,1.0,memory.read_graph memory.open_nodes memory.create_relations memory.delete_relations memory.add_observations memory.create_entities memory.search_nodes memory.delete_entities pagerduty.pagerduty_acknowledge_incident sequential_thinking.sequentialthinking +q010,Search the web with Brave,verb_product,brave_search.brave_web_search,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,341,1.0,1,637,1.0,brave_search.brave_web_search brave_search.brave_local_search github.search_pull_requests github.search_users github.search_orgs slack.slack_search_messages confluence.confluence_search grafana.grafana_search_dashboards hubspot.hubspot_search_contacts memory.search_nodes +q011,Fetch the contents of a URL,verb_product,fetch.fetch,1.0,1.0,1.0,1.0000,1.0000,1,1,205,1.0,1,369,1.0,1,668,1.0,fetch.fetch confluence.confluence_search playwright.browser_navigate filesystem.read_file github.get_file_contents docker.docker_logs puppeteer.puppeteer_navigate playwright.browser_take_screenshot google_drive.gdrive_get_file playwright.browser_tab_new +q012,Show git status,identifier,git.git_status,1.0,1.0,1.0,1.0000,1.0000,1,1,206,1.0,1,329,1.0,1,675,1.0,git.git_status git.git_diff_unstaged github.list_milestones git.git_show git.git_log git.git_diff_staged github.list_check_runs gitlab.list_snippets github.list_hooks git.git_diff +q013,Commit staged changes,paraphrase,git.git_commit,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,295,1.0,1,695,1.0,git.git_commit git.git_reset git.git_diff_staged git.git_show salesforce.salesforce_list_opportunities git.git_add github.list_milestones gitlab.list_releases gitlab.list_milestones zendesk.zendesk_update_ticket +q014,Create a Linear issue,verb_product,linear.linear_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,275,1.0,1,463,1.0,1,875,1.0,linear.linear_create_issue linear.linear_create_comment linear.linear_get_issue linear.linear_assign_issue linear.linear_update_issue linear.linear_create_project linear.create_label linear.create_cycle linear.linear_search_issues linear.linear_list_issues +q015,Make a task in my project tracker Linear,paraphrase,linear.linear_list_projects,0.0,1.0,1.0,0.3333,0.5000,3,1,284,1.0,1,402,1.0,1,764,1.0,linear.linear_list_projects linear.linear_create_project linear.linear_create_issue asana.asana_list_tasks linear.get_cycle linear.get_label linear.get_attachment linear.get_user linear.create_label linear.linear_update_issue +q016,Post a comment on a GitHub issue,verb_product,github.update_issue_comment,0.0,1.0,1.0,0.5000,0.6309,2,1,327,1.0,1,494,1.0,1,993,1.0,github.update_issue_comment github.add_issue_comment github.get_discussion_comments jira.jira_add_comment github.issue_read figma.figma_post_comment github.add_comment_to_pending_review linear.linear_create_comment github.assign_copilot_to_issue github.request_copilot_review +q017,Merge the pull request,ambiguous,github.merge_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,299,1.0,1,431,1.0,1,962,1.0,github.merge_pull_request gitlab.gitlab_merge_merge_request github.update_pull_request_branch gitlab.gitlab_get_merge_request github.search_pull_requests gitlab.gitlab_list_merge_requests github.pull_request_read bitbucket.bitbucket_list_prs github.update_pull_request github.create_pull_request +q018,Trigger a GitHub Actions workflow,verb_product,github.actions_run_trigger,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,468,1.0,1,727,1.0,github.actions_run_trigger github.actions_get github.actions_list github.actions_get_job_logs github.update_gist github.dismiss_notification github.mark_all_notifications_read github.get_hook github.get_notification_details github.update_runner +q019,Get Dependabot alerts,resource,github.get_dependabot_alert,1.0,1.0,1.0,1.0000,0.8597,1,1,286,1.0,1,463,1.0,1,779,1.0,github.get_dependabot_alert github.list_dependabot_alerts github.get_secret_scanning_alert github.get_code_scanning_alert github.list_secret_scanning_alerts github.list_code_scanning_alerts vercel.vercel_get_deployment github.get_notification_details github.get_hook github.get_check_run +q020,List Sentry issues,verb_product,sentry.sentry_list_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,396,1.0,1,806,1.0,sentry.sentry_list_issues sentry.sentry_list_events sentry.sentry_list_projects sentry.sentry_get_issue sentry.sentry_update_issue linear.linear_list_issues gitlab.gitlab_list_issues github.list_issues pagerduty.pagerduty_list_incidents linear.list_cycles +q021,Create a Stripe customer,verb_product,stripe.stripe_create_invoice,0.0,1.0,1.0,0.5000,0.6309,2,1,223,1.0,1,322,1.0,1,669,1.0,stripe.stripe_create_invoice stripe.stripe_create_customer stripe.stripe_create_payment_intent stripe.stripe_get_customer stripe.stripe_list_invoices stripe.stripe_create_price stripe.stripe_create_product stripe.stripe_list_customers stripe.stripe_create_refund stripe.stripe_list_subscriptions +q022,Refund a Stripe payment,paraphrase,stripe.stripe_create_refund,1.0,1.0,1.0,1.0000,1.0000,1,1,197,1.0,1,334,1.0,1,647,1.0,stripe.stripe_create_refund stripe.stripe_get_payment_intent stripe.stripe_get_customer stripe.stripe_cancel_subscription stripe.stripe_create_payment_intent stripe.stripe_create_price stripe.stripe_create_invoice stripe.stripe_create_product stripe.stripe_list_invoices stripe.stripe_create_customer +q023,Send an email with Gmail,verb_product,gmail.gmail_send_email,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,328,1.0,1,727,1.0,gmail.gmail_send_email gmail.gmail_get_email gmail.gmail_search_emails gmail.gmail_trash_email gmail.gmail_list_labels twilio.twilio_send_sms gmail.gmail_modify_email gmail.gmail_create_draft stripe.stripe_create_customer shopify.shopify_create_order +q024,Search my Gmail inbox,verb_product,gmail.gmail_search_emails,1.0,1.0,1.0,1.0000,1.0000,1,1,193,1.0,1,300,1.0,1,607,1.0,gmail.gmail_search_emails slack.slack_search_messages gmail.gmail_list_labels hubspot.hubspot_search_contacts github.search_users google_drive.gdrive_search gmail.gmail_get_email github.search_orgs github.search_pull_requests gmail.gmail_send_email +q025,Upload a file to Google Drive,paraphrase,google_drive.gdrive_update_file,0.0,0.0,1.0,0.2500,0.4307,4,2,496,0.0,1,418,1.0,1,765,1.0,google_drive.gdrive_update_file google_drive.gdrive_share_file google_drive.gdrive_search google_drive.gdrive_create_file google_drive.gdrive_list_folder google_drive.gdrive_get_file slack.slack_upload_file aws_s3.s3_put_object slack.update_file figma.figma_get_file +q026,Create a calendar event,resource,google_calendar.gcal_create_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,480,1.0,1,882,1.0,google_calendar.gcal_create_event google_calendar.gcal_update_event google_calendar.gcal_delete_event google_calendar.gcal_list_events clickup.clickup_create_task google_calendar.gcal_list_calendars pagerduty.pagerduty_create_incident linear.create_cycle trello.trello_create_board slack.create_canvase +q027,List Kubernetes pods,verb_product,kubernetes.k8s_list_pods,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,314,1.0,1,698,1.0,kubernetes.k8s_list_pods kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_list_deployments kubernetes.k8s_get_pod kubernetes.k8s_pod_logs kubernetes.k8s_apply docker.docker_images kubernetes.k8s_scale_deployment kubernetes.k8s_delete +q028,Get logs from a k8s pod,verb_product,kubernetes.k8s_pod_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,262,1.0,1,429,1.0,1,754,1.0,kubernetes.k8s_pod_logs kubernetes.k8s_get_pod docker.docker_logs github.actions_get_job_logs slack.slack_get_channel_history kubernetes.k8s_list_pods datadog.datadog_list_logs gitlab.get_snippet github.get_runner git.git_log +q029,Run a docker container,verb_product,docker.docker_run,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,331,1.0,1,647,1.0,docker.docker_run docker.docker_stop docker.docker_exec docker.docker_ps docker.docker_logs docker.docker_rm docker.docker_build docker.docker_images kubernetes.k8s_pod_logs salesforce.salesforce_query +q030,Show docker container logs,verb_product,docker.docker_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,226,1.0,1,365,1.0,1,649,1.0,docker.docker_logs docker.docker_stop kubernetes.k8s_pod_logs docker.docker_ps docker.docker_run docker.docker_images docker.docker_exec git.git_log docker.docker_rm docker.docker_build +q031,Navigate the browser to a URL,ambiguous,playwright.browser_navigate,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,294,1.0,1,581,1.0,playwright.browser_navigate puppeteer.puppeteer_navigate playwright.browser_tab_new playwright.browser_press_key playwright.browser_click playwright.browser_evaluate playwright.browser_tab_list playwright.browser_type playwright.browser_take_screenshot playwright.browser_tab_close +q032,Take a screenshot of the page,ambiguous,playwright.browser_take_screenshot,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,314,1.0,1,623,1.0,playwright.browser_take_screenshot puppeteer.puppeteer_screenshot playwright.browser_snapshot playwright.browser_evaluate playwright.browser_press_key playwright.browser_click pagerduty.pagerduty_acknowledge_incident confluence.confluence_get_page notion.notion_get_page pagerduty.pagerduty_resolve_incident +q033,Create a GitLab merge request,verb_product,gitlab.gitlab_create_merge_request,1.0,1.0,1.0,1.0000,1.0000,1,1,297,1.0,1,475,1.0,1,931,1.0,gitlab.gitlab_create_merge_request gitlab.gitlab_merge_merge_request gitlab.gitlab_get_merge_request gitlab.gitlab_list_merge_requests github.merge_pull_request gitlab.gitlab_create_pipeline gitlab.create_release gitlab.gitlab_create_issue gitlab.create_job gitlab.create_pipeline +q034,Search Notion pages,verb_product,notion.notion_search,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,353,1.0,1,692,1.0,notion.notion_search notion.notion_get_page notion.notion_update_page notion.notion_create_page notion.notion_get_comments github.search_repositories memory.search_nodes github.search_orgs notion.notion_create_database notion.notion_get_block +q035,Create a Notion page,verb_product,notion.notion_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,230,1.0,1,413,1.0,1,832,1.0,notion.notion_create_page notion.notion_get_page notion.notion_update_page notion.notion_create_database notion.create_comment notion.create_data_source notion.create_users_token notion.notion_get_comments confluence.confluence_create_page notion.update_comment +q036,Query a Notion database,verb_product,notion.notion_query_database,1.0,1.0,1.0,1.0000,1.0000,1,1,261,1.0,1,404,1.0,1,740,1.0,notion.notion_query_database notion.notion_search notion.notion_create_database notion.notion_create_page notion.get_data_source notion.update_data_source notion.notion_get_block notion.notion_get_page notion.delete_data_source notion.create_data_source +q037,Create a Zendesk ticket,verb_product,zendesk.zendesk_get_ticket,0.0,1.0,1.0,0.5000,0.6309,2,1,257,1.0,1,368,1.0,1,871,1.0,zendesk.zendesk_get_ticket zendesk.zendesk_create_ticket zendesk.zendesk_update_ticket zendesk.zendesk_list_tickets zendesk.zendesk_search jira.jira_create_issue jira.create_board slack.create_reminder jira.create_sprint confluence.confluence_create_page +q038,Create a HubSpot contact,verb_product,hubspot.hubspot_create_contact,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,364,1.0,1,826,1.0,hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_list_contacts hubspot.hubspot_search_contacts hubspot.hubspot_create_deal cloudflare.cf_create_dns zendesk.zendesk_create_ticket jira.create_attachment slack.slack_create_channel pagerduty.pagerduty_create_incident +q039,Create a Salesforce lead,verb_product,salesforce.salesforce_create_lead,1.0,1.0,1.0,1.0000,1.0000,1,1,252,1.0,1,380,1.0,1,830,1.0,salesforce.salesforce_create_lead salesforce.salesforce_convert_lead salesforce.salesforce_create_record salesforce.salesforce_list_opportunities salesforce.salesforce_get_record salesforce.salesforce_update_record salesforce.salesforce_query hubspot.hubspot_create_contact bitbucket.bitbucket_create_pr confluence.confluence_create_page +q040,Run a SOQL query,identifier,salesforce.salesforce_query,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,294,1.0,1,601,1.0,salesforce.salesforce_query mysql.mysql_query sqlite.sqlite_query postgres.query postgres.explain_query prometheus.prom_query notion.notion_query_database datadog.datadog_list_monitors hubspot.hubspot_search_contacts github.search_users +q041,List Shopify orders,verb_product,shopify.shopify_list_orders,1.0,1.0,1.0,1.0000,1.0000,1,1,193,1.0,1,333,1.0,1,669,1.0,shopify.shopify_list_orders shopify.shopify_get_order shopify.shopify_list_customers shopify.shopify_create_order shopify.shopify_list_products shopify.shopify_get_product shopify.shopify_update_product shopify.shopify_create_product okta.okta_list_users stripe.stripe_list_customers +q042,Create a PagerDuty incident,verb_product,pagerduty.pagerduty_create_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,403,1.0,1,831,1.0,pagerduty.pagerduty_create_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_list_incidents notion.notion_create_page confluence.confluence_create_page pagerduty.pagerduty_list_oncalls clickup.clickup_create_task notion.create_comment +q043,Who is on call in PagerDuty,paraphrase,pagerduty.pagerduty_list_oncalls,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,389,1.0,1,728,1.0,pagerduty.pagerduty_list_oncalls pagerduty.pagerduty_get_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_list_incidents playwright.browser_press_key twilio.twilio_make_call notion.notion_update_page notion.notion_get_page +q044,Query Datadog metrics,verb_product,datadog.datadog_query_metrics,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,458,1.0,1,783,1.0,datadog.datadog_query_metrics datadog.datadog_list_monitors datadog.datadog_create_monitor datadog.datadog_list_logs datadog.datadog_get_monitor notion.notion_search slack.slack_search_messages prometheus.prom_list_metrics notion.notion_query_database prometheus.prom_query +q045,Search Datadog logs,verb_product,datadog.datadog_list_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,363,1.0,1,751,1.0,datadog.datadog_list_logs datadog.datadog_list_monitors datadog.datadog_get_monitor slack.slack_search_messages github.search_orgs github.search_users datadog.datadog_query_metrics datadog.datadog_create_monitor github.search_pull_requests notion.notion_search +q046,List S3 buckets,verb_product,aws_s3.s3_list_buckets,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,327,1.0,1,656,1.0,aws_s3.s3_list_buckets aws_s3.s3_list_objects bitbucket.bitbucket_list_repos aws_s3.s3_get_object aws_s3.s3_delete_object bitbucket.bitbucket_list_prs aws_s3.s3_put_object jira.jira_list_projects gitlab.list_snippets salesforce.salesforce_list_opportunities +q047,Upload an object to S3,paraphrase,aws_s3.s3_put_object,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,405,1.0,1,808,1.0,aws_s3.s3_put_object aws_s3.s3_get_object aws_s3.s3_delete_object aws_s3.s3_list_objects slack.slack_upload_file twilio.twilio_send_sms salesforce.salesforce_update_record salesforce.salesforce_create_record gmail.gmail_create_draft salesforce.salesforce_get_record +q048,Purge Cloudflare cache,verb_product,cloudflare.cf_purge_cache,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,358,1.0,1,608,1.0,cloudflare.cf_purge_cache cloudflare.cf_list_zones cloudflare.cf_list_dns cloudflare.cf_create_dns google_calendar.gcal_delete_event slack.delete_bookmark slack.delete_canvase playwright.browser_tab_close playwright.browser_snapshot postgres.list_tables +q049,Create a Cloudflare DNS record,verb_product,cloudflare.cf_create_dns,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,356,1.0,1,844,1.0,cloudflare.cf_create_dns cloudflare.cf_list_dns salesforce.salesforce_create_record cloudflare.cf_purge_cache cloudflare.cf_list_zones salesforce.salesforce_update_record confluence.confluence_create_page slack.create_canvase slack.create_file slack.create_reminder +q050,List Vercel deployments,verb_product,vercel.vercel_list_deployments,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,291,1.0,1,558,1.0,vercel.vercel_list_deployments vercel.vercel_get_deployment vercel.vercel_list_projects kubernetes.k8s_list_deployments vercel.vercel_create_project github.list_environments sentry.sentry_list_projects jira.jira_list_projects sentry.sentry_list_events docker.docker_images +q051,Insert a row into Supabase,verb_product,supabase.supabase_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,389,1.0,1,701,1.0,supabase.supabase_insert supabase.supabase_update supabase.supabase_select mongodb.mongo_insert supabase.supabase_invoke_fn supabase.supabase_list_tables notion.notion_create_database linear.get_user gmail.gmail_create_draft gitlab.get_release +q052,Export images from Figma,paraphrase,figma.figma_get_images,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,330,1.0,1,724,1.0,figma.figma_get_images figma.figma_get_file_nodes figma.figma_get_file docker.docker_images filesystem.read_media_file figma.figma_get_comments figma.figma_post_comment docker.docker_run github.get_file_contents docker.docker_build +q053,Send an SMS with Twilio,verb_product,twilio.twilio_send_sms,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,409,1.0,1,890,1.0,twilio.twilio_send_sms twilio.twilio_list_messages twilio.twilio_make_call gmail.gmail_send_email discord.discord_send_message slack.slack_add_reaction slack.get_reminder sentry.sentry_list_issues aws_s3.s3_put_object github.add_comment_to_pending_review +q054,Create an Okta user,verb_product,okta.okta_create_user,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,329,1.0,1,806,1.0,okta.okta_create_user okta.okta_get_user okta.okta_deactivate_user okta.okta_list_users linear.create_user notion.create_users_token slack.create_usergroup asana.asana_create_project asana.asana_create_task jira.create_filter +q055,Instant query Prometheus,verb_product,prometheus.prom_query,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,382,1.0,1,667,1.0,prometheus.prom_query prometheus.prom_query_range prometheus.prom_list_metrics datadog.datadog_query_metrics salesforce.salesforce_query elasticsearch.es_search sqlite.sqlite_query memory.search_nodes postgres.query hubspot.hubspot_search_contacts +q056,Search Grafana dashboards,verb_product,grafana.grafana_search_dashboards,1.0,1.0,1.0,1.0000,1.0000,1,1,185,1.0,1,287,1.0,1,680,1.0,grafana.grafana_search_dashboards grafana.grafana_get_dashboard github.search_pull_requests github.search_users github.search_orgs github.search_issues grafana.grafana_create_annotation confluence.confluence_search slack.slack_search_messages datadog.datadog_list_logs +q057,Trigger a CircleCI pipeline,verb_product,circleci.circleci_trigger_pipeline,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,359,1.0,1,737,1.0,circleci.circleci_trigger_pipeline circleci.circleci_get_pipeline circleci.circleci_list_pipelines hubspot.hubspot_create_deal gitlab.update_pipeline gitlab.gitlab_create_pipeline gitlab.get_pipeline gitlab.list_pipelines github.actions_run_trigger gitlab.delete_pipeline +q058,Run terraform plan,identifier,terraform.tf_plan,1.0,1.0,1.0,1.0000,1.0000,1,1,172,1.0,1,295,1.0,1,672,1.0,terraform.tf_plan terraform.tf_apply terraform.tf_state_list github.get_check_run github.update_check_run docker.docker_run github.create_check_run docker.docker_exec github.update_runner github.delete_check_run +q059,Create an Asana task,verb_product,asana.asana_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,405,1.0,1,846,1.0,asana.asana_create_task asana.asana_get_task asana.asana_create_project asana.asana_update_task asana.asana_list_tasks clickup.clickup_create_task okta.okta_create_user grafana.grafana_create_annotation github.create_issue gmail.gmail_create_draft +q060,Create a Trello card,verb_product,trello.trello_create_card,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,364,1.0,1,764,1.0,trello.trello_create_card trello.trello_create_board trello.trello_add_comment trello.trello_list_cards stripe.stripe_create_invoice stripe.stripe_create_customer stripe.stripe_create_product grafana.grafana_create_annotation jira.jira_create_sprint shopify.shopify_create_order +q061,Create a ClickUp task,verb_product,clickup.clickup_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,415,1.0,1,839,1.0,clickup.clickup_create_task clickup.clickup_get_task clickup.clickup_list_tasks clickup.clickup_update_task asana.asana_create_task jira.create_sprint gmail.gmail_create_draft confluence.confluence_create_page trello.trello_create_board jira.create_filter +q062,Create a Confluence page,verb_product,confluence.confluence_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,401,1.0,1,922,1.0,confluence.confluence_create_page confluence.confluence_get_page confluence.confluence_update_page confluence.confluence_search notion.notion_create_page pagerduty.pagerduty_create_incident cloudflare.cf_create_dns bitbucket.bitbucket_create_issue slack.create_canvase datadog.datadog_create_monitor +q063,Send a Discord message,verb_product,discord.discord_send_message,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,350,1.0,1,665,1.0,discord.discord_send_message discord.discord_add_reaction discord.discord_get_channel slack.slack_post_message gmail.gmail_trash_email discord.discord_list_members slack.slack_add_reaction discord.discord_list_channels gmail.gmail_get_email figma.figma_post_comment +q064,Search Elasticsearch,verb_product,elasticsearch.es_search,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,315,1.0,1,633,1.0,elasticsearch.es_search confluence.confluence_search elasticsearch.es_get_doc hubspot.hubspot_search_contacts github.search_users elasticsearch.es_index_doc github.search_orgs github.search_pull_requests slack.slack_search_messages grafana.grafana_search_dashboards +q065,Get a Redis key,verb_product,redis.redis_get,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,283,1.0,1,548,1.0,redis.redis_get redis.redis_set redis.redis_delete redis.redis_keys figma.figma_get_file jira.jira_get_issue github.get_runner slack.get_reminder github.get_gist github.get_milestone +q066,Find MongoDB documents,verb_product,mongodb.mongo_find,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,381,1.0,1,726,1.0,mongodb.mongo_find mongodb.mongo_update mongodb.mongo_delete mongodb.mongo_list_collections mongodb.mongo_insert elasticsearch.es_get_doc elasticsearch.es_index_doc filesystem.list_directory filesystem.list_directory_with_sizes github.list_environments +q067,List SQLite tables,verb_product,sqlite.sqlite_list_tables,1.0,1.0,1.0,1.0000,1.0000,1,1,151,1.0,1,240,1.0,1,537,1.0,sqlite.sqlite_list_tables mysql.mysql_list_tables sqlite.sqlite_describe_table supabase.supabase_list_tables postgres.list_tables postgres.list_schemas linear.list_cycles okta.okta_list_users gitlab.list_milestones sqlite.sqlite_query +q068,Describe a postgres table,verb_product,postgres.describe_table,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,274,1.0,1,679,1.0,postgres.describe_table sqlite.sqlite_describe_table postgres.explain_query postgres.list_tables postgres.list_schemas postgres.query linear.create_user jira.create_filter jira.create_board supabase.supabase_insert +q069,github_create_issue,identifier,github.create_issue,1.0,1.0,1.0,1.0000,0.9239,1,1,320,1.0,1,536,1.0,1,998,1.0,github.create_issue gitlab.gitlab_create_issue github.create_gist github.issue_write github.create_repository bitbucket.bitbucket_create_issue github.create_package github.create_hook github.create_runner github.search_issues +q070,createIssue,identifier,gmail.gmail_create_draft,0.0,0.0,0.0,0.0000,0.0000,39,13,3380,0.0,8,3426,0.0,4,3354,0.0,gmail.gmail_create_draft okta.okta_create_user clickup.clickup_create_task stripe.stripe_create_product trello.trello_create_board linear.create_user stripe.stripe_create_invoice confluence.confluence_create_page vercel.vercel_create_project salesforce.salesforce_create_record +q071,list-directory,identifier,filesystem.list_directory,1.0,1.0,1.0,1.0000,1.0000,1,1,171,1.0,1,256,1.0,1,508,1.0,filesystem.list_directory filesystem.list_directory_with_sizes filesystem.list_allowed_directories sentry.sentry_list_projects jira.jira_list_projects datadog.datadog_list_monitors mongodb.mongo_list_collections vercel.vercel_list_projects gitlab.gitlab_list_projects okta.okta_list_users +q072,sequential thinking next thought,paraphrase,sequential_thinking.sequentialthinking,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,337,1.0,1,676,1.0,sequential_thinking.sequentialthinking memory.add_observations playwright.browser_wait_for memory.create_entities memory.read_graph grafana.grafana_create_annotation filesystem.edit_file memory.create_relations confluence.confluence_search memory.search_nodes +q073,What time is it in Tokyo,paraphrase,time.get_current_time,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,391,1.0,1,780,1.0,time.get_current_time time.convert_time playwright.browser_wait_for grafana.grafana_create_annotation google_calendar.gcal_list_events jira.jira_add_worklog slack.slack_add_reaction jira.list_sprints jira.list_boards zendesk.zendesk_list_tickets +q074,Convert time between timezones,verb_product,time.convert_time,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,338,1.0,1,776,1.0,time.convert_time time.get_current_time grafana.grafana_create_annotation cloudflare.cf_list_zones playwright.browser_wait_for cloudflare.cf_create_dns datadog.datadog_query_metrics salesforce.salesforce_convert_lead google_calendar.gcal_list_events slack.slack_add_reaction +q075,Star a GitHub repo,verb_product,github.star_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,214,1.0,1,399,1.0,1,809,1.0,github.star_repository github.unstar_repository github.list_starred_repositories github.list_pull_requests github.get_file_blame github.get_label github.list_releases github.list_labels github.actions_get_job_logs github.get_release_by_tag +q076,Fork a repository,resource,github.fork_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,378,1.0,1,783,1.0,github.fork_repository github.unstar_repository github.star_repository github.delete_repository github.list_releases github.list_dependabot_alerts github.create_repository github.get_repository_tree github.list_labels github.list_discussions +q077,Get the latest GitHub release,verb_product,github.get_latest_release,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,363,1.0,1,624,1.0,github.get_latest_release github.get_release_by_tag gitlab.get_release github.list_releases github.get_gist github.get_milestone github.get_hook github.get_artifact github.get_package github.get_environment +q078,Create a gist,resource,github.create_gist,1.0,1.0,1.0,1.0000,1.0000,1,1,280,1.0,1,473,1.0,1,963,1.0,github.create_gist github.update_gist gitlab.create_snippet github.create_hook gitlab.create_label gitlab.create_wiki gmail.gmail_create_draft gitlab.gitlab_create_merge_request github.projects_write github.create_milestone +q079,Search GitHub code,verb_product,github.search_code,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,296,1.0,1,719,1.0,github.search_code github.search_users github.search_orgs github.search_commits github.search_pull_requests github.search_repositories github.search_issues github.get_code_scanning_alert github.list_code_scanning_alerts gitlab.gitlab_list_projects +q080,List GitHub notifications,verb_product,github.list_notifications,1.0,1.0,1.0,1.0000,1.0000,1,1,176,1.0,1,279,1.0,1,673,1.0,github.list_notifications github.mark_all_notifications_read github.get_notification_details github.dismiss_notification github.list_gists github.actions_list github.list_secret_scanning_alerts github.list_releases github.list_code_scanning_alerts github.list_artifacts +q081,Assign Copilot to an issue,verb_product,github.assign_copilot_to_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,466,1.0,1,838,1.0,github.assign_copilot_to_issue linear.linear_assign_issue jira.jira_assign_issue github.create_issue linear.linear_create_comment github.request_copilot_review linear.linear_list_issues jira.jira_add_comment jira.jira_get_issue jira.jira_update_issue +q082,Get secret scanning alerts,resource,github.list_secret_scanning_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,483,1.0,1,775,1.0,github.list_secret_scanning_alerts github.get_secret_scanning_alert github.get_code_scanning_alert github.list_code_scanning_alerts github.get_dependabot_alert github.list_dependabot_alerts github.get_check_run gitlab.get_snippet github.get_hook slack.get_pin +q083,Create or update a file on GitHub,verb_product,github.create_or_update_file,1.0,1.0,1.0,1.0000,0.7602,1,1,333,1.0,1,476,1.0,1,876,1.0,github.create_or_update_file github.projects_write github.update_gist github.create_gist github.update_hook github.update_package github.update_runner github.update_environment github.issue_write github.update_check_run +q084,List collaborators on a repo,resource,github.list_repository_collaborators,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,382,1.0,1,792,1.0,github.list_repository_collaborators github.list_labels github.list_tags github.list_discussions github.list_releases github.list_commits github.list_branches github.list_starred_repositories github.list_issues github.list_dependabot_alerts +q085,Reply in a Slack thread,verb_product,slack.slack_get_thread_replies,0.0,1.0,1.0,0.5000,0.6309,2,1,242,1.0,1,365,1.0,1,688,1.0,slack.slack_get_thread_replies slack.slack_reply_to_thread slack.get_reminder slack.get_pin slack.update_reminder slack.get_canvase slack.get_file slack.delete_reminder slack.update_pin slack.slack_add_reaction +q086,Add an emoji reaction on Slack,verb_product,slack.slack_add_reaction,1.0,1.0,1.0,1.0000,1.0000,1,1,302,1.0,1,454,1.0,1,890,1.0,slack.slack_add_reaction discord.discord_add_reaction slack.create_reminder slack.get_reminder slack.create_pin slack.create_canvase slack.update_reminder slack.get_canvase slack.create_usergroup slack.create_file +q087,Search Slack messages,verb_product,slack.slack_search_messages,1.0,1.0,1.0,1.0000,1.0000,1,1,251,1.0,1,379,1.0,1,661,1.0,slack.slack_search_messages slack.slack_get_channel_history slack.slack_add_reaction slack.slack_post_message slack.get_canvase slack.get_reminder slack.get_file slack.get_pin gmail.gmail_search_emails github.search_users +q088,Create a Slack channel,verb_product,slack.slack_create_channel,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,339,1.0,1,795,1.0,slack.slack_create_channel slack.slack_leave_channel slack.slack_join_channel slack.slack_post_message slack.slack_set_channel_topic slack.slack_get_thread_replies slack.slack_reply_to_thread slack.slack_add_reaction slack.create_pin slack.create_usergroup +q089,Transition a Jira issue,verb_product,jira.jira_transition_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,342,1.0,1,721,1.0,jira.jira_transition_issue jira.jira_assign_issue jira.jira_get_issue jira.jira_update_issue jira.jira_add_comment jira.jira_create_issue jira.jira_search_issues jira.delete_version jira.get_version jira.jira_add_worklog +q090,Search Jira with JQL,identifier,jira.jira_search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,199,1.0,1,300,1.0,1,566,1.0,jira.jira_search_issues github.search_pull_requests jira.get_webhook github.search_users jira.get_filter hubspot.hubspot_search_contacts github.search_orgs jira.jira_list_projects confluence.confluence_search slack.slack_search_messages +q091,Add a worklog to Jira,verb_product,jira.jira_add_worklog,1.0,1.0,1.0,1.0000,1.0000,1,1,233,1.0,1,354,1.0,1,660,1.0,jira.jira_add_worklog jira.jira_add_comment jira.get_webhook jira.update_webhook jira.get_sprint jira.jira_list_projects jira.jira_transition_issue jira.create_webhook jira.get_board jira.jira_get_issue +q092,List Linear teams,verb_product,linear.linear_list_teams,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,339,1.0,1,745,1.0,linear.linear_list_teams linear.linear_list_projects linear.linear_list_issues linear.list_cycles linear.linear_create_project linear.list_labels linear.list_users linear.get_cycle linear.linear_create_issue linear.list_attachments +q093,Comment on a Linear issue,verb_product,linear.linear_create_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,195,1.0,1,320,1.0,1,781,1.0,linear.linear_create_comment linear.linear_get_issue linear.get_comment linear.linear_assign_issue linear.delete_comment linear.update_comment linear.linear_update_issue linear.linear_create_issue linear.linear_search_issues linear.create_comment +q094,Move a local file,paraphrase,filesystem.read_text_file,0.0,1.0,1.0,0.5000,0.6309,2,1,225,1.0,1,349,1.0,1,708,1.0,filesystem.read_text_file filesystem.move_file filesystem.read_media_file filesystem.read_file slack.update_file slack.get_file slack.delete_file filesystem.write_file google_drive.gdrive_update_file filesystem.edit_file +q095,Recursively search files by pattern,paraphrase,filesystem.search_files,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,304,1.0,1,641,1.0,filesystem.search_files filesystem.directory_tree google_drive.gdrive_search redis.redis_keys memory.search_nodes notion.notion_search linear.linear_search_issues github.search_repositories github.search_code filesystem.list_directory +q096,Edit a file in place with old and new string,paraphrase,filesystem.edit_file,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,405,1.0,1,812,1.0,filesystem.edit_file slack.update_file filesystem.write_file slack.create_file slack.get_file slack.delete_file filesystem.move_file github.create_or_update_file git.git_add slack.slack_upload_file +q097,Read the whole knowledge graph,paraphrase,memory.read_graph,1.0,1.0,1.0,1.0000,1.0000,1,1,154,1.0,1,259,1.0,1,513,1.0,memory.read_graph memory.open_nodes memory.delete_relations memory.search_nodes memory.create_entities memory.create_relations memory.add_observations memory.delete_entities github.mark_all_notifications_read filesystem.read_file +q098,Search memory nodes,verb_product,memory.search_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,195,1.0,1,302,1.0,1,624,1.0,memory.search_nodes memory.open_nodes notion.notion_search hubspot.hubspot_search_contacts github.search_users github.search_orgs github.search_repositories github.search_pull_requests slack.slack_search_messages zendesk.zendesk_search +q099,Show unstaged git diff,verb_product,git.git_diff_unstaged,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,299,1.0,1,615,1.0,git.git_diff_unstaged git.git_diff_staged git.git_diff git.git_reset git.git_show git.git_log git.git_status git.git_commit git.git_add github.list_packages +q100,Stage files with git add,identifier,git.git_add,1.0,1.0,1.0,1.0000,1.0000,1,1,198,1.0,1,329,1.0,1,773,1.0,git.git_add git.git_reset git.git_commit git.git_diff_staged github.update_gist github.create_gist salesforce.salesforce_list_opportunities github.push_files github.create_or_update_file git.git_diff_unstaged +q101,Create a Shopify product,verb_product,shopify.shopify_create_product,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,377,1.0,1,700,1.0,shopify.shopify_create_product shopify.shopify_get_product shopify.shopify_update_product shopify.shopify_create_order stripe.stripe_create_product shopify.shopify_list_products shopify.shopify_get_order stripe.stripe_create_price stripe.stripe_create_invoice stripe.stripe_create_customer +q102,Cancel a Stripe subscription,verb_product,stripe.stripe_cancel_subscription,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,301,1.0,1,663,1.0,stripe.stripe_cancel_subscription stripe.stripe_list_subscriptions stripe.stripe_create_invoice stripe.stripe_get_customer stripe.stripe_create_product stripe.stripe_get_payment_intent stripe.stripe_create_customer stripe.stripe_create_refund stripe.stripe_create_price stripe.stripe_create_payment_intent +q103,Create a payment intent,resource,stripe.stripe_create_refund,0.0,1.0,1.0,0.5000,0.6309,2,1,234,1.0,1,386,1.0,1,866,1.0,stripe.stripe_create_refund stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent pagerduty.pagerduty_create_incident stripe.stripe_create_invoice jira.create_attachment notion.create_comment hubspot.hubspot_create_deal slack.create_pin jira.create_sprint +q104,Acknowledge a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,405,1.0,1,690,1.0,pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls notion.notion_get_page playwright.browser_snapshot playwright.browser_take_screenshot confluence.confluence_get_page +q105,Create a Datadog monitor,verb_product,datadog.datadog_create_monitor,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,441,1.0,1,933,1.0,datadog.datadog_create_monitor datadog.datadog_get_monitor datadog.datadog_list_monitors clickup.clickup_create_task hubspot.hubspot_create_deal cloudflare.cf_create_dns notion.create_data_source datadog.datadog_query_metrics notion.notion_create_database datadog.datadog_list_logs +q106,Resolve a Sentry issue,paraphrase,sentry.sentry_get_issue,0.0,1.0,1.0,0.5000,0.6309,2,1,274,1.0,1,412,1.0,1,729,1.0,sentry.sentry_get_issue sentry.sentry_update_issue sentry.sentry_list_issues sentry.sentry_list_events pagerduty.pagerduty_resolve_incident jira.jira_assign_issue bitbucket.bitbucket_create_issue sentry.sentry_list_projects jira.jira_get_issue linear.linear_get_issue +q107,Scale a Kubernetes deployment,verb_product,kubernetes.k8s_scale_deployment,1.0,1.0,1.0,1.0000,1.0000,1,1,241,1.0,1,407,1.0,1,726,1.0,kubernetes.k8s_scale_deployment kubernetes.k8s_list_deployments kubernetes.k8s_apply kubernetes.k8s_delete kubernetes.k8s_get_pod vercel.vercel_get_deployment kubernetes.k8s_list_services kubernetes.k8s_pod_logs kubernetes.k8s_list_pods kubernetes.k8s_get_events +q108,Apply a Kubernetes manifest,verb_product,kubernetes.k8s_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,222,1.0,1,330,1.0,1,727,1.0,kubernetes.k8s_apply kubernetes.k8s_get_pod kubernetes.k8s_list_pods kubernetes.k8s_list_services kubernetes.k8s_list_deployments kubernetes.k8s_get_events kubernetes.k8s_delete kubernetes.k8s_scale_deployment kubernetes.k8s_pod_logs terraform.tf_apply +q109,Stop a docker container,verb_product,docker.docker_stop,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,327,1.0,1,639,1.0,docker.docker_stop docker.docker_rm docker.docker_logs docker.docker_run docker.docker_ps docker.docker_exec docker.docker_build docker.docker_images hubspot.hubspot_list_contacts kubernetes.k8s_pod_logs +q110,Build a docker image,verb_product,docker.docker_build,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,319,1.0,1,635,1.0,docker.docker_build docker.docker_images docker.docker_run docker.docker_stop docker.docker_logs docker.docker_ps docker.docker_exec figma.figma_get_images docker.docker_rm filesystem.read_media_file +q111,Click an element in Playwright,verb_product,playwright.browser_click,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,318,1.0,1,566,1.0,playwright.browser_click playwright.browser_type puppeteer.puppeteer_click playwright.browser_press_key playwright.browser_navigate playwright.browser_tab_list playwright.browser_take_screenshot playwright.browser_tab_close playwright.browser_tab_new playwright.browser_snapshot +q112,Type text into a browser input,paraphrase,playwright.browser_type,1.0,1.0,1.0,1.0000,0.8956,1,1,217,1.0,1,318,1.0,1,584,1.0,playwright.browser_type playwright.browser_navigate playwright.browser_wait_for playwright.browser_press_key playwright.browser_tab_new puppeteer.puppeteer_fill playwright.browser_fill_form playwright.browser_tab_list playwright.browser_click playwright.browser_tab_close +q113,List GitLab projects,verb_product,gitlab.gitlab_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,288,1.0,1,681,1.0,gitlab.gitlab_list_projects gitlab.gitlab_list_issues gitlab.gitlab_list_merge_requests vercel.vercel_list_projects jira.jira_list_projects gitlab.list_snippets github.projects_list gitlab.list_jobs gitlab.list_pipelines gitlab.gitlab_create_issue +q114,Create a Bitbucket pull request,verb_product,bitbucket.bitbucket_create_pr,1.0,1.0,1.0,1.0000,1.0000,1,1,313,1.0,1,606,1.0,1,1037,1.0,bitbucket.bitbucket_create_pr bitbucket.bitbucket_list_prs bitbucket.bitbucket_create_issue github.create_pull_request github.pull_request_review_write github.pull_request_read github.list_pull_requests github.update_pull_request_branch github.search_pull_requests github.merge_pull_request +q115,Create a Gmail draft,verb_product,gmail.gmail_create_draft,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,431,1.0,1,830,1.0,gmail.gmail_create_draft stripe.stripe_create_customer hubspot.hubspot_create_contact shopify.shopify_create_order salesforce.salesforce_create_lead gmail.gmail_trash_email gmail.gmail_get_email github.create_pull_request clickup.clickup_create_task stripe.stripe_create_invoice +q116,Share a Google Drive file,verb_product,google_drive.gdrive_share_file,1.0,1.0,1.0,1.0000,1.0000,1,1,306,1.0,1,418,1.0,1,785,1.0,google_drive.gdrive_share_file google_drive.gdrive_update_file google_drive.gdrive_create_file google_drive.gdrive_list_folder google_drive.gdrive_get_file google_drive.gdrive_search slack.get_file filesystem.move_file github.create_or_update_file slack.delete_file +q117,List Google Calendar events,verb_product,google_calendar.gcal_list_events,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,427,1.0,1,716,1.0,google_calendar.gcal_list_events google_calendar.gcal_list_calendars google_calendar.gcal_delete_event google_calendar.gcal_create_event google_calendar.gcal_update_event kubernetes.k8s_get_events sentry.sentry_list_events pagerduty.pagerduty_list_incidents gmail.gmail_list_labels zendesk.zendesk_list_tickets +q118,Delete a calendar event,resource,google_calendar.gcal_delete_event,1.0,1.0,1.0,1.0000,1.0000,1,1,270,1.0,1,409,1.0,1,706,1.0,google_calendar.gcal_delete_event slack.delete_canvase google_calendar.gcal_create_event linear.delete_cycle google_calendar.gcal_update_event google_calendar.gcal_list_events linear.delete_label linear.delete_attachment notion.delete_comment jira.delete_sprint +q119,List Vercel projects,verb_product,vercel.vercel_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,175,1.0,1,255,1.0,1,519,1.0,vercel.vercel_list_projects vercel.vercel_list_deployments vercel.vercel_create_project jira.jira_list_projects linear.linear_list_projects sentry.sentry_list_projects gitlab.gitlab_list_projects asana.asana_list_tasks supabase.supabase_list_tables circleci.circleci_list_pipelines +q120,Select rows from Supabase,verb_product,supabase.supabase_select,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,354,1.0,1,672,1.0,supabase.supabase_select supabase.supabase_insert supabase.supabase_update supabase.supabase_list_tables supabase.supabase_invoke_fn mongodb.mongo_list_collections sqlite.sqlite_list_tables postgres.list_tables notion.notion_query_database datadog.datadog_list_logs +q121,Post a Figma comment,verb_product,figma.figma_post_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,201,1.0,1,308,1.0,1,653,1.0,figma.figma_post_comment figma.figma_get_comments linear.get_comment notion.get_comment figma.figma_get_file linear.update_comment notion.update_comment linear.linear_create_comment linear.create_comment linear.delete_comment +q122,List Okta users,verb_product,okta.okta_list_users,1.0,1.0,1.0,1.0000,1.0000,1,1,163,1.0,1,271,1.0,1,622,1.0,okta.okta_list_users okta.okta_get_user notion.notion_list_users okta.okta_create_user okta.okta_deactivate_user notion.list_users_tokens slack.slack_get_users github.list_gists linear.list_users slack.list_usergroups +q123,Range query Prometheus,verb_product,prometheus.prom_query_range,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,393,1.0,1,761,1.0,prometheus.prom_query_range prometheus.prom_query prometheus.prom_list_metrics datadog.datadog_query_metrics shopify.shopify_list_products memory.search_nodes datadog.datadog_list_logs datadog.datadog_list_monitors confluence.confluence_search sentry.sentry_list_issues +q124,Create a Grafana annotation,verb_product,grafana.grafana_create_annotation,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,461,1.0,1,960,1.0,grafana.grafana_create_annotation grafana.grafana_get_dashboard github.create_artifact notion.create_comment asana.asana_create_task linear.create_attachment notion.create_data_source gitlab.create_label jira.create_attachment slack.create_canvase +q125,Apply terraform,identifier,terraform.tf_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,171,1.0,1,287,1.0,1,648,1.0,terraform.tf_apply terraform.tf_plan terraform.tf_state_list kubernetes.k8s_apply playwright.browser_fill_form filesystem.search_files asana.asana_get_task asana.asana_update_task gmail.gmail_create_draft asana.asana_create_project +q126,List Asana tasks,verb_product,asana.asana_list_tasks,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,386,1.0,1,674,1.0,asana.asana_list_tasks asana.asana_get_task asana.asana_create_task asana.asana_update_task clickup.clickup_list_tasks okta.okta_list_users aws_s3.s3_list_buckets clickup.clickup_create_task asana.asana_create_project jira.jira_list_projects +q127,Comment on a Trello card,verb_product,trello.trello_add_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,390,1.0,1,682,1.0,trello.trello_add_comment trello.trello_list_cards trello.trello_create_card trello.trello_create_board figma.figma_post_comment linear.get_comment notion.get_comment figma.figma_get_comments linear.update_comment notion.update_comment +q128,Update a ClickUp task,verb_product,clickup.clickup_update_task,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,399,1.0,1,747,1.0,clickup.clickup_update_task clickup.clickup_get_task clickup.clickup_list_tasks clickup.clickup_create_task asana.asana_update_task linear.update_comment slack.update_canvase slack.update_bookmark gitlab.update_snippet jira.update_sprint +q129,Search Confluence,verb_product,confluence.confluence_search,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,369,1.0,1,651,1.0,confluence.confluence_search confluence.confluence_get_page confluence.confluence_update_page confluence.confluence_create_page hubspot.hubspot_search_contacts github.search_users grafana.grafana_search_dashboards github.search_orgs github.search_commits github.search_code +q130,Create a HubSpot deal,verb_product,hubspot.hubspot_create_deal,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,355,1.0,1,776,1.0,hubspot.hubspot_create_deal hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_list_contacts stripe.stripe_create_invoice stripe.stripe_create_customer slack.slack_create_channel slack.create_pin linear.linear_create_project slack.create_reminder +q131,List Zendesk tickets,verb_product,zendesk.zendesk_list_tickets,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,367,1.0,1,644,1.0,zendesk.zendesk_list_tickets zendesk.zendesk_get_ticket zendesk.zendesk_search zendesk.zendesk_update_ticket zendesk.zendesk_create_ticket google_calendar.gcal_list_calendars vercel.vercel_list_projects google_calendar.gcal_list_events jira.jira_list_projects jira.list_sprints +q132,List Discord channels,verb_product,discord.discord_list_channels,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,300,1.0,1,637,1.0,discord.discord_list_channels discord.discord_get_channel discord.discord_send_message discord.discord_list_members slack.slack_list_channels discord.discord_add_reaction slack.slack_get_channel_history slack.slack_join_channel slack.slack_leave_channel slack.slack_create_channel +q133,Set a Redis key,verb_product,redis.redis_get,0.0,1.0,1.0,0.5000,0.6309,2,1,182,1.0,1,284,1.0,1,611,1.0,redis.redis_get redis.redis_set redis.redis_delete redis.redis_keys slack.get_reminder slack.update_reminder playwright.browser_press_key aws_s3.s3_put_object jira.jira_get_issue slack.update_pin +q134,Insert MongoDB documents,verb_product,mongodb.mongo_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,381,1.0,1,733,1.0,mongodb.mongo_insert mongodb.mongo_update mongodb.mongo_find mongodb.mongo_list_collections mongodb.mongo_delete supabase.supabase_insert elasticsearch.es_index_doc elasticsearch.es_get_doc github.update_environment linear.update_attachment +q135,Run a MySQL query,verb_product,mysql.mysql_query,1.0,1.0,1.0,1.0000,1.0000,1,1,179,1.0,1,294,1.0,1,591,1.0,mysql.mysql_query sqlite.sqlite_query postgres.query salesforce.salesforce_query postgres.explain_query mysql.mysql_list_tables prometheus.prom_query github.search_users datadog.datadog_list_monitors notion.notion_query_database +q136,Index a document in Elasticsearch,verb_product,elasticsearch.es_index_doc,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,392,1.0,1,703,1.0,elasticsearch.es_index_doc elasticsearch.es_get_doc elasticsearch.es_search mongodb.mongo_find mongodb.mongo_insert google_drive.gdrive_search confluence.confluence_search mongodb.mongo_update memory.search_nodes hubspot.hubspot_search_contacts +q137,Get an S3 object,verb_product,aws_s3.s3_get_object,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,397,1.0,1,670,1.0,aws_s3.s3_get_object aws_s3.s3_delete_object aws_s3.s3_put_object aws_s3.s3_list_objects salesforce.salesforce_get_record gitlab.get_snippet slack.get_file asana.asana_get_task slack.get_canvase okta.okta_get_user +q138,List Cloudflare zones,verb_product,cloudflare.cf_list_zones,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,340,1.0,1,591,1.0,cloudflare.cf_list_zones cloudflare.cf_list_dns cloudflare.cf_purge_cache cloudflare.cf_create_dns circleci.circleci_list_pipelines docker.docker_images linear.linear_list_teams hubspot.hubspot_list_contacts gitlab.list_pipelines kubernetes.k8s_list_services +q139,Create a Stripe invoice,verb_product,stripe.stripe_create_invoice,1.0,1.0,1.0,1.0000,1.0000,1,1,185,1.0,1,352,1.0,1,694,1.0,stripe.stripe_create_invoice stripe.stripe_create_customer stripe.stripe_list_invoices stripe.stripe_create_price stripe.stripe_create_payment_intent stripe.stripe_create_product stripe.stripe_create_refund stripe.stripe_get_customer slack.create_file stripe.stripe_get_payment_intent +q140,List GitHub Actions workflows,verb_product,github.actions_list,1.0,1.0,1.0,1.0000,1.0000,1,1,293,1.0,1,452,1.0,1,803,1.0,github.actions_list github.actions_get github.actions_run_trigger github.actions_get_job_logs github.list_notifications github.list_gists github.list_hooks github.list_artifacts github.list_check_runs github.list_discussions +q141,Get GitHub Actions job logs,verb_product,github.actions_get_job_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,442,1.0,1,704,1.0,github.actions_get_job_logs github.actions_get github.actions_list github.actions_run_trigger github.get_check_run github.get_notification_details github.get_hook github.get_runner github.search_orgs github.get_milestone +q142,Create a GitHub repository,verb_product,github.create_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,333,1.0,1,545,1.0,1,915,1.0,github.create_repository github.create_branch github.create_issue github.create_or_update_file github.star_repository github.create_gist github.unstar_repository github.delete_repository github.fork_repository github.list_releases +q143,Checkout a git branch,verb_product,git.git_checkout,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,338,1.0,1,710,1.0,git.git_checkout git.git_branch github.get_check_run github.list_branches git.git_create_branch github.create_branch github.update_pull_request_branch github.get_gist git.git_diff git.git_show +q144,Show git log,identifier,git.git_log,1.0,1.0,1.0,1.0000,1.0000,1,1,194,1.0,1,312,1.0,1,651,1.0,git.git_log git.git_show git.git_status git.git_diff_unstaged git.git_diff github.actions_get_job_logs git.git_diff_staged git.git_branch git.git_commit git.git_create_branch +q145,List postgres schemas,verb_product,postgres.list_schemas,1.0,1.0,1.0,1.0000,1.0000,1,1,152,1.0,1,249,1.0,1,469,1.0,postgres.list_schemas postgres.list_tables mysql.mysql_list_tables postgres.describe_table docker.docker_images aws_s3.s3_list_buckets supabase.supabase_list_tables terraform.tf_state_list sqlite.sqlite_list_tables postgres.explain_query +q146,Open nodes in the knowledge graph,verb_product,memory.open_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,262,1.0,1,551,1.0,memory.open_nodes memory.search_nodes memory.create_entities memory.read_graph memory.create_relations memory.add_observations memory.delete_relations memory.delete_entities figma.figma_get_file_nodes notion.get_data_source +q147,Get file metadata from the filesystem,paraphrase,filesystem.get_file_info,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,334,1.0,1,685,1.0,filesystem.get_file_info filesystem.read_file google_drive.gdrive_get_file filesystem.read_text_file filesystem.read_media_file filesystem.list_directory filesystem.directory_tree filesystem.list_directory_with_sizes figma.figma_get_file_nodes github.get_file_contents +q148,Write a new local file,paraphrase,filesystem.write_file,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,381,1.0,1,764,1.0,filesystem.write_file filesystem.read_text_file filesystem.read_file filesystem.create_directory filesystem.edit_file slack.create_file filesystem.read_media_file slack.update_file google_drive.gdrive_create_file slack.get_file +q149,Search GitHub issues,verb_product,github.search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,209,1.0,1,314,1.0,1,702,1.0,github.search_issues github.search_users github.search_pull_requests github.search_orgs github.search_commits github.search_repositories github.search_code github.list_issues jira.jira_search_issues linear.linear_search_issues +q150,Get the authenticated GitHub user,paraphrase,github.get_teams,0.0,1.0,1.0,0.5000,0.6309,2,1,179,1.0,1,317,1.0,1,605,1.0,github.get_teams github.get_me github.list_notifications github.get_latest_release github.projects_get github.search_users github.get_runner github.get_notification_details github.list_gists github.get_label diff --git a/experiments/mcp-tool-search/results/hybrid_rrf_trigram_embed.json b/experiments/mcp-tool-search/results/hybrid_rrf_trigram_embed.json new file mode 100644 index 00000000..c2d96160 --- /dev/null +++ b/experiments/mcp-tool-search/results/hybrid_rrf_trigram_embed.json @@ -0,0 +1,7013 @@ +{ + "summary": { + "n": 150, + "index_s": 1.9648816620000389, + "rss_mb": 0.0, + "p50_ms": 2.585485999929915, + "p95_ms": 3.0305255500081776, + "recall@1": 0.9, + "recall@1_lo": 0.8533333333333334, + "recall@1_hi": 0.9466666666666667, + "recall@3": 0.9733333333333334, + "recall@3_lo": 0.9466666666666667, + "recall@3_hi": 0.9933333333333333, + "recall@5": 0.9933333333333333, + "recall@5_lo": 0.98, + "recall@5_hi": 1.0, + "mrr@10": 0.9398888888888888, + "mrr@10_lo": 0.9082222222222222, + "mrr@10_hi": 0.9683333333333334, + "ndcg@10": 0.9496191211692434, + "ndcg@10_lo": 0.9252086419271226, + "ndcg@10_hi": 0.9720002237819023 + }, + "meta": { + "name": "hybrid_rrf_trigram_embed", + "needs_model": true, + "needs_api": false, + "notes": "RRF char-trigram + bge-small" + }, + "rows": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "github.issue_write", + "gitlab.gitlab_create_issue", + "github.create_gist", + "github.create_repository", + "github.issue_read", + "github.update_issue_comment", + "github.projects_write", + "github.search_issues", + "github.list_issues" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 380, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 538, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1014, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "kind": "verb_product", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_create_issue", + "jira.create_board", + "jira.create_sprint", + "jira.create_version", + "jira.jira_create_sprint", + "jira.create_filter", + "jira.create_webhook", + "zendesk.zendesk_create_ticket", + "jira.create_attachment", + "jira.update_sprint" + ], + "hit1": "jira.jira_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 330, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 535, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 981, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "kind": "verb_product", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "ranking": [ + "slack.slack_post_message", + "slack.slack_add_reaction", + "slack.slack_get_channel_history", + "slack.slack_search_messages", + "slack.get_reminder", + "slack.slack_leave_channel", + "slack.get_pin", + "discord.discord_send_message", + "slack.slack_join_channel", + "slack.slack_create_channel" + ], + "hit1": "slack.slack_post_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 253, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "kind": "resource", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "ranking": [ + "bitbucket.bitbucket_list_prs", + "github.list_pull_requests", + "github.search_pull_requests", + "github.create_pull_request", + "github.pull_request_read", + "gitlab.gitlab_list_merge_requests", + "github.pull_request_review_write", + "github.update_pull_request", + "github.merge_pull_request", + "playwright.browser_tab_list" + ], + "hit1": "bitbucket.bitbucket_list_prs", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.66967181649423, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 477, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 976, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "ranking": [ + "github.create_pull_request", + "github.search_pull_requests", + "github.list_pull_requests", + "github.pull_request_read", + "github.merge_pull_request", + "github.request_copilot_review", + "github.pull_request_review_write", + "github.update_pull_request_branch", + "github.update_pull_request", + "github.add_comment_to_pending_review" + ], + "hit1": "github.create_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 314, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 509, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1144, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "kind": "verb_product", + "gold": [ + "postgres.query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "mysql.mysql_query", + "sqlite.sqlite_query", + "salesforce.salesforce_query", + "postgres.explain_query", + "postgres.list_schemas", + "prometheus.prom_query", + "postgres.describe_table", + "postgres.list_tables", + "datadog.datadog_list_monitors" + ], + "hit1": "postgres.query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 560, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "kind": "paraphrase", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "ranking": [ + "filesystem.read_text_file", + "filesystem.read_file", + "filesystem.read_media_file", + "github.get_file_contents", + "figma.figma_get_file", + "figma.figma_get_file_nodes", + "gitlab.gitlab_get_file", + "slack.get_file", + "filesystem.get_file_info", + "filesystem.directory_tree" + ], + "hit1": "filesystem.read_text_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8597186998521972, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "kind": "paraphrase", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "ranking": [ + "google_drive.gdrive_list_folder", + "git.git_add", + "filesystem.list_allowed_directories", + "slack.get_file", + "filesystem.list_directory", + "slack.list_files", + "slack.delete_file", + "google_drive.gdrive_search", + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "hit1": "google_drive.gdrive_list_folder", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.2, + "ndcg@10": 0.43558959919022855, + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 387, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 290, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 587, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "ranking": [ + "memory.read_graph", + "memory.open_nodes", + "memory.create_relations", + "memory.delete_relations", + "memory.add_observations", + "memory.create_entities", + "memory.search_nodes", + "memory.delete_entities", + "pagerduty.pagerduty_acknowledge_incident", + "sequential_thinking.sequentialthinking" + ], + "hit1": "memory.read_graph", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.2, + "ndcg@10": 0.42947281281199595, + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 334, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 263, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "kind": "verb_product", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "ranking": [ + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "github.search_pull_requests", + "github.search_users", + "github.search_orgs", + "slack.slack_search_messages", + "confluence.confluence_search", + "grafana.grafana_search_dashboards", + "hubspot.hubspot_search_contacts", + "memory.search_nodes" + ], + "hit1": "brave_search.brave_web_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 637, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "kind": "verb_product", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "ranking": [ + "fetch.fetch", + "confluence.confluence_search", + "playwright.browser_navigate", + "filesystem.read_file", + "github.get_file_contents", + "docker.docker_logs", + "puppeteer.puppeteer_navigate", + "playwright.browser_take_screenshot", + "google_drive.gdrive_get_file", + "playwright.browser_tab_new" + ], + "hit1": "fetch.fetch", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 369, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 668, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "kind": "identifier", + "gold": [ + "git.git_status" + ], + "alts": [], + "ranking": [ + "git.git_status", + "git.git_diff_unstaged", + "github.list_milestones", + "git.git_show", + "git.git_log", + "git.git_diff_staged", + "github.list_check_runs", + "gitlab.list_snippets", + "github.list_hooks", + "git.git_diff" + ], + "hit1": "git.git_status", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "kind": "paraphrase", + "gold": [ + "git.git_commit" + ], + "alts": [], + "ranking": [ + "git.git_commit", + "git.git_reset", + "git.git_diff_staged", + "git.git_show", + "salesforce.salesforce_list_opportunities", + "git.git_add", + "github.list_milestones", + "gitlab.list_releases", + "gitlab.list_milestones", + "zendesk.zendesk_update_ticket" + ], + "hit1": "git.git_commit", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 695, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_issue", + "linear.linear_create_comment", + "linear.linear_get_issue", + "linear.linear_assign_issue", + "linear.linear_update_issue", + "linear.linear_create_project", + "linear.create_label", + "linear.create_cycle", + "linear.linear_search_issues", + "linear.linear_list_issues" + ], + "hit1": "linear.linear_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 275, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 875, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "kind": "paraphrase", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_list_projects", + "linear.linear_create_project", + "linear.linear_create_issue", + "asana.asana_list_tasks", + "linear.get_cycle", + "linear.get_label", + "linear.get_attachment", + "linear.get_user", + "linear.create_label", + "linear.linear_update_issue" + ], + "hit1": "linear.linear_list_projects", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 284, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "ranking": [ + "github.update_issue_comment", + "github.add_issue_comment", + "github.get_discussion_comments", + "jira.jira_add_comment", + "github.issue_read", + "figma.figma_post_comment", + "github.add_comment_to_pending_review", + "linear.linear_create_comment", + "github.assign_copilot_to_issue", + "github.request_copilot_review" + ], + "hit1": "github.update_issue_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 327, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 494, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 993, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "kind": "ambiguous", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "ranking": [ + "github.merge_pull_request", + "gitlab.gitlab_merge_merge_request", + "github.update_pull_request_branch", + "gitlab.gitlab_get_merge_request", + "github.search_pull_requests", + "gitlab.gitlab_list_merge_requests", + "github.pull_request_read", + "bitbucket.bitbucket_list_prs", + "github.update_pull_request", + "github.create_pull_request" + ], + "hit1": "github.merge_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 299, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 431, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 962, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "kind": "verb_product", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "ranking": [ + "github.actions_run_trigger", + "github.actions_get", + "github.actions_list", + "github.actions_get_job_logs", + "github.update_gist", + "github.dismiss_notification", + "github.mark_all_notifications_read", + "github.get_hook", + "github.get_notification_details", + "github.update_runner" + ], + "hit1": "github.actions_run_trigger", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 468, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "kind": "resource", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "ranking": [ + "github.get_dependabot_alert", + "github.list_dependabot_alerts", + "github.get_secret_scanning_alert", + "github.get_code_scanning_alert", + "github.list_secret_scanning_alerts", + "github.list_code_scanning_alerts", + "vercel.vercel_get_deployment", + "github.get_notification_details", + "github.get_hook", + "github.get_check_run" + ], + "hit1": "github.get_dependabot_alert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8597186998521972, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 286, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 779, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "kind": "verb_product", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_issues", + "sentry.sentry_list_events", + "sentry.sentry_list_projects", + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "linear.linear_list_issues", + "gitlab.gitlab_list_issues", + "github.list_issues", + "pagerduty.pagerduty_list_incidents", + "linear.list_cycles" + ], + "hit1": "sentry.sentry_list_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 806, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_create_customer", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_customer", + "stripe.stripe_list_invoices", + "stripe.stripe_create_price", + "stripe.stripe_create_product", + "stripe.stripe_list_customers", + "stripe.stripe_create_refund", + "stripe.stripe_list_subscriptions" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "kind": "paraphrase", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_refund", + "stripe.stripe_get_payment_intent", + "stripe.stripe_get_customer", + "stripe.stripe_cancel_subscription", + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_price", + "stripe.stripe_create_invoice", + "stripe.stripe_create_product", + "stripe.stripe_list_invoices", + "stripe.stripe_create_customer" + ], + "hit1": "stripe.stripe_create_refund", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 197, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "kind": "verb_product", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "ranking": [ + "gmail.gmail_send_email", + "gmail.gmail_get_email", + "gmail.gmail_search_emails", + "gmail.gmail_trash_email", + "gmail.gmail_list_labels", + "twilio.twilio_send_sms", + "gmail.gmail_modify_email", + "gmail.gmail_create_draft", + "stripe.stripe_create_customer", + "shopify.shopify_create_order" + ], + "hit1": "gmail.gmail_send_email", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "kind": "verb_product", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "ranking": [ + "gmail.gmail_search_emails", + "slack.slack_search_messages", + "gmail.gmail_list_labels", + "hubspot.hubspot_search_contacts", + "github.search_users", + "google_drive.gdrive_search", + "gmail.gmail_get_email", + "github.search_orgs", + "github.search_pull_requests", + "gmail.gmail_send_email" + ], + "hit1": "gmail.gmail_search_emails", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 607, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "kind": "paraphrase", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_update_file", + "google_drive.gdrive_share_file", + "google_drive.gdrive_search", + "google_drive.gdrive_create_file", + "google_drive.gdrive_list_folder", + "google_drive.gdrive_get_file", + "slack.slack_upload_file", + "aws_s3.s3_put_object", + "slack.update_file", + "figma.figma_get_file" + ], + "hit1": "google_drive.gdrive_update_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 496, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_create_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_list_events", + "clickup.clickup_create_task", + "google_calendar.gcal_list_calendars", + "pagerduty.pagerduty_create_incident", + "linear.create_cycle", + "trello.trello_create_board", + "slack.create_canvase" + ], + "hit1": "google_calendar.gcal_create_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 480, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 882, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_list_pods", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_apply", + "docker.docker_images", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_delete" + ], + "hit1": "kubernetes.k8s_list_pods", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_get_pod", + "docker.docker_logs", + "github.actions_get_job_logs", + "slack.slack_get_channel_history", + "kubernetes.k8s_list_pods", + "datadog.datadog_list_logs", + "gitlab.get_snippet", + "github.get_runner", + "git.git_log" + ], + "hit1": "kubernetes.k8s_pod_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 754, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "ranking": [ + "docker.docker_run", + "docker.docker_stop", + "docker.docker_exec", + "docker.docker_ps", + "docker.docker_logs", + "docker.docker_rm", + "docker.docker_build", + "docker.docker_images", + "kubernetes.k8s_pod_logs", + "salesforce.salesforce_query" + ], + "hit1": "docker.docker_run", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "kind": "verb_product", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "ranking": [ + "docker.docker_logs", + "docker.docker_stop", + "kubernetes.k8s_pod_logs", + "docker.docker_ps", + "docker.docker_run", + "docker.docker_images", + "docker.docker_exec", + "git.git_log", + "docker.docker_rm", + "docker.docker_build" + ], + "hit1": "docker.docker_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 365, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 649, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "kind": "ambiguous", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "ranking": [ + "playwright.browser_navigate", + "puppeteer.puppeteer_navigate", + "playwright.browser_tab_new", + "playwright.browser_press_key", + "playwright.browser_click", + "playwright.browser_evaluate", + "playwright.browser_tab_list", + "playwright.browser_type", + "playwright.browser_take_screenshot", + "playwright.browser_tab_close" + ], + "hit1": "playwright.browser_navigate", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 581, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "kind": "ambiguous", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "ranking": [ + "playwright.browser_take_screenshot", + "puppeteer.puppeteer_screenshot", + "playwright.browser_snapshot", + "playwright.browser_evaluate", + "playwright.browser_press_key", + "playwright.browser_click", + "pagerduty.pagerduty_acknowledge_incident", + "confluence.confluence_get_page", + "notion.notion_get_page", + "pagerduty.pagerduty_resolve_incident" + ], + "hit1": "playwright.browser_take_screenshot", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 623, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_create_merge_request", + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_get_merge_request", + "gitlab.gitlab_list_merge_requests", + "github.merge_pull_request", + "gitlab.gitlab_create_pipeline", + "gitlab.create_release", + "gitlab.gitlab_create_issue", + "gitlab.create_job", + "gitlab.create_pipeline" + ], + "hit1": "gitlab.gitlab_create_merge_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 931, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "kind": "verb_product", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "ranking": [ + "notion.notion_search", + "notion.notion_get_page", + "notion.notion_update_page", + "notion.notion_create_page", + "notion.notion_get_comments", + "github.search_repositories", + "memory.search_nodes", + "github.search_orgs", + "notion.notion_create_database", + "notion.notion_get_block" + ], + "hit1": "notion.notion_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 692, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "kind": "verb_product", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "ranking": [ + "notion.notion_create_page", + "notion.notion_get_page", + "notion.notion_update_page", + "notion.notion_create_database", + "notion.create_comment", + "notion.create_data_source", + "notion.create_users_token", + "notion.notion_get_comments", + "confluence.confluence_create_page", + "notion.update_comment" + ], + "hit1": "notion.notion_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 413, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 832, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "kind": "verb_product", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "ranking": [ + "notion.notion_query_database", + "notion.notion_search", + "notion.notion_create_database", + "notion.notion_create_page", + "notion.get_data_source", + "notion.update_data_source", + "notion.notion_get_block", + "notion.notion_get_page", + "notion.delete_data_source", + "notion.create_data_source" + ], + "hit1": "notion.notion_query_database", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_search", + "jira.jira_create_issue", + "jira.create_board", + "slack.create_reminder", + "jira.create_sprint", + "confluence.confluence_create_page" + ], + "hit1": "zendesk.zendesk_get_ticket", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 871, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_search_contacts", + "hubspot.hubspot_create_deal", + "cloudflare.cf_create_dns", + "zendesk.zendesk_create_ticket", + "jira.create_attachment", + "slack.slack_create_channel", + "pagerduty.pagerduty_create_incident" + ], + "hit1": "hubspot.hubspot_create_contact", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 826, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "kind": "verb_product", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_create_lead", + "salesforce.salesforce_convert_lead", + "salesforce.salesforce_create_record", + "salesforce.salesforce_list_opportunities", + "salesforce.salesforce_get_record", + "salesforce.salesforce_update_record", + "salesforce.salesforce_query", + "hubspot.hubspot_create_contact", + "bitbucket.bitbucket_create_pr", + "confluence.confluence_create_page" + ], + "hit1": "salesforce.salesforce_create_lead", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 830, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "kind": "identifier", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_query", + "mysql.mysql_query", + "sqlite.sqlite_query", + "postgres.query", + "postgres.explain_query", + "prometheus.prom_query", + "notion.notion_query_database", + "datadog.datadog_list_monitors", + "hubspot.hubspot_search_contacts", + "github.search_users" + ], + "hit1": "salesforce.salesforce_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "kind": "verb_product", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "ranking": [ + "shopify.shopify_list_orders", + "shopify.shopify_get_order", + "shopify.shopify_list_customers", + "shopify.shopify_create_order", + "shopify.shopify_list_products", + "shopify.shopify_get_product", + "shopify.shopify_update_product", + "shopify.shopify_create_product", + "okta.okta_list_users", + "stripe.stripe_list_customers" + ], + "hit1": "shopify.shopify_list_orders", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_list_incidents", + "notion.notion_create_page", + "confluence.confluence_create_page", + "pagerduty.pagerduty_list_oncalls", + "clickup.clickup_create_task", + "notion.create_comment" + ], + "hit1": "pagerduty.pagerduty_create_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 831, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "kind": "paraphrase", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_list_oncalls", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_list_incidents", + "playwright.browser_press_key", + "twilio.twilio_make_call", + "notion.notion_update_page", + "notion.notion_get_page" + ], + "hit1": "pagerduty.pagerduty_list_oncalls", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "kind": "verb_product", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "ranking": [ + "datadog.datadog_query_metrics", + "datadog.datadog_list_monitors", + "datadog.datadog_create_monitor", + "datadog.datadog_list_logs", + "datadog.datadog_get_monitor", + "notion.notion_search", + "slack.slack_search_messages", + "prometheus.prom_list_metrics", + "notion.notion_query_database", + "prometheus.prom_query" + ], + "hit1": "datadog.datadog_query_metrics", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 783, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "kind": "verb_product", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "ranking": [ + "datadog.datadog_list_logs", + "datadog.datadog_list_monitors", + "datadog.datadog_get_monitor", + "slack.slack_search_messages", + "github.search_orgs", + "github.search_users", + "datadog.datadog_query_metrics", + "datadog.datadog_create_monitor", + "github.search_pull_requests", + "notion.notion_search" + ], + "hit1": "datadog.datadog_list_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_list_buckets", + "aws_s3.s3_list_objects", + "bitbucket.bitbucket_list_repos", + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "bitbucket.bitbucket_list_prs", + "aws_s3.s3_put_object", + "jira.jira_list_projects", + "gitlab.list_snippets", + "salesforce.salesforce_list_opportunities" + ], + "hit1": "aws_s3.s3_list_buckets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 656, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "kind": "paraphrase", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_put_object", + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_list_objects", + "slack.slack_upload_file", + "twilio.twilio_send_sms", + "salesforce.salesforce_update_record", + "salesforce.salesforce_create_record", + "gmail.gmail_create_draft", + "salesforce.salesforce_get_record" + ], + "hit1": "aws_s3.s3_put_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 808, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "cloudflare.cf_list_dns", + "cloudflare.cf_create_dns", + "google_calendar.gcal_delete_event", + "slack.delete_bookmark", + "slack.delete_canvase", + "playwright.browser_tab_close", + "playwright.browser_snapshot", + "postgres.list_tables" + ], + "hit1": "cloudflare.cf_purge_cache", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 608, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "salesforce.salesforce_create_record", + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "salesforce.salesforce_update_record", + "confluence.confluence_create_page", + "slack.create_canvase", + "slack.create_file", + "slack.create_reminder" + ], + "hit1": "cloudflare.cf_create_dns", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 844, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_deployments", + "vercel.vercel_get_deployment", + "vercel.vercel_list_projects", + "kubernetes.k8s_list_deployments", + "vercel.vercel_create_project", + "github.list_environments", + "sentry.sentry_list_projects", + "jira.jira_list_projects", + "sentry.sentry_list_events", + "docker.docker_images" + ], + "hit1": "vercel.vercel_list_deployments", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 558, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "ranking": [ + "supabase.supabase_insert", + "supabase.supabase_update", + "supabase.supabase_select", + "mongodb.mongo_insert", + "supabase.supabase_invoke_fn", + "supabase.supabase_list_tables", + "notion.notion_create_database", + "linear.get_user", + "gmail.gmail_create_draft", + "gitlab.get_release" + ], + "hit1": "supabase.supabase_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 701, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "kind": "paraphrase", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "ranking": [ + "figma.figma_get_images", + "figma.figma_get_file_nodes", + "figma.figma_get_file", + "docker.docker_images", + "filesystem.read_media_file", + "figma.figma_get_comments", + "figma.figma_post_comment", + "docker.docker_run", + "github.get_file_contents", + "docker.docker_build" + ], + "hit1": "figma.figma_get_images", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "kind": "verb_product", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "ranking": [ + "twilio.twilio_send_sms", + "twilio.twilio_list_messages", + "twilio.twilio_make_call", + "gmail.gmail_send_email", + "discord.discord_send_message", + "slack.slack_add_reaction", + "slack.get_reminder", + "sentry.sentry_list_issues", + "aws_s3.s3_put_object", + "github.add_comment_to_pending_review" + ], + "hit1": "twilio.twilio_send_sms", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 890, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "kind": "verb_product", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "ranking": [ + "okta.okta_create_user", + "okta.okta_get_user", + "okta.okta_deactivate_user", + "okta.okta_list_users", + "linear.create_user", + "notion.create_users_token", + "slack.create_usergroup", + "asana.asana_create_project", + "asana.asana_create_task", + "jira.create_filter" + ], + "hit1": "okta.okta_create_user", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 806, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query", + "prometheus.prom_query_range", + "prometheus.prom_list_metrics", + "datadog.datadog_query_metrics", + "salesforce.salesforce_query", + "elasticsearch.es_search", + "sqlite.sqlite_query", + "memory.search_nodes", + "postgres.query", + "hubspot.hubspot_search_contacts" + ], + "hit1": "prometheus.prom_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "kind": "verb_product", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "ranking": [ + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "github.search_pull_requests", + "github.search_users", + "github.search_orgs", + "github.search_issues", + "grafana.grafana_create_annotation", + "confluence.confluence_search", + "slack.slack_search_messages", + "datadog.datadog_list_logs" + ], + "hit1": "grafana.grafana_search_dashboards", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 287, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "kind": "verb_product", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "ranking": [ + "circleci.circleci_trigger_pipeline", + "circleci.circleci_get_pipeline", + "circleci.circleci_list_pipelines", + "hubspot.hubspot_create_deal", + "gitlab.update_pipeline", + "gitlab.gitlab_create_pipeline", + "gitlab.get_pipeline", + "gitlab.list_pipelines", + "github.actions_run_trigger", + "gitlab.delete_pipeline" + ], + "hit1": "circleci.circleci_trigger_pipeline", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 359, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "kind": "identifier", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "ranking": [ + "terraform.tf_plan", + "terraform.tf_apply", + "terraform.tf_state_list", + "github.get_check_run", + "github.update_check_run", + "docker.docker_run", + "github.create_check_run", + "docker.docker_exec", + "github.update_runner", + "github.delete_check_run" + ], + "hit1": "terraform.tf_plan", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 672, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "kind": "verb_product", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "ranking": [ + "asana.asana_create_task", + "asana.asana_get_task", + "asana.asana_create_project", + "asana.asana_update_task", + "asana.asana_list_tasks", + "clickup.clickup_create_task", + "okta.okta_create_user", + "grafana.grafana_create_annotation", + "github.create_issue", + "gmail.gmail_create_draft" + ], + "hit1": "asana.asana_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 846, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "ranking": [ + "trello.trello_create_card", + "trello.trello_create_board", + "trello.trello_add_comment", + "trello.trello_list_cards", + "stripe.stripe_create_invoice", + "stripe.stripe_create_customer", + "stripe.stripe_create_product", + "grafana.grafana_create_annotation", + "jira.jira_create_sprint", + "shopify.shopify_create_order" + ], + "hit1": "trello.trello_create_card", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_create_task", + "clickup.clickup_get_task", + "clickup.clickup_list_tasks", + "clickup.clickup_update_task", + "asana.asana_create_task", + "jira.create_sprint", + "gmail.gmail_create_draft", + "confluence.confluence_create_page", + "trello.trello_create_board", + "jira.create_filter" + ], + "hit1": "clickup.clickup_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 415, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 839, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "kind": "verb_product", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "ranking": [ + "confluence.confluence_create_page", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "confluence.confluence_search", + "notion.notion_create_page", + "pagerduty.pagerduty_create_incident", + "cloudflare.cf_create_dns", + "bitbucket.bitbucket_create_issue", + "slack.create_canvase", + "datadog.datadog_create_monitor" + ], + "hit1": "confluence.confluence_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 922, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "kind": "verb_product", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "discord.discord_add_reaction", + "discord.discord_get_channel", + "slack.slack_post_message", + "gmail.gmail_trash_email", + "discord.discord_list_members", + "slack.slack_add_reaction", + "discord.discord_list_channels", + "gmail.gmail_get_email", + "figma.figma_post_comment" + ], + "hit1": "discord.discord_send_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_search", + "confluence.confluence_search", + "elasticsearch.es_get_doc", + "hubspot.hubspot_search_contacts", + "github.search_users", + "elasticsearch.es_index_doc", + "github.search_orgs", + "github.search_pull_requests", + "slack.slack_search_messages", + "grafana.grafana_search_dashboards" + ], + "hit1": "elasticsearch.es_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 633, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "redis.redis_set", + "redis.redis_delete", + "redis.redis_keys", + "figma.figma_get_file", + "jira.jira_get_issue", + "github.get_runner", + "slack.get_reminder", + "github.get_gist", + "github.get_milestone" + ], + "hit1": "redis.redis_get", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 283, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 548, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_find", + "mongodb.mongo_update", + "mongodb.mongo_delete", + "mongodb.mongo_list_collections", + "mongodb.mongo_insert", + "elasticsearch.es_get_doc", + "elasticsearch.es_index_doc", + "filesystem.list_directory", + "filesystem.list_directory_with_sizes", + "github.list_environments" + ], + "hit1": "mongodb.mongo_find", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "kind": "verb_product", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "ranking": [ + "sqlite.sqlite_list_tables", + "mysql.mysql_list_tables", + "sqlite.sqlite_describe_table", + "supabase.supabase_list_tables", + "postgres.list_tables", + "postgres.list_schemas", + "linear.list_cycles", + "okta.okta_list_users", + "gitlab.list_milestones", + "sqlite.sqlite_query" + ], + "hit1": "sqlite.sqlite_list_tables", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 240, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 537, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "kind": "verb_product", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "ranking": [ + "postgres.describe_table", + "sqlite.sqlite_describe_table", + "postgres.explain_query", + "postgres.list_tables", + "postgres.list_schemas", + "postgres.query", + "linear.create_user", + "jira.create_filter", + "jira.create_board", + "supabase.supabase_insert" + ], + "hit1": "postgres.describe_table", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 274, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 679, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "gitlab.gitlab_create_issue", + "github.create_gist", + "github.issue_write", + "github.create_repository", + "bitbucket.bitbucket_create_issue", + "github.create_package", + "github.create_hook", + "github.create_runner", + "github.search_issues" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9238850086262381, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 320, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 536, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 998, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "ranking": [ + "gmail.gmail_create_draft", + "okta.okta_create_user", + "clickup.clickup_create_task", + "stripe.stripe_create_product", + "trello.trello_create_board", + "linear.create_user", + "stripe.stripe_create_invoice", + "confluence.confluence_create_page", + "vercel.vercel_create_project", + "salesforce.salesforce_create_record" + ], + "hit1": "gmail.gmail_create_draft", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 39, + "k3_calls": 13, + "k3_tokens": 3380, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 8, + "k5_tokens": 3426, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3354, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q071", + "query": "list-directory", + "kind": "identifier", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "ranking": [ + "filesystem.list_directory", + "filesystem.list_directory_with_sizes", + "filesystem.list_allowed_directories", + "sentry.sentry_list_projects", + "jira.jira_list_projects", + "datadog.datadog_list_monitors", + "mongodb.mongo_list_collections", + "vercel.vercel_list_projects", + "gitlab.gitlab_list_projects", + "okta.okta_list_users" + ], + "hit1": "filesystem.list_directory", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 256, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 508, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "kind": "paraphrase", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "ranking": [ + "sequential_thinking.sequentialthinking", + "memory.add_observations", + "playwright.browser_wait_for", + "memory.create_entities", + "memory.read_graph", + "grafana.grafana_create_annotation", + "filesystem.edit_file", + "memory.create_relations", + "confluence.confluence_search", + "memory.search_nodes" + ], + "hit1": "sequential_thinking.sequentialthinking", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 676, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "kind": "paraphrase", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "ranking": [ + "time.get_current_time", + "time.convert_time", + "playwright.browser_wait_for", + "grafana.grafana_create_annotation", + "google_calendar.gcal_list_events", + "jira.jira_add_worklog", + "slack.slack_add_reaction", + "jira.list_sprints", + "jira.list_boards", + "zendesk.zendesk_list_tickets" + ], + "hit1": "time.get_current_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 780, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "kind": "verb_product", + "gold": [ + "time.convert_time" + ], + "alts": [], + "ranking": [ + "time.convert_time", + "time.get_current_time", + "grafana.grafana_create_annotation", + "cloudflare.cf_list_zones", + "playwright.browser_wait_for", + "cloudflare.cf_create_dns", + "datadog.datadog_query_metrics", + "salesforce.salesforce_convert_lead", + "google_calendar.gcal_list_events", + "slack.slack_add_reaction" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 776, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "kind": "verb_product", + "gold": [ + "github.star_repository" + ], + "alts": [], + "ranking": [ + "github.star_repository", + "github.unstar_repository", + "github.list_starred_repositories", + "github.list_pull_requests", + "github.get_file_blame", + "github.get_label", + "github.list_releases", + "github.list_labels", + "github.actions_get_job_logs", + "github.get_release_by_tag" + ], + "hit1": "github.star_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 809, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "kind": "resource", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "ranking": [ + "github.fork_repository", + "github.unstar_repository", + "github.star_repository", + "github.delete_repository", + "github.list_releases", + "github.list_dependabot_alerts", + "github.create_repository", + "github.get_repository_tree", + "github.list_labels", + "github.list_discussions" + ], + "hit1": "github.fork_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 783, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "kind": "verb_product", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "ranking": [ + "github.get_latest_release", + "github.get_release_by_tag", + "gitlab.get_release", + "github.list_releases", + "github.get_gist", + "github.get_milestone", + "github.get_hook", + "github.get_artifact", + "github.get_package", + "github.get_environment" + ], + "hit1": "github.get_latest_release", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 624, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "kind": "resource", + "gold": [ + "github.create_gist" + ], + "alts": [], + "ranking": [ + "github.create_gist", + "github.update_gist", + "gitlab.create_snippet", + "github.create_hook", + "gitlab.create_label", + "gitlab.create_wiki", + "gmail.gmail_create_draft", + "gitlab.gitlab_create_merge_request", + "github.projects_write", + "github.create_milestone" + ], + "hit1": "github.create_gist", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 280, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 473, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 963, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "kind": "verb_product", + "gold": [ + "github.search_code" + ], + "alts": [], + "ranking": [ + "github.search_code", + "github.search_users", + "github.search_orgs", + "github.search_commits", + "github.search_pull_requests", + "github.search_repositories", + "github.search_issues", + "github.get_code_scanning_alert", + "github.list_code_scanning_alerts", + "gitlab.gitlab_list_projects" + ], + "hit1": "github.search_code", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 719, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "kind": "verb_product", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "ranking": [ + "github.list_notifications", + "github.mark_all_notifications_read", + "github.get_notification_details", + "github.dismiss_notification", + "github.list_gists", + "github.actions_list", + "github.list_secret_scanning_alerts", + "github.list_releases", + "github.list_code_scanning_alerts", + "github.list_artifacts" + ], + "hit1": "github.list_notifications", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 176, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 279, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "kind": "verb_product", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "ranking": [ + "github.assign_copilot_to_issue", + "linear.linear_assign_issue", + "jira.jira_assign_issue", + "github.create_issue", + "linear.linear_create_comment", + "github.request_copilot_review", + "linear.linear_list_issues", + "jira.jira_add_comment", + "jira.jira_get_issue", + "jira.jira_update_issue" + ], + "hit1": "github.assign_copilot_to_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 466, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 838, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "kind": "resource", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "ranking": [ + "github.list_secret_scanning_alerts", + "github.get_secret_scanning_alert", + "github.get_code_scanning_alert", + "github.list_code_scanning_alerts", + "github.get_dependabot_alert", + "github.list_dependabot_alerts", + "github.get_check_run", + "gitlab.get_snippet", + "github.get_hook", + "slack.get_pin" + ], + "hit1": "github.list_secret_scanning_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 483, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 775, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "ranking": [ + "github.create_or_update_file", + "github.projects_write", + "github.update_gist", + "github.create_gist", + "github.update_hook", + "github.update_package", + "github.update_runner", + "github.update_environment", + "github.issue_write", + "github.update_check_run" + ], + "hit1": "github.create_or_update_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 476, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 876, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "kind": "resource", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "ranking": [ + "github.list_repository_collaborators", + "github.list_labels", + "github.list_tags", + "github.list_discussions", + "github.list_releases", + "github.list_commits", + "github.list_branches", + "github.list_starred_repositories", + "github.list_issues", + "github.list_dependabot_alerts" + ], + "hit1": "github.list_repository_collaborators", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 792, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "kind": "verb_product", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "ranking": [ + "slack.slack_get_thread_replies", + "slack.slack_reply_to_thread", + "slack.get_reminder", + "slack.get_pin", + "slack.update_reminder", + "slack.get_canvase", + "slack.get_file", + "slack.delete_reminder", + "slack.update_pin", + "slack.slack_add_reaction" + ], + "hit1": "slack.slack_get_thread_replies", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 365, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 688, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "kind": "verb_product", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "ranking": [ + "slack.slack_add_reaction", + "discord.discord_add_reaction", + "slack.create_reminder", + "slack.get_reminder", + "slack.create_pin", + "slack.create_canvase", + "slack.update_reminder", + "slack.get_canvase", + "slack.create_usergroup", + "slack.create_file" + ], + "hit1": "slack.slack_add_reaction", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 302, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 454, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 890, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "kind": "verb_product", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "ranking": [ + "slack.slack_search_messages", + "slack.slack_get_channel_history", + "slack.slack_add_reaction", + "slack.slack_post_message", + "slack.get_canvase", + "slack.get_reminder", + "slack.get_file", + "slack.get_pin", + "gmail.gmail_search_emails", + "github.search_users" + ], + "hit1": "slack.slack_search_messages", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 661, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "kind": "verb_product", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "ranking": [ + "slack.slack_create_channel", + "slack.slack_leave_channel", + "slack.slack_join_channel", + "slack.slack_post_message", + "slack.slack_set_channel_topic", + "slack.slack_get_thread_replies", + "slack.slack_reply_to_thread", + "slack.slack_add_reaction", + "slack.create_pin", + "slack.create_usergroup" + ], + "hit1": "slack.slack_create_channel", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 795, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "kind": "verb_product", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_transition_issue", + "jira.jira_assign_issue", + "jira.jira_get_issue", + "jira.jira_update_issue", + "jira.jira_add_comment", + "jira.jira_create_issue", + "jira.jira_search_issues", + "jira.delete_version", + "jira.get_version", + "jira.jira_add_worklog" + ], + "hit1": "jira.jira_transition_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "kind": "identifier", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "ranking": [ + "jira.jira_search_issues", + "github.search_pull_requests", + "jira.get_webhook", + "github.search_users", + "jira.get_filter", + "hubspot.hubspot_search_contacts", + "github.search_orgs", + "jira.jira_list_projects", + "confluence.confluence_search", + "slack.slack_search_messages" + ], + "hit1": "jira.jira_search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 199, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 566, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "kind": "verb_product", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "ranking": [ + "jira.jira_add_worklog", + "jira.jira_add_comment", + "jira.get_webhook", + "jira.update_webhook", + "jira.get_sprint", + "jira.jira_list_projects", + "jira.jira_transition_issue", + "jira.create_webhook", + "jira.get_board", + "jira.jira_get_issue" + ], + "hit1": "jira.jira_add_worklog", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 233, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "kind": "verb_product", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "ranking": [ + "linear.linear_list_teams", + "linear.linear_list_projects", + "linear.linear_list_issues", + "linear.list_cycles", + "linear.linear_create_project", + "linear.list_labels", + "linear.list_users", + "linear.get_cycle", + "linear.linear_create_issue", + "linear.list_attachments" + ], + "hit1": "linear.linear_list_teams", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 745, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "ranking": [ + "linear.linear_create_comment", + "linear.linear_get_issue", + "linear.get_comment", + "linear.linear_assign_issue", + "linear.delete_comment", + "linear.update_comment", + "linear.linear_update_issue", + "linear.linear_create_issue", + "linear.linear_search_issues", + "linear.create_comment" + ], + "hit1": "linear.linear_create_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "kind": "paraphrase", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "ranking": [ + "filesystem.read_text_file", + "filesystem.move_file", + "filesystem.read_media_file", + "filesystem.read_file", + "slack.update_file", + "slack.get_file", + "slack.delete_file", + "filesystem.write_file", + "google_drive.gdrive_update_file", + "filesystem.edit_file" + ], + "hit1": "filesystem.read_text_file", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 225, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 708, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "kind": "paraphrase", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "ranking": [ + "filesystem.search_files", + "filesystem.directory_tree", + "google_drive.gdrive_search", + "redis.redis_keys", + "memory.search_nodes", + "notion.notion_search", + "linear.linear_search_issues", + "github.search_repositories", + "github.search_code", + "filesystem.list_directory" + ], + "hit1": "filesystem.search_files", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 304, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "kind": "paraphrase", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "ranking": [ + "filesystem.edit_file", + "slack.update_file", + "filesystem.write_file", + "slack.create_file", + "slack.get_file", + "slack.delete_file", + "filesystem.move_file", + "github.create_or_update_file", + "git.git_add", + "slack.slack_upload_file" + ], + "hit1": "filesystem.edit_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 812, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "ranking": [ + "memory.read_graph", + "memory.open_nodes", + "memory.delete_relations", + "memory.search_nodes", + "memory.create_entities", + "memory.create_relations", + "memory.add_observations", + "memory.delete_entities", + "github.mark_all_notifications_read", + "filesystem.read_file" + ], + "hit1": "memory.read_graph", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 154, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 259, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 513, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "kind": "verb_product", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "memory.open_nodes", + "notion.notion_search", + "hubspot.hubspot_search_contacts", + "github.search_users", + "github.search_orgs", + "github.search_repositories", + "github.search_pull_requests", + "slack.slack_search_messages", + "zendesk.zendesk_search" + ], + "hit1": "memory.search_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 624, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "kind": "verb_product", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "ranking": [ + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_diff", + "git.git_reset", + "git.git_show", + "git.git_log", + "git.git_status", + "git.git_commit", + "git.git_add", + "github.list_packages" + ], + "hit1": "git.git_diff_unstaged", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 615, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "kind": "identifier", + "gold": [ + "git.git_add" + ], + "alts": [], + "ranking": [ + "git.git_add", + "git.git_reset", + "git.git_commit", + "git.git_diff_staged", + "github.update_gist", + "github.create_gist", + "salesforce.salesforce_list_opportunities", + "github.push_files", + "github.create_or_update_file", + "git.git_diff_unstaged" + ], + "hit1": "git.git_add", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 198, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 773, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "kind": "verb_product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "ranking": [ + "shopify.shopify_create_product", + "shopify.shopify_get_product", + "shopify.shopify_update_product", + "shopify.shopify_create_order", + "stripe.stripe_create_product", + "shopify.shopify_list_products", + "shopify.shopify_get_order", + "stripe.stripe_create_price", + "stripe.stripe_create_invoice", + "stripe.stripe_create_customer" + ], + "hit1": "shopify.shopify_create_product", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 700, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "kind": "verb_product", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "ranking": [ + "stripe.stripe_cancel_subscription", + "stripe.stripe_list_subscriptions", + "stripe.stripe_create_invoice", + "stripe.stripe_get_customer", + "stripe.stripe_create_product", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_customer", + "stripe.stripe_create_refund", + "stripe.stripe_create_price", + "stripe.stripe_create_payment_intent" + ], + "hit1": "stripe.stripe_cancel_subscription", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "kind": "resource", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_refund", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent", + "pagerduty.pagerduty_create_incident", + "stripe.stripe_create_invoice", + "jira.create_attachment", + "notion.create_comment", + "hubspot.hubspot_create_deal", + "slack.create_pin", + "jira.create_sprint" + ], + "hit1": "stripe.stripe_create_refund", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 866, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "notion.notion_get_page", + "playwright.browser_snapshot", + "playwright.browser_take_screenshot", + "confluence.confluence_get_page" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 690, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "kind": "verb_product", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "ranking": [ + "datadog.datadog_create_monitor", + "datadog.datadog_get_monitor", + "datadog.datadog_list_monitors", + "clickup.clickup_create_task", + "hubspot.hubspot_create_deal", + "cloudflare.cf_create_dns", + "notion.create_data_source", + "datadog.datadog_query_metrics", + "notion.notion_create_database", + "datadog.datadog_list_logs" + ], + "hit1": "datadog.datadog_create_monitor", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 441, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 933, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "kind": "paraphrase", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "ranking": [ + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "sentry.sentry_list_issues", + "sentry.sentry_list_events", + "pagerduty.pagerduty_resolve_incident", + "jira.jira_assign_issue", + "bitbucket.bitbucket_create_issue", + "sentry.sentry_list_projects", + "jira.jira_get_issue", + "linear.linear_get_issue" + ], + "hit1": "sentry.sentry_get_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 274, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_apply", + "kubernetes.k8s_delete", + "kubernetes.k8s_get_pod", + "vercel.vercel_get_deployment", + "kubernetes.k8s_list_services", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_get_events" + ], + "hit1": "kubernetes.k8s_scale_deployment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 241, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_apply", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_list_services", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_get_events", + "kubernetes.k8s_delete", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_pod_logs", + "terraform.tf_apply" + ], + "hit1": "kubernetes.k8s_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 222, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "ranking": [ + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_run", + "docker.docker_ps", + "docker.docker_exec", + "docker.docker_build", + "docker.docker_images", + "hubspot.hubspot_list_contacts", + "kubernetes.k8s_pod_logs" + ], + "hit1": "docker.docker_stop", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 639, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "kind": "verb_product", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "ranking": [ + "docker.docker_build", + "docker.docker_images", + "docker.docker_run", + "docker.docker_stop", + "docker.docker_logs", + "docker.docker_ps", + "docker.docker_exec", + "figma.figma_get_images", + "docker.docker_rm", + "filesystem.read_media_file" + ], + "hit1": "docker.docker_build", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 635, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "kind": "verb_product", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "ranking": [ + "playwright.browser_click", + "playwright.browser_type", + "puppeteer.puppeteer_click", + "playwright.browser_press_key", + "playwright.browser_navigate", + "playwright.browser_tab_list", + "playwright.browser_take_screenshot", + "playwright.browser_tab_close", + "playwright.browser_tab_new", + "playwright.browser_snapshot" + ], + "hit1": "playwright.browser_click", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 318, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 566, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "kind": "paraphrase", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "ranking": [ + "playwright.browser_type", + "playwright.browser_navigate", + "playwright.browser_wait_for", + "playwright.browser_press_key", + "playwright.browser_tab_new", + "puppeteer.puppeteer_fill", + "playwright.browser_fill_form", + "playwright.browser_tab_list", + "playwright.browser_click", + "playwright.browser_tab_close" + ], + "hit1": "playwright.browser_type", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8955796649110442, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 318, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 584, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_list_projects", + "gitlab.gitlab_list_issues", + "gitlab.gitlab_list_merge_requests", + "vercel.vercel_list_projects", + "jira.jira_list_projects", + "gitlab.list_snippets", + "github.projects_list", + "gitlab.list_jobs", + "gitlab.list_pipelines", + "gitlab.gitlab_create_issue" + ], + "hit1": "gitlab.gitlab_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 288, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 681, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "kind": "verb_product", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "ranking": [ + "bitbucket.bitbucket_create_pr", + "bitbucket.bitbucket_list_prs", + "bitbucket.bitbucket_create_issue", + "github.create_pull_request", + "github.pull_request_review_write", + "github.pull_request_read", + "github.list_pull_requests", + "github.update_pull_request_branch", + "github.search_pull_requests", + "github.merge_pull_request" + ], + "hit1": "bitbucket.bitbucket_create_pr", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 606, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1037, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "kind": "verb_product", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "ranking": [ + "gmail.gmail_create_draft", + "stripe.stripe_create_customer", + "hubspot.hubspot_create_contact", + "shopify.shopify_create_order", + "salesforce.salesforce_create_lead", + "gmail.gmail_trash_email", + "gmail.gmail_get_email", + "github.create_pull_request", + "clickup.clickup_create_task", + "stripe.stripe_create_invoice" + ], + "hit1": "gmail.gmail_create_draft", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 431, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 830, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "kind": "verb_product", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_share_file", + "google_drive.gdrive_update_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_list_folder", + "google_drive.gdrive_get_file", + "google_drive.gdrive_search", + "slack.get_file", + "filesystem.move_file", + "github.create_or_update_file", + "slack.delete_file" + ], + "hit1": "google_drive.gdrive_share_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 306, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 785, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "kind": "verb_product", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_update_event", + "kubernetes.k8s_get_events", + "sentry.sentry_list_events", + "pagerduty.pagerduty_list_incidents", + "gmail.gmail_list_labels", + "zendesk.zendesk_list_tickets" + ], + "hit1": "google_calendar.gcal_list_events", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 716, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "slack.delete_canvase", + "google_calendar.gcal_create_event", + "linear.delete_cycle", + "google_calendar.gcal_update_event", + "google_calendar.gcal_list_events", + "linear.delete_label", + "linear.delete_attachment", + "notion.delete_comment", + "jira.delete_sprint" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 270, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_projects", + "vercel.vercel_list_deployments", + "vercel.vercel_create_project", + "jira.jira_list_projects", + "linear.linear_list_projects", + "sentry.sentry_list_projects", + "gitlab.gitlab_list_projects", + "asana.asana_list_tasks", + "supabase.supabase_list_tables", + "circleci.circleci_list_pipelines" + ], + "hit1": "vercel.vercel_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 175, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 255, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 519, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "ranking": [ + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update", + "supabase.supabase_list_tables", + "supabase.supabase_invoke_fn", + "mongodb.mongo_list_collections", + "sqlite.sqlite_list_tables", + "postgres.list_tables", + "notion.notion_query_database", + "datadog.datadog_list_logs" + ], + "hit1": "supabase.supabase_select", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 672, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "kind": "verb_product", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "figma.figma_get_comments", + "linear.get_comment", + "notion.get_comment", + "figma.figma_get_file", + "linear.update_comment", + "notion.update_comment", + "linear.linear_create_comment", + "linear.create_comment", + "linear.delete_comment" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 201, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 308, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 653, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "kind": "verb_product", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "ranking": [ + "okta.okta_list_users", + "okta.okta_get_user", + "notion.notion_list_users", + "okta.okta_create_user", + "okta.okta_deactivate_user", + "notion.list_users_tokens", + "slack.slack_get_users", + "github.list_gists", + "linear.list_users", + "slack.list_usergroups" + ], + "hit1": "okta.okta_list_users", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 271, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 622, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query_range", + "prometheus.prom_query", + "prometheus.prom_list_metrics", + "datadog.datadog_query_metrics", + "shopify.shopify_list_products", + "memory.search_nodes", + "datadog.datadog_list_logs", + "datadog.datadog_list_monitors", + "confluence.confluence_search", + "sentry.sentry_list_issues" + ], + "hit1": "prometheus.prom_query_range", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 393, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 761, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "kind": "verb_product", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "ranking": [ + "grafana.grafana_create_annotation", + "grafana.grafana_get_dashboard", + "github.create_artifact", + "notion.create_comment", + "asana.asana_create_task", + "linear.create_attachment", + "notion.create_data_source", + "gitlab.create_label", + "jira.create_attachment", + "slack.create_canvase" + ], + "hit1": "grafana.grafana_create_annotation", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 461, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 960, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "kind": "identifier", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "ranking": [ + "terraform.tf_apply", + "terraform.tf_plan", + "terraform.tf_state_list", + "kubernetes.k8s_apply", + "playwright.browser_fill_form", + "filesystem.search_files", + "asana.asana_get_task", + "asana.asana_update_task", + "gmail.gmail_create_draft", + "asana.asana_create_project" + ], + "hit1": "terraform.tf_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 287, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 648, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "kind": "verb_product", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "ranking": [ + "asana.asana_list_tasks", + "asana.asana_get_task", + "asana.asana_create_task", + "asana.asana_update_task", + "clickup.clickup_list_tasks", + "okta.okta_list_users", + "aws_s3.s3_list_buckets", + "clickup.clickup_create_task", + "asana.asana_create_project", + "jira.jira_list_projects" + ], + "hit1": "asana.asana_list_tasks", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 674, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "ranking": [ + "trello.trello_add_comment", + "trello.trello_list_cards", + "trello.trello_create_card", + "trello.trello_create_board", + "figma.figma_post_comment", + "linear.get_comment", + "notion.get_comment", + "figma.figma_get_comments", + "linear.update_comment", + "notion.update_comment" + ], + "hit1": "trello.trello_add_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_update_task", + "clickup.clickup_get_task", + "clickup.clickup_list_tasks", + "clickup.clickup_create_task", + "asana.asana_update_task", + "linear.update_comment", + "slack.update_canvase", + "slack.update_bookmark", + "gitlab.update_snippet", + "jira.update_sprint" + ], + "hit1": "clickup.clickup_update_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 747, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "kind": "verb_product", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "ranking": [ + "confluence.confluence_search", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "confluence.confluence_create_page", + "hubspot.hubspot_search_contacts", + "github.search_users", + "grafana.grafana_search_dashboards", + "github.search_orgs", + "github.search_commits", + "github.search_code" + ], + "hit1": "confluence.confluence_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 369, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 651, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_deal", + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_list_contacts", + "stripe.stripe_create_invoice", + "stripe.stripe_create_customer", + "slack.slack_create_channel", + "slack.create_pin", + "linear.linear_create_project", + "slack.create_reminder" + ], + "hit1": "hubspot.hubspot_create_deal", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 776, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_search", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_create_ticket", + "google_calendar.gcal_list_calendars", + "vercel.vercel_list_projects", + "google_calendar.gcal_list_events", + "jira.jira_list_projects", + "jira.list_sprints" + ], + "hit1": "zendesk.zendesk_list_tickets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 644, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "kind": "verb_product", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "ranking": [ + "discord.discord_list_channels", + "discord.discord_get_channel", + "discord.discord_send_message", + "discord.discord_list_members", + "slack.slack_list_channels", + "discord.discord_add_reaction", + "slack.slack_get_channel_history", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_create_channel" + ], + "hit1": "discord.discord_list_channels", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 637, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "redis.redis_set", + "redis.redis_delete", + "redis.redis_keys", + "slack.get_reminder", + "slack.update_reminder", + "playwright.browser_press_key", + "aws_s3.s3_put_object", + "jira.jira_get_issue", + "slack.update_pin" + ], + "hit1": "redis.redis_get", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 284, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 611, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert", + "mongodb.mongo_update", + "mongodb.mongo_find", + "mongodb.mongo_list_collections", + "mongodb.mongo_delete", + "supabase.supabase_insert", + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "github.update_environment", + "linear.update_attachment" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "kind": "verb_product", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "ranking": [ + "mysql.mysql_query", + "sqlite.sqlite_query", + "postgres.query", + "salesforce.salesforce_query", + "postgres.explain_query", + "mysql.mysql_list_tables", + "prometheus.prom_query", + "github.search_users", + "datadog.datadog_list_monitors", + "notion.notion_query_database" + ], + "hit1": "mysql.mysql_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 591, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "elasticsearch.es_search", + "mongodb.mongo_find", + "mongodb.mongo_insert", + "google_drive.gdrive_search", + "confluence.confluence_search", + "mongodb.mongo_update", + "memory.search_nodes", + "hubspot.hubspot_search_contacts" + ], + "hit1": "elasticsearch.es_index_doc", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 392, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_put_object", + "aws_s3.s3_list_objects", + "salesforce.salesforce_get_record", + "gitlab.get_snippet", + "slack.get_file", + "asana.asana_get_task", + "slack.get_canvase", + "okta.okta_get_user" + ], + "hit1": "aws_s3.s3_get_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 670, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones", + "cloudflare.cf_list_dns", + "cloudflare.cf_purge_cache", + "cloudflare.cf_create_dns", + "circleci.circleci_list_pipelines", + "docker.docker_images", + "linear.linear_list_teams", + "hubspot.hubspot_list_contacts", + "gitlab.list_pipelines", + "kubernetes.k8s_list_services" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 591, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_create_customer", + "stripe.stripe_list_invoices", + "stripe.stripe_create_price", + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_product", + "stripe.stripe_create_refund", + "stripe.stripe_get_customer", + "slack.create_file", + "stripe.stripe_get_payment_intent" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "kind": "verb_product", + "gold": [ + "github.actions_list" + ], + "alts": [], + "ranking": [ + "github.actions_list", + "github.actions_get", + "github.actions_run_trigger", + "github.actions_get_job_logs", + "github.list_notifications", + "github.list_gists", + "github.list_hooks", + "github.list_artifacts", + "github.list_check_runs", + "github.list_discussions" + ], + "hit1": "github.actions_list", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 452, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 803, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "kind": "verb_product", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "ranking": [ + "github.actions_get_job_logs", + "github.actions_get", + "github.actions_list", + "github.actions_run_trigger", + "github.get_check_run", + "github.get_notification_details", + "github.get_hook", + "github.get_runner", + "github.search_orgs", + "github.get_milestone" + ], + "hit1": "github.actions_get_job_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 442, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "kind": "verb_product", + "gold": [ + "github.create_repository" + ], + "alts": [], + "ranking": [ + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.create_or_update_file", + "github.star_repository", + "github.create_gist", + "github.unstar_repository", + "github.delete_repository", + "github.fork_repository", + "github.list_releases" + ], + "hit1": "github.create_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 545, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 915, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "kind": "verb_product", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "ranking": [ + "git.git_checkout", + "git.git_branch", + "github.get_check_run", + "github.list_branches", + "git.git_create_branch", + "github.create_branch", + "github.update_pull_request_branch", + "github.get_gist", + "git.git_diff", + "git.git_show" + ], + "hit1": "git.git_checkout", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 710, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "kind": "identifier", + "gold": [ + "git.git_log" + ], + "alts": [], + "ranking": [ + "git.git_log", + "git.git_show", + "git.git_status", + "git.git_diff_unstaged", + "git.git_diff", + "github.actions_get_job_logs", + "git.git_diff_staged", + "git.git_branch", + "git.git_commit", + "git.git_create_branch" + ], + "hit1": "git.git_log", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 651, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "kind": "verb_product", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "ranking": [ + "postgres.list_schemas", + "postgres.list_tables", + "mysql.mysql_list_tables", + "postgres.describe_table", + "docker.docker_images", + "aws_s3.s3_list_buckets", + "supabase.supabase_list_tables", + "terraform.tf_state_list", + "sqlite.sqlite_list_tables", + "postgres.explain_query" + ], + "hit1": "postgres.list_schemas", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 249, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 469, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "kind": "verb_product", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "ranking": [ + "memory.open_nodes", + "memory.search_nodes", + "memory.create_entities", + "memory.read_graph", + "memory.create_relations", + "memory.add_observations", + "memory.delete_relations", + "memory.delete_entities", + "figma.figma_get_file_nodes", + "notion.get_data_source" + ], + "hit1": "memory.open_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 551, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "kind": "paraphrase", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "ranking": [ + "filesystem.get_file_info", + "filesystem.read_file", + "google_drive.gdrive_get_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "filesystem.list_directory", + "filesystem.directory_tree", + "filesystem.list_directory_with_sizes", + "figma.figma_get_file_nodes", + "github.get_file_contents" + ], + "hit1": "filesystem.get_file_info", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 685, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "kind": "paraphrase", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "ranking": [ + "filesystem.write_file", + "filesystem.read_text_file", + "filesystem.read_file", + "filesystem.create_directory", + "filesystem.edit_file", + "slack.create_file", + "filesystem.read_media_file", + "slack.update_file", + "google_drive.gdrive_create_file", + "slack.get_file" + ], + "hit1": "filesystem.write_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "kind": "verb_product", + "gold": [ + "github.search_issues" + ], + "alts": [], + "ranking": [ + "github.search_issues", + "github.search_users", + "github.search_pull_requests", + "github.search_orgs", + "github.search_commits", + "github.search_repositories", + "github.search_code", + "github.list_issues", + "jira.jira_search_issues", + "linear.linear_search_issues" + ], + "hit1": "github.search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 209, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "kind": "paraphrase", + "gold": [ + "github.get_me" + ], + "alts": [], + "ranking": [ + "github.get_teams", + "github.get_me", + "github.list_notifications", + "github.get_latest_release", + "github.projects_get", + "github.search_users", + "github.get_runner", + "github.get_notification_details", + "github.list_gists", + "github.get_label" + ], + "hit1": "github.get_teams", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 605, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "cost": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.1, + "p95_calls": 1.0, + "success_within_1": 0.9733333333333334, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 253.88, + "p95_tokens": 328.65, + "total_tokens": 38082, + "tokens_per_successful_query": 253.88, + "total_tokens_successful": 38082 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.0466666666666666, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 388.06, + "p95_tokens": 489.04999999999984, + "total_tokens": 58209, + "tokens_per_successful_query": 388.06, + "total_tokens_successful": 58209 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.02, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 745.4866666666667, + "p95_tokens": 970.1499999999997, + "total_tokens": 111823, + "tokens_per_successful_query": 745.4866666666667, + "total_tokens_successful": 111823 + } + } +} diff --git a/experiments/mcp-tool-search/results/hybrid_rrf_weighted_embed.csv b/experiments/mcp-tool-search/results/hybrid_rrf_weighted_embed.csv new file mode 100644 index 00000000..e8faa294 --- /dev/null +++ b/experiments/mcp-tool-search/results/hybrid_rrf_weighted_embed.csv @@ -0,0 +1,151 @@ +id,query,kind,hit1,recall@1,recall@3,recall@5,mrr@10,ndcg@10,hit_rank,k3_calls,k3_tokens,k3_success1,k5_calls,k5_tokens,k5_success1,k10_calls,k10_tokens,k10_success1,ranking +q001,Create a GitHub issue,verb_product,github.create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,394,1.0,1,598,1.0,1,1037,1.0,github.create_issue github.issue_write github.add_issue_comment github.issue_read github.update_issue_comment github.assign_copilot_to_issue github.create_repository gitlab.gitlab_create_issue github.create_gist github.search_issues +q002,Create a Jira ticket,verb_product,jira.jira_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,345,1.0,1,474,1.0,1,884,1.0,jira.jira_create_issue jira.jira_create_sprint jira.create_sprint jira.jira_list_projects jira.create_board jira.jira_assign_issue jira.create_version jira.create_webhook jira.create_filter jira.get_sprint +q003,Send a Slack message,verb_product,slack.slack_post_message,1.0,1.0,1.0,1.0000,1.0000,1,1,252,1.0,1,424,1.0,1,729,1.0,slack.slack_post_message slack.slack_add_reaction slack.slack_create_channel slack.slack_upload_file slack.slack_reply_to_thread slack.slack_leave_channel slack.get_reminder slack.slack_get_thread_replies slack.slack_join_channel slack.slack_search_messages +q004,List open pull requests,resource,github.list_pull_requests,1.0,1.0,1.0,1.0000,1.0000,1,1,311,1.0,1,477,1.0,1,1013,1.0,github.list_pull_requests github.search_pull_requests github.create_pull_request github.pull_request_read bitbucket.bitbucket_list_prs gitlab.gitlab_list_merge_requests github.pull_request_review_write github.update_pull_request github.list_hooks github.merge_pull_request +q005,Open a pull request on GitHub,verb_product,github.create_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,368,1.0,1,564,1.0,1,1144,1.0,github.create_pull_request github.merge_pull_request github.pull_request_read github.pull_request_review_write github.search_pull_requests github.update_pull_request github.list_pull_requests github.update_pull_request_branch github.request_copilot_review github.add_comment_to_pending_review +q006,Run a SQL query against postgres,verb_product,postgres.query,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,272,1.0,1,596,1.0,postgres.query postgres.explain_query mysql.mysql_query postgres.list_schemas sqlite.sqlite_query salesforce.salesforce_query postgres.describe_table postgres.list_tables prometheus.prom_query prometheus.prom_query_range +q007,Read a local file from disk,paraphrase,filesystem.read_file,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,370,1.0,1,684,1.0,filesystem.read_file filesystem.read_text_file filesystem.read_media_file filesystem.get_file_info filesystem.edit_file filesystem.write_file figma.figma_get_file figma.figma_get_file_nodes slack.get_file filesystem.move_file +q008,What files are in this folder,paraphrase,google_drive.gdrive_list_folder,0.0,1.0,1.0,0.5000,0.6765,2,1,191,1.0,1,316,1.0,1,615,1.0,google_drive.gdrive_list_folder filesystem.list_directory filesystem.list_directory_with_sizes filesystem.search_files filesystem.read_file filesystem.directory_tree filesystem.move_file filesystem.get_file_info filesystem.read_media_file filesystem.write_file +q009,Remember this fact in the knowledge graph,paraphrase,memory.read_graph,0.0,1.0,1.0,0.3333,0.5438,3,1,164,1.0,1,261,1.0,1,585,1.0,memory.read_graph memory.create_relations memory.add_observations memory.create_entities memory.delete_relations memory.open_nodes memory.delete_entities memory.search_nodes memory.delete_observations sequential_thinking.sequentialthinking +q010,Search the web with Brave,verb_product,brave_search.brave_web_search,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,349,1.0,1,673,1.0,brave_search.brave_web_search brave_search.brave_local_search confluence.confluence_search memory.search_nodes github.search_orgs github.search_users jira.jira_search_issues grafana.grafana_search_dashboards github.search_issues hubspot.hubspot_search_contacts +q011,Fetch the contents of a URL,verb_product,fetch.fetch,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,372,1.0,1,675,1.0,fetch.fetch playwright.browser_navigate github.get_file_contents puppeteer.puppeteer_navigate filesystem.read_file playwright.browser_take_screenshot playwright.browser_tab_new filesystem.list_directory playwright.browser_click docker.docker_logs +q012,Show git status,identifier,git.git_status,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,312,1.0,1,600,1.0,git.git_status git.git_show git.git_diff git.git_log git.git_diff_unstaged git.git_branch git.git_commit git.git_diff_staged git.git_checkout git.git_add +q013,Commit staged changes,paraphrase,git.git_commit,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,327,1.0,1,706,1.0,git.git_commit git.git_reset git.git_diff_staged git.git_show github.get_commit github.update_milestone git.git_log filesystem.move_file github.push_files slack.update_reminder +q014,Create a Linear issue,verb_product,linear.linear_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,322,1.0,1,463,1.0,1,875,1.0,linear.linear_create_issue linear.linear_get_issue linear.linear_update_issue linear.linear_assign_issue linear.linear_create_comment linear.linear_search_issues linear.linear_create_project linear.linear_list_issues linear.create_label linear.create_cycle +q015,Make a task in my project tracker Linear,paraphrase,linear.linear_create_project,0.0,1.0,1.0,0.5000,0.6309,2,1,284,1.0,1,483,1.0,1,769,1.0,linear.linear_create_project linear.linear_create_issue linear.linear_list_projects linear.linear_update_issue linear.linear_list_issues linear.linear_create_comment linear.linear_get_issue linear.linear_list_teams linear.linear_assign_issue linear.linear_search_issues +q016,Post a comment on a GitHub issue,verb_product,github.update_issue_comment,0.0,1.0,1.0,0.5000,0.6309,2,1,370,1.0,1,573,1.0,1,1087,1.0,github.update_issue_comment github.add_issue_comment github.issue_write github.issue_read github.assign_copilot_to_issue github.create_issue github.add_comment_to_pending_review jira.jira_add_comment github.get_discussion_comments linear.linear_create_comment +q017,Merge the pull request,ambiguous,github.merge_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,283,1.0,1,528,1.0,1,1093,1.0,github.merge_pull_request gitlab.gitlab_merge_merge_request gitlab.gitlab_get_merge_request github.update_pull_request_branch github.create_pull_request github.pull_request_read github.update_pull_request gitlab.gitlab_create_merge_request gitlab.gitlab_list_merge_requests github.pull_request_review_write +q018,Trigger a GitHub Actions workflow,verb_product,github.actions_run_trigger,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,458,1.0,1,805,1.0,github.actions_run_trigger github.actions_get github.actions_list github.actions_get_job_logs circleci.circleci_trigger_pipeline github.update_gist github.search_commits github.issue_read github.search_pull_requests git.git_checkout +q019,Get Dependabot alerts,resource,github.list_dependabot_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,287,1.0,1,422,1.0,1,708,1.0,github.list_dependabot_alerts github.get_dependabot_alert github.list_code_scanning_alerts github.list_secret_scanning_alerts slack.get_reminder github.get_milestone github.get_hook gitlab.get_release sentry.sentry_get_issue gitlab.get_snippet +q020,List Sentry issues,verb_product,sentry.sentry_list_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,396,1.0,1,812,1.0,sentry.sentry_list_issues sentry.sentry_list_projects sentry.sentry_list_events sentry.sentry_get_issue sentry.sentry_update_issue linear.linear_list_issues github.list_issues gitlab.gitlab_list_issues github.list_artifacts gitlab.list_pipelines +q021,Create a Stripe customer,verb_product,stripe.stripe_create_invoice,0.0,1.0,1.0,0.5000,0.6309,2,1,186,1.0,1,322,1.0,1,669,1.0,stripe.stripe_create_invoice stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_list_invoices stripe.stripe_create_payment_intent stripe.stripe_create_product stripe.stripe_list_subscriptions stripe.stripe_create_price stripe.stripe_create_refund stripe.stripe_list_customers +q022,Refund a Stripe payment,paraphrase,stripe.stripe_create_refund,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,332,1.0,1,644,1.0,stripe.stripe_create_refund stripe.stripe_get_payment_intent stripe.stripe_create_payment_intent stripe.stripe_get_customer stripe.stripe_create_invoice stripe.stripe_create_customer stripe.stripe_cancel_subscription stripe.stripe_list_invoices stripe.stripe_list_customers stripe.stripe_create_price +q023,Send an email with Gmail,verb_product,gmail.gmail_send_email,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,399,1.0,1,743,1.0,gmail.gmail_send_email gmail.gmail_get_email gmail.gmail_trash_email gmail.gmail_modify_email gmail.gmail_create_draft gmail.gmail_search_emails gmail.gmail_list_labels twilio.twilio_send_sms discord.discord_send_message google_drive.gdrive_share_file +q024,Search my Gmail inbox,verb_product,gmail.gmail_search_emails,1.0,1.0,1.0,1.0000,1.0000,1,1,178,1.0,1,315,1.0,1,661,1.0,gmail.gmail_search_emails gmail.gmail_list_labels gmail.gmail_get_email gmail.gmail_create_draft gmail.gmail_trash_email gmail.gmail_send_email gmail.gmail_modify_email github.search_users hubspot.hubspot_search_contacts github.search_orgs +q025,Upload a file to Google Drive,paraphrase,google_drive.gdrive_update_file,0.0,1.0,1.0,0.3333,0.5000,3,1,307,1.0,1,418,1.0,1,810,1.0,google_drive.gdrive_update_file google_drive.gdrive_share_file google_drive.gdrive_create_file google_drive.gdrive_get_file google_drive.gdrive_search google_drive.gdrive_list_folder slack.slack_upload_file filesystem.write_file filesystem.edit_file gitlab.gitlab_get_file +q026,Create a calendar event,resource,google_calendar.gcal_create_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,427,1.0,1,945,1.0,google_calendar.gcal_create_event google_calendar.gcal_update_event google_calendar.gcal_delete_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars github.create_milestone slack.create_reminder gitlab.create_milestone linear.create_cycle slack.create_canvase +q027,List Kubernetes pods,verb_product,kubernetes.k8s_list_pods,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,314,1.0,1,706,1.0,kubernetes.k8s_list_pods kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_get_pod kubernetes.k8s_pod_logs kubernetes.k8s_apply kubernetes.k8s_scale_deployment kubernetes.k8s_delete aws_s3.s3_list_buckets +q028,Get logs from a k8s pod,verb_product,kubernetes.k8s_pod_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,262,1.0,1,455,1.0,1,741,1.0,kubernetes.k8s_pod_logs kubernetes.k8s_get_pod docker.docker_logs github.actions_get_job_logs datadog.datadog_list_logs git.git_log kubernetes.k8s_get_events slack.slack_get_channel_history memory.delete_observations redis.redis_get +q029,Run a docker container,verb_product,docker.docker_run,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,345,1.0,1,644,1.0,docker.docker_run docker.docker_stop docker.docker_exec docker.docker_logs docker.docker_rm docker.docker_ps docker.docker_images docker.docker_build github.create_check_run github.get_check_run +q030,Show docker container logs,verb_product,docker.docker_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,352,1.0,1,649,1.0,docker.docker_logs docker.docker_run docker.docker_stop docker.docker_images kubernetes.k8s_pod_logs docker.docker_exec docker.docker_ps docker.docker_rm git.git_log docker.docker_build +q031,Navigate the browser to a URL,ambiguous,playwright.browser_navigate,1.0,1.0,1.0,1.0000,0.9502,1,1,180,1.0,1,279,1.0,1,543,1.0,playwright.browser_navigate playwright.browser_tab_new puppeteer.puppeteer_navigate playwright.browser_click playwright.browser_tab_list playwright.browser_press_key playwright.browser_take_screenshot playwright.browser_evaluate playwright.browser_snapshot playwright.browser_tab_close +q032,Take a screenshot of the page,ambiguous,playwright.browser_take_screenshot,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,316,1.0,1,669,1.0,playwright.browser_take_screenshot puppeteer.puppeteer_screenshot playwright.browser_snapshot playwright.browser_evaluate notion.notion_get_page playwright.browser_click playwright.browser_press_key confluence.confluence_get_page confluence.confluence_create_page confluence.confluence_update_page +q033,Create a GitLab merge request,verb_product,gitlab.gitlab_merge_merge_request,0.0,1.0,1.0,0.5000,0.6309,2,1,297,1.0,1,475,1.0,1,951,1.0,gitlab.gitlab_merge_merge_request gitlab.gitlab_create_merge_request gitlab.gitlab_get_merge_request github.merge_pull_request gitlab.gitlab_list_merge_requests gitlab.gitlab_create_pipeline github.create_pull_request gitlab.gitlab_create_issue gitlab.gitlab_get_file gitlab.gitlab_get_issue +q034,Search Notion pages,verb_product,notion.notion_search,1.0,1.0,1.0,1.0000,1.0000,1,1,227,1.0,1,349,1.0,1,671,1.0,notion.notion_search notion.notion_get_page notion.notion_query_database notion.notion_update_page notion.notion_get_block notion.notion_create_database notion.notion_create_page notion.notion_list_users notion.notion_get_comments notion.get_users_token +q035,Create a Notion page,verb_product,notion.notion_get_page,0.0,1.0,1.0,0.5000,0.6309,2,1,230,1.0,1,385,1.0,1,737,1.0,notion.notion_get_page notion.notion_create_page notion.notion_update_page notion.notion_create_database notion.notion_search notion.notion_get_comments notion.notion_list_users notion.create_data_source notion.notion_get_block notion.create_users_token +q036,Query a Notion database,verb_product,notion.notion_query_database,1.0,1.0,1.0,1.0000,1.0000,1,1,261,1.0,1,401,1.0,1,669,1.0,notion.notion_query_database notion.notion_create_database notion.notion_search notion.notion_create_page notion.notion_get_page notion.notion_get_block notion.notion_list_users notion.notion_get_comments notion.notion_update_page notion.get_comment +q037,Create a Zendesk ticket,verb_product,zendesk.zendesk_create_ticket,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,368,1.0,1,891,1.0,zendesk.zendesk_create_ticket zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_list_tickets zendesk.zendesk_search jira.jira_create_issue jira.create_board jira.create_sprint github.create_milestone gitlab.create_milestone +q038,Create a HubSpot contact,verb_product,hubspot.hubspot_create_contact,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,925,1.0,hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_create_deal hubspot.hubspot_search_contacts hubspot.hubspot_list_contacts github.create_pull_request github.get_discussion github.pull_request_review_write github.create_gist slack.create_usergroup +q039,Create a Salesforce lead,verb_product,salesforce.salesforce_convert_lead,0.0,1.0,1.0,0.5000,0.6309,2,1,252,1.0,1,380,1.0,1,815,1.0,salesforce.salesforce_convert_lead salesforce.salesforce_create_lead salesforce.salesforce_create_record salesforce.salesforce_list_opportunities salesforce.salesforce_get_record salesforce.salesforce_query salesforce.salesforce_update_record github.create_hook github.create_runner linear.create_user +q040,Run a SOQL query,identifier,salesforce.salesforce_query,1.0,1.0,1.0,1.0000,1.0000,1,1,187,1.0,1,291,1.0,1,694,1.0,salesforce.salesforce_query postgres.query mysql.mysql_query sqlite.sqlite_query prometheus.prom_query postgres.explain_query prometheus.prom_query_range datadog.datadog_query_metrics notion.notion_query_database datadog.datadog_list_monitors +q041,List Shopify orders,verb_product,shopify.shopify_list_orders,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,320,1.0,1,647,1.0,shopify.shopify_list_orders shopify.shopify_list_products shopify.shopify_list_customers shopify.shopify_get_order shopify.shopify_get_product shopify.shopify_create_order shopify.shopify_create_product shopify.shopify_update_product clickup.clickup_list_tasks aws_s3.s3_list_buckets +q042,Create a PagerDuty incident,verb_product,pagerduty.pagerduty_create_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,403,1.0,1,874,1.0,pagerduty.pagerduty_create_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls slack.create_bookmark slack.create_reminder slack.create_canvase notion.create_comment +q043,Who is on call in PagerDuty,paraphrase,pagerduty.pagerduty_list_oncalls,1.0,1.0,1.0,1.0000,1.0000,1,1,227,1.0,1,401,1.0,1,709,1.0,pagerduty.pagerduty_list_oncalls pagerduty.pagerduty_get_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_create_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident twilio.twilio_make_call playwright.browser_press_key playwright.browser_click filesystem.list_allowed_directories +q044,Query Datadog metrics,verb_product,datadog.datadog_query_metrics,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,458,1.0,1,769,1.0,datadog.datadog_query_metrics datadog.datadog_list_monitors datadog.datadog_create_monitor datadog.datadog_list_logs datadog.datadog_get_monitor prometheus.prom_list_metrics memory.search_nodes prometheus.prom_query github.search_pull_requests prometheus.prom_query_range +q045,Search Datadog logs,verb_product,datadog.datadog_list_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,458,1.0,1,802,1.0,datadog.datadog_list_logs datadog.datadog_list_monitors datadog.datadog_get_monitor datadog.datadog_create_monitor datadog.datadog_query_metrics docker.docker_logs github.search_users github.search_issues github.actions_get_job_logs github.search_orgs +q046,List S3 buckets,verb_product,aws_s3.s3_list_buckets,1.0,1.0,1.0,1.0000,1.0000,1,1,185,1.0,1,318,1.0,1,690,1.0,aws_s3.s3_list_buckets aws_s3.s3_list_objects kubernetes.k8s_list_services gitlab.list_snippets clickup.clickup_list_tasks kubernetes.k8s_list_deployments jira.list_webhooks gitlab.list_pipelines github.list_environments github.list_milestones +q047,Upload an object to S3,paraphrase,aws_s3.s3_put_object,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,405,1.0,1,847,1.0,aws_s3.s3_put_object aws_s3.s3_get_object aws_s3.s3_delete_object slack.slack_upload_file aws_s3.s3_list_objects twilio.twilio_send_sms filesystem.write_file filesystem.edit_file github.assign_copilot_to_issue slack.slack_reply_to_thread +q048,Purge Cloudflare cache,verb_product,cloudflare.cf_purge_cache,1.0,1.0,1.0,1.0000,1.0000,1,1,231,1.0,1,341,1.0,1,618,1.0,cloudflare.cf_purge_cache cloudflare.cf_list_zones cloudflare.cf_create_dns cloudflare.cf_list_dns slack.delete_bookmark slack.delete_canvase memory.delete_observations redis.redis_delete aws_s3.s3_delete_object linear.delete_attachment +q049,Create a Cloudflare DNS record,verb_product,cloudflare.cf_create_dns,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,356,1.0,1,812,1.0,cloudflare.cf_create_dns cloudflare.cf_list_dns cloudflare.cf_purge_cache cloudflare.cf_list_zones salesforce.salesforce_create_record salesforce.salesforce_get_record salesforce.salesforce_update_record slack.create_file github.create_environment gitlab.create_pipeline +q050,List Vercel deployments,verb_product,vercel.vercel_list_deployments,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,291,1.0,1,650,1.0,vercel.vercel_list_deployments vercel.vercel_list_projects vercel.vercel_get_deployment vercel.vercel_create_project kubernetes.k8s_list_deployments kubernetes.k8s_list_services github.list_packages github.list_environments gitlab.list_releases github.list_artifacts +q051,Insert a row into Supabase,verb_product,supabase.supabase_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,355,1.0,1,701,1.0,supabase.supabase_insert supabase.supabase_select supabase.supabase_update supabase.supabase_invoke_fn supabase.supabase_list_tables mongodb.mongo_insert playwright.browser_type linear.update_cycle linear.update_attachment linear.update_user +q052,Export images from Figma,paraphrase,figma.figma_get_images,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,368,1.0,1,701,1.0,figma.figma_get_images figma.figma_get_file figma.figma_get_file_nodes figma.figma_get_comments figma.figma_post_comment docker.docker_images filesystem.read_media_file github.get_file_contents elasticsearch.es_get_doc linear.update_attachment +q053,Send an SMS with Twilio,verb_product,twilio.twilio_send_sms,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,409,1.0,1,811,1.0,twilio.twilio_send_sms twilio.twilio_list_messages twilio.twilio_make_call gmail.gmail_send_email discord.discord_send_message slack.slack_add_reaction aws_s3.s3_put_object playwright.browser_type filesystem.list_directory_with_sizes slack.slack_reply_to_thread +q054,Create an Okta user,verb_product,okta.okta_get_user,0.0,1.0,1.0,0.5000,0.6309,2,1,183,1.0,1,329,1.0,1,676,1.0,okta.okta_get_user okta.okta_create_user okta.okta_deactivate_user okta.okta_list_users linear.create_user asana.asana_create_task asana.asana_create_project linear.get_user linear.update_user slack.slack_get_user_profile +q055,Instant query Prometheus,verb_product,prometheus.prom_query,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,379,1.0,1,682,1.0,prometheus.prom_query prometheus.prom_query_range prometheus.prom_list_metrics datadog.datadog_query_metrics memory.search_nodes postgres.query salesforce.salesforce_query github.search_pull_requests sqlite.sqlite_query notion.notion_query_database +q056,Search Grafana dashboards,verb_product,grafana.grafana_search_dashboards,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,319,1.0,1,677,1.0,grafana.grafana_search_dashboards grafana.grafana_get_dashboard grafana.grafana_create_annotation github.search_orgs github.search_users github.search_issues brave_search.brave_local_search confluence.confluence_search brave_search.brave_web_search github.search_pull_requests +q057,Trigger a CircleCI pipeline,verb_product,circleci.circleci_trigger_pipeline,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,390,1.0,1,755,1.0,circleci.circleci_trigger_pipeline circleci.circleci_get_pipeline circleci.circleci_list_pipelines github.actions_run_trigger gitlab.gitlab_create_pipeline gitlab.update_pipeline hubspot.hubspot_create_deal gitlab.get_pipeline gitlab.create_pipeline gitlab.delete_pipeline +q058,Run terraform plan,identifier,terraform.tf_plan,1.0,1.0,1.0,1.0000,1.0000,1,1,172,1.0,1,334,1.0,1,758,1.0,terraform.tf_plan terraform.tf_apply terraform.tf_state_list github.update_check_run docker.docker_run github.get_check_run github.create_check_run github.actions_run_trigger docker.docker_exec github.actions_get +q059,Create an Asana task,verb_product,asana.asana_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,405,1.0,1,837,1.0,asana.asana_create_task asana.asana_get_task asana.asana_create_project asana.asana_update_task asana.asana_list_tasks clickup.clickup_create_task okta.okta_create_user github.create_issue clickup.clickup_get_task gitlab.create_snippet +q060,Create a Trello card,verb_product,trello.trello_create_card,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,411,1.0,1,819,1.0,trello.trello_create_card trello.trello_add_comment trello.trello_create_board trello.trello_list_cards linear.create_label linear.create_cycle stripe.stripe_create_invoice slack.create_reminder linear.create_attachment stripe.stripe_create_customer +q061,Create a ClickUp task,verb_product,clickup.clickup_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,364,1.0,1,787,1.0,clickup.clickup_create_task clickup.clickup_get_task clickup.clickup_list_tasks clickup.clickup_update_task puppeteer.puppeteer_click playwright.browser_click asana.asana_create_task gitlab.create_snippet asana.asana_get_task jira.create_sprint +q062,Create a Confluence page,verb_product,confluence.confluence_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,401,1.0,1,791,1.0,confluence.confluence_create_page confluence.confluence_get_page confluence.confluence_update_page confluence.confluence_search notion.notion_create_page notion.notion_get_page github.create_artifact playwright.browser_snapshot github.create_environment gitlab.create_wiki +q063,Send a Discord message,verb_product,discord.discord_send_message,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,343,1.0,1,718,1.0,discord.discord_send_message discord.discord_add_reaction discord.discord_get_channel slack.slack_post_message discord.discord_list_channels slack.slack_add_reaction gmail.gmail_trash_email twilio.twilio_send_sms discord.discord_list_members gmail.gmail_send_email +q064,Search Elasticsearch,verb_product,elasticsearch.es_search,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,345,1.0,1,680,1.0,elasticsearch.es_search confluence.confluence_search elasticsearch.es_get_doc elasticsearch.es_index_doc github.search_orgs github.search_users brave_search.brave_web_search github.search_issues hubspot.hubspot_search_contacts github.search_pull_requests +q065,Get a Redis key,verb_product,redis.redis_get,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,280,1.0,1,566,1.0,redis.redis_get redis.redis_set redis.redis_delete redis.redis_keys playwright.browser_press_key figma.figma_get_file jira.jira_get_issue aws_s3.s3_get_object github.get_milestone gitlab.get_pipeline +q066,Find MongoDB documents,verb_product,mongodb.mongo_find,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,381,1.0,1,729,1.0,mongodb.mongo_find mongodb.mongo_insert mongodb.mongo_list_collections mongodb.mongo_update mongodb.mongo_delete elasticsearch.es_get_doc filesystem.search_files confluence.confluence_search datadog.datadog_list_logs hubspot.hubspot_search_contacts +q067,List SQLite tables,verb_product,sqlite.sqlite_list_tables,1.0,1.0,1.0,1.0000,1.0000,1,1,151,1.0,1,240,1.0,1,590,1.0,sqlite.sqlite_list_tables mysql.mysql_list_tables postgres.list_tables supabase.supabase_list_tables sqlite.sqlite_describe_table postgres.list_schemas trello.trello_list_cards gitlab.list_wikis gitlab.list_pipelines gitlab.list_labels +q068,Describe a postgres table,verb_product,postgres.describe_table,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,274,1.0,1,590,1.0,postgres.describe_table sqlite.sqlite_describe_table postgres.explain_query postgres.list_tables postgres.list_schemas postgres.query supabase.supabase_select supabase.supabase_insert supabase.supabase_update mysql.mysql_list_tables +q069,github_create_issue,identifier,github.create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,378,1.0,1,573,1.0,1,1072,1.0,github.create_issue github.issue_write gitlab.gitlab_create_issue bitbucket.bitbucket_create_issue github.assign_copilot_to_issue github.add_issue_comment github.create_environment github.create_package github.update_issue_comment github.issue_read +q070,createIssue,identifier,linear.create_user,0.0,0.0,0.0,0.0000,0.0000,39,13,3608,0.0,8,3654,0.0,4,3582,0.0,linear.create_user linear.create_cycle linear.create_comment github.create_artifact jira.create_board linear.create_attachment jira.create_filter linear.create_label github.create_gist jira.create_attachment +q071,list-directory,identifier,filesystem.list_directory,1.0,1.0,1.0,1.0000,1.0000,1,1,171,1.0,1,301,1.0,1,623,1.0,filesystem.list_directory filesystem.list_directory_with_sizes filesystem.list_allowed_directories clickup.clickup_list_tasks github.list_environments filesystem.directory_tree aws_s3.s3_list_buckets trello.trello_list_cards github.list_runners github.list_packages +q072,sequential thinking next thought,paraphrase,sequential_thinking.sequentialthinking,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,344,1.0,1,650,1.0,sequential_thinking.sequentialthinking memory.create_relations memory.add_observations memory.search_nodes memory.create_entities memory.delete_relations notion.update_comment notion.notion_search memory.open_nodes notion.notion_append_block +q073,What time is it in Tokyo,paraphrase,time.get_current_time,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,391,1.0,1,757,1.0,time.get_current_time time.convert_time google_calendar.gcal_list_events playwright.browser_wait_for grafana.grafana_create_annotation jira.jira_add_worklog zendesk.zendesk_list_tickets filesystem.list_allowed_directories slack.slack_create_channel jira.jira_create_sprint +q074,Convert time between timezones,verb_product,time.convert_time,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,390,1.0,1,698,1.0,time.convert_time time.get_current_time playwright.browser_wait_for grafana.grafana_create_annotation google_calendar.gcal_list_events salesforce.salesforce_convert_lead git.git_diff datadog.datadog_query_metrics memory.create_relations cloudflare.cf_list_zones +q075,Star a GitHub repo,verb_product,github.star_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,425,1.0,1,801,1.0,github.star_repository github.unstar_repository github.get_label github.search_issues github.get_file_blame github.list_releases git.git_show github.create_gist github.list_code_scanning_alerts github.actions_list +q076,Fork a repository,resource,github.fork_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,386,1.0,1,875,1.0,github.fork_repository github.unstar_repository github.star_repository github.delete_repository github.create_repository github.get_repository_tree github.list_dependabot_alerts github.list_releases github.list_pull_requests github.create_or_update_file +q077,Get the latest GitHub release,verb_product,github.get_latest_release,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,344,1.0,1,728,1.0,github.get_latest_release github.get_release_by_tag gitlab.get_release github.get_gist github.get_notification_details github.get_code_scanning_alert github.get_secret_scanning_alert github.get_commit github.get_teams gitlab.update_release +q078,Create a gist,resource,github.create_gist,1.0,1.0,1.0,1.0000,1.0000,1,1,229,1.0,1,429,1.0,1,925,1.0,github.create_gist github.update_gist github.get_gist github.create_milestone github.create_hook gitlab.create_snippet github.create_artifact github.create_package gitlab.create_wiki github.create_runner +q079,Search GitHub code,verb_product,github.search_code,1.0,1.0,1.0,1.0000,1.0000,1,1,283,1.0,1,385,1.0,1,751,1.0,github.search_code github.get_code_scanning_alert github.list_code_scanning_alerts github.search_orgs github.search_users github.search_pull_requests github.search_commits github.search_repositories github.search_issues github.get_label +q080,List GitHub notifications,verb_product,github.list_notifications,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,368,1.0,1,704,1.0,github.list_notifications github.mark_all_notifications_read github.list_code_scanning_alerts github.list_secret_scanning_alerts github.actions_list github.list_gists github.list_releases github.list_starred_repositories github.list_pull_requests github.list_tags +q081,Assign Copilot to an issue,verb_product,github.assign_copilot_to_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,500,1.0,1,862,1.0,github.assign_copilot_to_issue linear.linear_assign_issue jira.jira_assign_issue github.request_copilot_review github.create_issue jira.jira_update_issue jira.jira_get_issue jira.jira_transition_issue sentry.sentry_get_issue github.add_issue_comment +q082,Get secret scanning alerts,resource,github.list_secret_scanning_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,295,1.0,1,470,1.0,1,776,1.0,github.list_secret_scanning_alerts github.get_secret_scanning_alert github.list_code_scanning_alerts github.get_code_scanning_alert github.list_dependabot_alerts github.get_dependabot_alert github.get_hook slack.get_reminder gitlab.get_snippet slack.get_pin +q083,Create or update a file on GitHub,verb_product,github.create_or_update_file,1.0,1.0,1.0,1.0000,0.7602,1,1,333,1.0,1,476,1.0,1,1078,1.0,github.create_or_update_file github.update_gist github.projects_write git.git_add github.create_gist slack.update_file github.get_file_contents github.issue_write github.create_pull_request github.update_pull_request +q084,List collaborators on a repo,resource,github.list_repository_collaborators,1.0,1.0,1.0,1.0000,1.0000,1,1,271,1.0,1,410,1.0,1,804,1.0,github.list_repository_collaborators github.list_commits github.list_labels github.list_releases github.actions_list github.list_tags github.list_discussions github.list_branches github.list_issues github.list_dependabot_alerts +q085,Reply in a Slack thread,verb_product,slack.slack_reply_to_thread,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,416,1.0,1,748,1.0,slack.slack_reply_to_thread slack.slack_get_thread_replies slack.slack_post_message slack.slack_search_messages slack.slack_add_reaction slack.slack_create_channel slack.slack_upload_file slack.get_reminder slack.update_reminder slack.slack_join_channel +q086,Add an emoji reaction on Slack,verb_product,slack.slack_add_reaction,1.0,1.0,1.0,1.0000,0.9072,1,1,271,1.0,1,436,1.0,1,785,1.0,slack.slack_add_reaction slack.slack_reply_to_thread slack.slack_post_message slack.slack_get_thread_replies discord.discord_add_reaction slack.slack_upload_file slack.slack_create_channel slack.get_reminder slack.update_reminder slack.slack_search_messages +q087,Search Slack messages,verb_product,slack.slack_search_messages,1.0,1.0,1.0,1.0000,1.0000,1,1,251,1.0,1,393,1.0,1,748,1.0,slack.slack_search_messages slack.slack_get_channel_history slack.slack_add_reaction slack.slack_post_message slack.slack_create_channel slack.slack_get_thread_replies slack.slack_reply_to_thread slack.slack_list_channels slack.get_reminder slack.slack_get_users +q088,Create a Slack channel,verb_product,slack.slack_join_channel,0.0,1.0,1.0,0.5000,0.6309,2,1,196,1.0,1,339,1.0,1,751,1.0,slack.slack_join_channel slack.slack_create_channel slack.slack_leave_channel slack.slack_post_message slack.slack_set_channel_topic slack.slack_add_reaction slack.slack_get_channel_history slack.slack_reply_to_thread slack.slack_upload_file slack.slack_get_thread_replies +q089,Transition a Jira issue,verb_product,jira.jira_transition_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,222,1.0,1,342,1.0,1,760,1.0,jira.jira_transition_issue jira.jira_assign_issue jira.jira_update_issue jira.jira_get_issue jira.jira_add_comment jira.jira_create_issue jira.jira_search_issues jira.jira_create_sprint jira.update_sprint jira.get_sprint +q090,Search Jira with JQL,identifier,jira.jira_search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,179,1.0,1,325,1.0,1,627,1.0,jira.jira_search_issues jira.jira_get_issue jira.jira_list_projects jira.get_filter jira.create_filter jira.jira_update_issue jira.get_board jira.jira_assign_issue jira.update_filter jira.get_sprint +q091,Add a worklog to Jira,verb_product,jira.jira_add_worklog,1.0,1.0,1.0,1.0000,1.0000,1,1,214,1.0,1,439,1.0,1,821,1.0,jira.jira_add_worklog jira.jira_add_comment jira.jira_list_projects jira.jira_create_issue jira.jira_create_sprint jira.jira_update_issue jira.jira_get_issue jira.create_board jira.create_sprint jira.jira_transition_issue +q092,List Linear teams,verb_product,linear.linear_list_teams,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,398,1.0,1,728,1.0,linear.linear_list_teams linear.linear_list_projects linear.linear_list_issues linear.linear_create_issue linear.linear_create_project linear.linear_get_issue linear.list_cycles linear.linear_search_issues linear.list_users linear.list_labels +q093,Comment on a Linear issue,verb_product,linear.linear_create_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,464,1.0,1,781,1.0,linear.linear_create_comment linear.linear_get_issue linear.linear_assign_issue linear.linear_update_issue linear.linear_create_issue linear.get_comment linear.update_comment linear.create_comment linear.linear_search_issues linear.delete_comment +q094,Move a local file,paraphrase,filesystem.move_file,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,352,1.0,1,715,1.0,filesystem.move_file filesystem.read_text_file filesystem.read_file filesystem.read_media_file filesystem.write_file slack.update_file slack.get_file slack.create_file figma.figma_get_file filesystem.edit_file +q095,Recursively search files by pattern,paraphrase,filesystem.search_files,1.0,1.0,1.0,1.0000,1.0000,1,1,192,1.0,1,305,1.0,1,617,1.0,filesystem.search_files filesystem.directory_tree redis.redis_keys filesystem.list_directory filesystem.list_directory_with_sizes google_drive.gdrive_search filesystem.get_file_info filesystem.move_file filesystem.read_text_file linear.linear_search_issues +q096,Edit a file in place with old and new string,paraphrase,filesystem.edit_file,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,413,1.0,1,779,1.0,filesystem.edit_file filesystem.write_file slack.update_file slack.create_file filesystem.move_file google_drive.gdrive_update_file slack.get_file slack.delete_file filesystem.read_text_file slack.slack_upload_file +q097,Read the whole knowledge graph,paraphrase,memory.read_graph,1.0,1.0,1.0,1.0000,1.0000,1,1,154,1.0,1,258,1.0,1,510,1.0,memory.read_graph memory.delete_relations memory.open_nodes memory.add_observations memory.create_entities memory.search_nodes memory.create_relations memory.delete_entities github.mark_all_notifications_read memory.delete_observations +q098,Search memory nodes,verb_product,memory.search_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,202,1.0,1,288,1.0,1,564,1.0,memory.search_nodes memory.open_nodes figma.figma_get_file_nodes memory.read_graph memory.delete_entities github.search_orgs memory.add_observations filesystem.search_files memory.create_relations memory.delete_relations +q099,Show unstaged git diff,verb_product,git.git_diff_unstaged,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,303,1.0,1,588,1.0,git.git_diff_unstaged git.git_diff git.git_diff_staged git.git_show git.git_status git.git_reset git.git_commit git.git_log git.git_add git.git_branch +q100,Stage files with git add,identifier,git.git_add,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,326,1.0,1,651,1.0,git.git_add git.git_commit git.git_show git.git_diff_staged git.git_diff git.git_reset git.git_branch github.push_files git.git_diff_unstaged git.git_log +q101,Create a Shopify product,verb_product,shopify.shopify_create_product,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,380,1.0,1,702,1.0,shopify.shopify_create_product shopify.shopify_get_product shopify.shopify_update_product shopify.shopify_create_order shopify.shopify_list_products shopify.shopify_get_order shopify.shopify_list_orders stripe.stripe_create_product shopify.shopify_list_customers stripe.stripe_create_price +q102,Cancel a Stripe subscription,verb_product,stripe.stripe_cancel_subscription,1.0,1.0,1.0,1.0000,1.0000,1,1,167,1.0,1,297,1.0,1,622,1.0,stripe.stripe_cancel_subscription stripe.stripe_list_invoices stripe.stripe_create_invoice stripe.stripe_list_subscriptions stripe.stripe_list_customers stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_create_refund stripe.stripe_get_payment_intent stripe.stripe_create_price +q103,Create a payment intent,resource,stripe.stripe_create_payment_intent,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,441,1.0,1,800,1.0,stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent stripe.stripe_create_refund github.create_milestone slack.create_reminder stripe.stripe_create_invoice stripe.stripe_create_price github.create_hook stripe.stripe_create_product stripe.stripe_create_customer +q104,Acknowledge a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,261,1.0,1,405,1.0,1,702,1.0,pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls sentry.sentry_list_events slack.get_bookmark slack.get_reminder notion.notion_get_page +q105,Create a Datadog monitor,verb_product,datadog.datadog_get_monitor,0.0,1.0,1.0,0.5000,0.6309,2,1,259,1.0,1,459,1.0,1,840,1.0,datadog.datadog_get_monitor datadog.datadog_create_monitor datadog.datadog_list_monitors datadog.datadog_list_logs datadog.datadog_query_metrics memory.create_entities slack.create_bookmark slack.create_reminder memory.create_relations github.create_gist +q106,Resolve a Sentry issue,paraphrase,sentry.sentry_get_issue,0.0,1.0,1.0,0.5000,0.6309,2,1,274,1.0,1,397,1.0,1,832,1.0,sentry.sentry_get_issue sentry.sentry_update_issue sentry.sentry_list_issues sentry.sentry_list_events sentry.sentry_list_projects pagerduty.pagerduty_resolve_incident jira.jira_assign_issue jira.jira_get_issue github.issue_write linear.linear_update_issue +q107,Scale a Kubernetes deployment,verb_product,kubernetes.k8s_scale_deployment,1.0,1.0,1.0,1.0000,1.0000,1,1,241,1.0,1,371,1.0,1,726,1.0,kubernetes.k8s_scale_deployment kubernetes.k8s_apply kubernetes.k8s_list_deployments kubernetes.k8s_get_pod kubernetes.k8s_list_services kubernetes.k8s_delete kubernetes.k8s_pod_logs kubernetes.k8s_list_pods kubernetes.k8s_get_events vercel.vercel_get_deployment +q108,Apply a Kubernetes manifest,verb_product,kubernetes.k8s_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,371,1.0,1,727,1.0,kubernetes.k8s_apply kubernetes.k8s_get_pod kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_scale_deployment kubernetes.k8s_delete kubernetes.k8s_list_pods kubernetes.k8s_get_events kubernetes.k8s_pod_logs terraform.tf_apply +q109,Stop a docker container,verb_product,docker.docker_stop,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,345,1.0,1,664,1.0,docker.docker_stop docker.docker_rm docker.docker_logs docker.docker_run docker.docker_exec docker.docker_images docker.docker_ps docker.docker_build kubernetes.k8s_pod_logs github.list_environments +q110,Build a docker image,verb_product,docker.docker_build,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,319,1.0,1,642,1.0,docker.docker_build docker.docker_run docker.docker_images docker.docker_stop docker.docker_logs docker.docker_ps docker.docker_exec docker.docker_rm filesystem.read_media_file github.create_environment +q111,Click an element in Playwright,verb_product,playwright.browser_click,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,341,1.0,1,566,1.0,playwright.browser_click playwright.browser_type puppeteer.puppeteer_click playwright.browser_navigate playwright.browser_take_screenshot playwright.browser_press_key playwright.browser_tab_list playwright.browser_tab_new playwright.browser_snapshot playwright.browser_fill_form +q112,Type text into a browser input,paraphrase,playwright.browser_type,1.0,1.0,1.0,1.0000,0.8869,1,1,213,1.0,1,330,1.0,1,586,1.0,playwright.browser_type playwright.browser_navigate playwright.browser_click playwright.browser_wait_for playwright.browser_tab_new playwright.browser_fill_form puppeteer.puppeteer_fill playwright.browser_press_key playwright.browser_tab_list playwright.browser_evaluate +q113,List GitLab projects,verb_product,gitlab.gitlab_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,300,1.0,1,635,1.0,gitlab.gitlab_list_projects gitlab.gitlab_list_issues gitlab.gitlab_list_merge_requests git.git_branch vercel.vercel_list_projects jira.jira_list_projects github.projects_list gitlab.gitlab_create_issue gitlab.gitlab_get_issue gitlab.gitlab_create_pipeline +q114,Create a Bitbucket pull request,verb_product,bitbucket.bitbucket_create_pr,1.0,1.0,1.0,1.0000,1.0000,1,1,313,1.0,1,553,1.0,1,1002,1.0,bitbucket.bitbucket_create_pr bitbucket.bitbucket_list_prs bitbucket.bitbucket_create_issue github.create_pull_request github.pull_request_read github.merge_pull_request bitbucket.bitbucket_list_repos github.pull_request_review_write github.update_pull_request_branch github.search_pull_requests +q115,Create a Gmail draft,verb_product,gmail.gmail_create_draft,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,375,1.0,1,855,1.0,gmail.gmail_create_draft gmail.gmail_list_labels gmail.gmail_get_email gmail.gmail_modify_email gmail.gmail_send_email gmail.gmail_trash_email gmail.gmail_search_emails github.create_pull_request github.create_artifact gitlab.create_snippet +q116,Share a Google Drive file,verb_product,google_drive.gdrive_share_file,1.0,1.0,1.0,1.0000,1.0000,1,1,306,1.0,1,417,1.0,1,741,1.0,google_drive.gdrive_share_file google_drive.gdrive_update_file google_drive.gdrive_create_file google_drive.gdrive_get_file google_drive.gdrive_search google_drive.gdrive_list_folder filesystem.write_file gitlab.gitlab_get_file figma.figma_get_file slack.get_file +q117,List Google Calendar events,verb_product,google_calendar.gcal_list_events,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,427,1.0,1,769,1.0,google_calendar.gcal_list_events google_calendar.gcal_list_calendars google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event kubernetes.k8s_get_events sentry.sentry_list_events gitlab.list_milestones clickup.clickup_list_tasks gitlab.list_wikis +q118,Delete a calendar event,resource,google_calendar.gcal_delete_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,444,1.0,1,697,1.0,google_calendar.gcal_delete_event google_calendar.gcal_create_event google_calendar.gcal_update_event github.delete_milestone google_calendar.gcal_list_events gitlab.delete_milestone redis.redis_delete google_calendar.gcal_list_calendars slack.delete_reminder slack.delete_canvase +q119,List Vercel projects,verb_product,vercel.vercel_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,175,1.0,1,269,1.0,1,539,1.0,vercel.vercel_list_projects vercel.vercel_list_deployments vercel.vercel_create_project vercel.vercel_get_deployment jira.jira_list_projects sentry.sentry_list_projects linear.linear_list_projects gitlab.gitlab_list_projects github.projects_list kubernetes.k8s_list_deployments +q120,Select rows from Supabase,verb_product,supabase.supabase_select,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,354,1.0,1,752,1.0,supabase.supabase_select supabase.supabase_insert supabase.supabase_update supabase.supabase_invoke_fn supabase.supabase_list_tables datadog.datadog_list_logs figma.figma_get_file_nodes datadog.datadog_query_metrics elasticsearch.es_get_doc sqlite.sqlite_query +q121,Post a Figma comment,verb_product,figma.figma_post_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,350,1.0,1,729,1.0,figma.figma_post_comment figma.figma_get_file figma.figma_get_comments linear.get_comment linear.create_comment linear.update_comment notion.get_comment figma.figma_get_images notion.create_comment figma.figma_get_file_nodes +q122,List Okta users,verb_product,okta.okta_list_users,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,271,1.0,1,606,1.0,okta.okta_list_users okta.okta_get_user okta.okta_create_user okta.okta_deactivate_user notion.notion_list_users notion.list_users_tokens linear.list_users slack.slack_list_users slack.slack_get_users github.search_users +q123,Range query Prometheus,verb_product,prometheus.prom_query_range,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,379,1.0,1,661,1.0,prometheus.prom_query_range prometheus.prom_query prometheus.prom_list_metrics datadog.datadog_query_metrics memory.search_nodes github.search_pull_requests datadog.datadog_list_monitors postgres.query salesforce.salesforce_query github.search_commits +q124,Create a Grafana annotation,verb_product,grafana.grafana_create_annotation,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,421,1.0,1,915,1.0,grafana.grafana_create_annotation grafana.grafana_search_dashboards grafana.grafana_get_dashboard github.create_artifact gitlab.create_snippet github.create_hook gitlab.create_label slack.create_canvase gitlab.create_pipeline linear.create_comment +q125,Apply terraform,identifier,terraform.tf_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,171,1.0,1,303,1.0,1,653,1.0,terraform.tf_apply terraform.tf_plan terraform.tf_state_list kubernetes.k8s_apply github.update_environment cloudflare.cf_create_dns cloudflare.cf_purge_cache cloudflare.cf_list_zones playwright.browser_fill_form hubspot.hubspot_create_deal +q126,List Asana tasks,verb_product,asana.asana_list_tasks,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,404,1.0,1,707,1.0,asana.asana_list_tasks asana.asana_create_task asana.asana_get_task asana.asana_update_task asana.asana_create_project clickup.clickup_list_tasks aws_s3.s3_list_buckets gitlab.list_jobs github.list_milestones kubernetes.k8s_list_services +q127,Comment on a Trello card,verb_product,trello.trello_add_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,368,1.0,1,721,1.0,trello.trello_add_comment trello.trello_create_card trello.trello_list_cards trello.trello_create_board linear.get_comment figma.figma_post_comment linear.update_comment linear.create_comment notion.get_comment notion.update_comment +q128,Update a ClickUp task,verb_product,clickup.clickup_update_task,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,364,1.0,1,726,1.0,clickup.clickup_update_task clickup.clickup_get_task clickup.clickup_list_tasks clickup.clickup_create_task puppeteer.puppeteer_click asana.asana_update_task playwright.browser_click gitlab.update_snippet github.update_milestone jira.update_sprint +q129,Search Confluence,verb_product,confluence.confluence_search,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,364,1.0,1,694,1.0,confluence.confluence_search confluence.confluence_get_page confluence.confluence_create_page confluence.confluence_update_page github.search_orgs github.search_users notion.notion_search github.search_issues elasticsearch.es_search grafana.grafana_search_dashboards +q130,Create a HubSpot deal,verb_product,hubspot.hubspot_create_deal,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,927,1.0,hubspot.hubspot_create_deal hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_list_contacts hubspot.hubspot_search_contacts github.create_pull_request github.merge_pull_request github.get_discussion github.pull_request_review_write github.create_gist +q131,List Zendesk tickets,verb_product,zendesk.zendesk_list_tickets,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,367,1.0,1,753,1.0,zendesk.zendesk_list_tickets zendesk.zendesk_search zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_create_ticket github.list_milestones github.list_packages jira.list_boards github.list_runners jira.list_webhooks +q132,List Discord channels,verb_product,discord.discord_list_channels,1.0,1.0,1.0,1.0000,1.0000,1,1,166,1.0,1,300,1.0,1,705,1.0,discord.discord_list_channels discord.discord_get_channel discord.discord_list_members slack.slack_list_channels discord.discord_send_message discord.discord_add_reaction slack.list_usergroups github.list_runners slack.list_pins gitlab.list_wikis +q133,Set a Redis key,verb_product,redis.redis_set,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,280,1.0,1,621,1.0,redis.redis_set redis.redis_get redis.redis_delete redis.redis_keys playwright.browser_press_key jira.jira_update_issue aws_s3.s3_put_object jira.jira_transition_issue jira.jira_get_issue slack.update_pin +q134,Insert MongoDB documents,verb_product,mongodb.mongo_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,256,1.0,1,399,1.0,1,759,1.0,mongodb.mongo_insert mongodb.mongo_update mongodb.mongo_delete supabase.supabase_insert mongodb.mongo_find mongodb.mongo_list_collections aws_s3.s3_put_object salesforce.salesforce_create_record elasticsearch.es_index_doc elasticsearch.es_get_doc +q135,Run a MySQL query,verb_product,postgres.query,0.0,1.0,1.0,0.5000,0.6309,2,1,182,1.0,1,294,1.0,1,678,1.0,postgres.query mysql.mysql_query postgres.explain_query sqlite.sqlite_query salesforce.salesforce_query mysql.mysql_list_tables prometheus.prom_query notion.notion_query_database datadog.datadog_query_metrics prometheus.prom_query_range +q136,Index a document in Elasticsearch,verb_product,elasticsearch.es_index_doc,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,367,1.0,1,717,1.0,elasticsearch.es_index_doc elasticsearch.es_search elasticsearch.es_get_doc playwright.browser_tab_close mongodb.mongo_find github.create_milestone confluence.confluence_search github.get_milestone google_drive.gdrive_search github.list_milestones +q137,Get an S3 object,verb_product,aws_s3.s3_get_object,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,373,1.0,1,632,1.0,aws_s3.s3_get_object aws_s3.s3_delete_object aws_s3.s3_put_object aws_s3.s3_list_objects asana.asana_get_task okta.okta_get_user redis.redis_get gitlab.get_snippet jira.get_sprint jira.get_attachment +q138,List Cloudflare zones,verb_product,cloudflare.cf_list_zones,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,326,1.0,1,671,1.0,cloudflare.cf_list_zones cloudflare.cf_list_dns cloudflare.cf_purge_cache cloudflare.cf_create_dns aws_s3.s3_list_buckets kubernetes.k8s_list_services kubernetes.k8s_list_deployments github.list_environments gitlab.list_pipelines slack.list_usergroups +q139,Create a Stripe invoice,verb_product,stripe.stripe_create_invoice,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,322,1.0,1,669,1.0,stripe.stripe_create_invoice stripe.stripe_create_customer stripe.stripe_create_product stripe.stripe_list_invoices stripe.stripe_create_refund stripe.stripe_create_price stripe.stripe_create_payment_intent stripe.stripe_get_customer stripe.stripe_list_customers stripe.stripe_list_subscriptions +q140,List GitHub Actions workflows,verb_product,github.actions_list,1.0,1.0,1.0,1.0000,1.0000,1,1,293,1.0,1,452,1.0,1,821,1.0,github.actions_list github.actions_get github.actions_run_trigger github.actions_get_job_logs github.list_notifications github.list_issues github.list_code_scanning_alerts github.list_gists git.git_branch github.list_releases +q141,Get GitHub Actions job logs,verb_product,github.actions_get_job_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,450,1.0,1,813,1.0,github.actions_get_job_logs github.actions_get github.actions_list github.actions_run_trigger git.git_log github.get_notification_details git.git_status github.get_code_scanning_alert github.get_secret_scanning_alert git.git_commit +q142,Create a GitHub repository,verb_product,github.create_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,268,1.0,1,428,1.0,1,927,1.0,github.create_repository github.star_repository github.create_branch github.fork_repository github.create_gist github.unstar_repository github.create_or_update_file git.git_create_branch github.delete_repository github.create_issue +q143,Checkout a git branch,verb_product,git.git_checkout,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,325,1.0,1,718,1.0,git.git_checkout git.git_branch git.git_show git.git_create_branch git.git_status github.create_branch git.git_diff github.update_pull_request_branch git.git_log github.list_branches +q144,Show git log,identifier,git.git_log,1.0,1.0,1.0,1.0000,1.0000,1,1,194,1.0,1,322,1.0,1,600,1.0,git.git_log git.git_show git.git_status git.git_diff git.git_commit git.git_branch git.git_diff_unstaged git.git_checkout git.git_diff_staged git.git_add +q145,List postgres schemas,verb_product,postgres.list_schemas,1.0,1.0,1.0,1.0000,1.0000,1,1,169,1.0,1,287,1.0,1,600,1.0,postgres.list_schemas postgres.list_tables postgres.describe_table aws_s3.s3_list_buckets github.list_artifacts kubernetes.k8s_list_deployments postgres.explain_query kubernetes.k8s_list_services github.list_environments github.list_packages +q146,Open nodes in the knowledge graph,verb_product,memory.open_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,163,1.0,1,262,1.0,1,549,1.0,memory.open_nodes memory.read_graph memory.search_nodes memory.create_entities memory.create_relations memory.delete_relations memory.add_observations memory.delete_entities figma.figma_get_file_nodes memory.delete_observations +q147,Get file metadata from the filesystem,paraphrase,filesystem.read_file,0.0,1.0,1.0,0.5000,0.6309,2,1,192,1.0,1,334,1.0,1,649,1.0,filesystem.read_file filesystem.get_file_info filesystem.read_media_file filesystem.read_text_file google_drive.gdrive_get_file figma.figma_get_file_nodes filesystem.move_file filesystem.write_file figma.figma_get_file filesystem.list_directory +q148,Write a new local file,paraphrase,filesystem.write_file,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,423,1.0,1,759,1.0,filesystem.write_file filesystem.edit_file filesystem.read_text_file slack.create_file filesystem.read_file filesystem.read_media_file playwright.browser_tab_new slack.update_file slack.get_file google_drive.gdrive_create_file +q149,Search GitHub issues,verb_product,github.search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,363,1.0,1,702,1.0,github.search_issues github.list_issues github.search_users github.search_orgs github.search_commits github.search_pull_requests github.search_code github.search_repositories linear.linear_search_issues jira.jira_search_issues +q150,Get the authenticated GitHub user,paraphrase,github.get_teams,0.0,1.0,1.0,0.5000,0.6309,2,1,179,1.0,1,285,1.0,1,700,1.0,github.get_teams github.get_me github.list_notifications slack.slack_get_user_profile github.list_gists github.get_latest_release github.projects_get github.get_secret_scanning_alert github.get_code_scanning_alert github.get_label diff --git a/experiments/mcp-tool-search/results/hybrid_rrf_weighted_embed.json b/experiments/mcp-tool-search/results/hybrid_rrf_weighted_embed.json new file mode 100644 index 00000000..90b618a1 --- /dev/null +++ b/experiments/mcp-tool-search/results/hybrid_rrf_weighted_embed.json @@ -0,0 +1,7013 @@ +{ + "summary": { + "n": 150, + "index_s": 1.9626727130001882, + "rss_mb": 0.0, + "p50_ms": 2.6834579999786, + "p95_ms": 3.2046850499682473, + "recall@1": 0.8866666666666667, + "recall@1_lo": 0.8333333333333334, + "recall@1_hi": 0.9333333333333333, + "recall@3": 0.9933333333333333, + "recall@3_lo": 0.98, + "recall@3_hi": 1.0, + "recall@5": 0.9933333333333333, + "recall@5_lo": 0.98, + "recall@5_hi": 1.0, + "mrr@10": 0.9377777777777777, + "mrr@10_lo": 0.9077777777777777, + "mrr@10_hi": 0.9655555555555556, + "ndcg@10": 0.9495127399456762, + "ndcg@10_lo": 0.9253370106437457, + "ndcg@10_hi": 0.9710196028397542 + }, + "meta": { + "name": "hybrid_rrf_weighted_embed", + "needs_model": true, + "needs_api": false, + "notes": "RRF weighted BM25 + bge-small" + }, + "rows": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "github.issue_write", + "github.add_issue_comment", + "github.issue_read", + "github.update_issue_comment", + "github.assign_copilot_to_issue", + "github.create_repository", + "gitlab.gitlab_create_issue", + "github.create_gist", + "github.search_issues" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 394, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 598, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1037, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "kind": "verb_product", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_create_issue", + "jira.jira_create_sprint", + "jira.create_sprint", + "jira.jira_list_projects", + "jira.create_board", + "jira.jira_assign_issue", + "jira.create_version", + "jira.create_webhook", + "jira.create_filter", + "jira.get_sprint" + ], + "hit1": "jira.jira_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 345, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 474, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 884, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "kind": "verb_product", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "ranking": [ + "slack.slack_post_message", + "slack.slack_add_reaction", + "slack.slack_create_channel", + "slack.slack_upload_file", + "slack.slack_reply_to_thread", + "slack.slack_leave_channel", + "slack.get_reminder", + "slack.slack_get_thread_replies", + "slack.slack_join_channel", + "slack.slack_search_messages" + ], + "hit1": "slack.slack_post_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 424, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "kind": "resource", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "ranking": [ + "github.list_pull_requests", + "github.search_pull_requests", + "github.create_pull_request", + "github.pull_request_read", + "bitbucket.bitbucket_list_prs", + "gitlab.gitlab_list_merge_requests", + "github.pull_request_review_write", + "github.update_pull_request", + "github.list_hooks", + "github.merge_pull_request" + ], + "hit1": "github.list_pull_requests", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 311, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 477, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1013, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "ranking": [ + "github.create_pull_request", + "github.merge_pull_request", + "github.pull_request_read", + "github.pull_request_review_write", + "github.search_pull_requests", + "github.update_pull_request", + "github.list_pull_requests", + "github.update_pull_request_branch", + "github.request_copilot_review", + "github.add_comment_to_pending_review" + ], + "hit1": "github.create_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 368, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 564, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1144, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "kind": "verb_product", + "gold": [ + "postgres.query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "postgres.explain_query", + "mysql.mysql_query", + "postgres.list_schemas", + "sqlite.sqlite_query", + "salesforce.salesforce_query", + "postgres.describe_table", + "postgres.list_tables", + "prometheus.prom_query", + "prometheus.prom_query_range" + ], + "hit1": "postgres.query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 272, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 596, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "kind": "paraphrase", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "ranking": [ + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "filesystem.get_file_info", + "filesystem.edit_file", + "filesystem.write_file", + "figma.figma_get_file", + "figma.figma_get_file_nodes", + "slack.get_file", + "filesystem.move_file" + ], + "hit1": "filesystem.read_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "kind": "paraphrase", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "ranking": [ + "google_drive.gdrive_list_folder", + "filesystem.list_directory", + "filesystem.list_directory_with_sizes", + "filesystem.search_files", + "filesystem.read_file", + "filesystem.directory_tree", + "filesystem.move_file", + "filesystem.get_file_info", + "filesystem.read_media_file", + "filesystem.write_file" + ], + "hit1": "google_drive.gdrive_list_folder", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6764976735216925, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 191, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 615, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "ranking": [ + "memory.read_graph", + "memory.create_relations", + "memory.add_observations", + "memory.create_entities", + "memory.delete_relations", + "memory.open_nodes", + "memory.delete_entities", + "memory.search_nodes", + "memory.delete_observations", + "sequential_thinking.sequentialthinking" + ], + "hit1": "memory.read_graph", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.543791241910304, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 164, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 261, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 585, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "kind": "verb_product", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "ranking": [ + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "confluence.confluence_search", + "memory.search_nodes", + "github.search_orgs", + "github.search_users", + "jira.jira_search_issues", + "grafana.grafana_search_dashboards", + "github.search_issues", + "hubspot.hubspot_search_contacts" + ], + "hit1": "brave_search.brave_web_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "kind": "verb_product", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "ranking": [ + "fetch.fetch", + "playwright.browser_navigate", + "github.get_file_contents", + "puppeteer.puppeteer_navigate", + "filesystem.read_file", + "playwright.browser_take_screenshot", + "playwright.browser_tab_new", + "filesystem.list_directory", + "playwright.browser_click", + "docker.docker_logs" + ], + "hit1": "fetch.fetch", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 372, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "kind": "identifier", + "gold": [ + "git.git_status" + ], + "alts": [], + "ranking": [ + "git.git_status", + "git.git_show", + "git.git_diff", + "git.git_log", + "git.git_diff_unstaged", + "git.git_branch", + "git.git_commit", + "git.git_diff_staged", + "git.git_checkout", + "git.git_add" + ], + "hit1": "git.git_status", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "kind": "paraphrase", + "gold": [ + "git.git_commit" + ], + "alts": [], + "ranking": [ + "git.git_commit", + "git.git_reset", + "git.git_diff_staged", + "git.git_show", + "github.get_commit", + "github.update_milestone", + "git.git_log", + "filesystem.move_file", + "github.push_files", + "slack.update_reminder" + ], + "hit1": "git.git_commit", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_issue", + "linear.linear_get_issue", + "linear.linear_update_issue", + "linear.linear_assign_issue", + "linear.linear_create_comment", + "linear.linear_search_issues", + "linear.linear_create_project", + "linear.linear_list_issues", + "linear.create_label", + "linear.create_cycle" + ], + "hit1": "linear.linear_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 322, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 875, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "kind": "paraphrase", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_project", + "linear.linear_create_issue", + "linear.linear_list_projects", + "linear.linear_update_issue", + "linear.linear_list_issues", + "linear.linear_create_comment", + "linear.linear_get_issue", + "linear.linear_list_teams", + "linear.linear_assign_issue", + "linear.linear_search_issues" + ], + "hit1": "linear.linear_create_project", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 284, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 483, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 769, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "ranking": [ + "github.update_issue_comment", + "github.add_issue_comment", + "github.issue_write", + "github.issue_read", + "github.assign_copilot_to_issue", + "github.create_issue", + "github.add_comment_to_pending_review", + "jira.jira_add_comment", + "github.get_discussion_comments", + "linear.linear_create_comment" + ], + "hit1": "github.update_issue_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 370, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 573, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1087, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "kind": "ambiguous", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "ranking": [ + "github.merge_pull_request", + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_get_merge_request", + "github.update_pull_request_branch", + "github.create_pull_request", + "github.pull_request_read", + "github.update_pull_request", + "gitlab.gitlab_create_merge_request", + "gitlab.gitlab_list_merge_requests", + "github.pull_request_review_write" + ], + "hit1": "github.merge_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 528, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1093, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "kind": "verb_product", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "ranking": [ + "github.actions_run_trigger", + "github.actions_get", + "github.actions_list", + "github.actions_get_job_logs", + "circleci.circleci_trigger_pipeline", + "github.update_gist", + "github.search_commits", + "github.issue_read", + "github.search_pull_requests", + "git.git_checkout" + ], + "hit1": "github.actions_run_trigger", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 805, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "kind": "resource", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "ranking": [ + "github.list_dependabot_alerts", + "github.get_dependabot_alert", + "github.list_code_scanning_alerts", + "github.list_secret_scanning_alerts", + "slack.get_reminder", + "github.get_milestone", + "github.get_hook", + "gitlab.get_release", + "sentry.sentry_get_issue", + "gitlab.get_snippet" + ], + "hit1": "github.list_dependabot_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 287, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 422, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 708, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "kind": "verb_product", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_issues", + "sentry.sentry_list_projects", + "sentry.sentry_list_events", + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "linear.linear_list_issues", + "github.list_issues", + "gitlab.gitlab_list_issues", + "github.list_artifacts", + "gitlab.list_pipelines" + ], + "hit1": "sentry.sentry_list_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 812, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_list_invoices", + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_product", + "stripe.stripe_list_subscriptions", + "stripe.stripe_create_price", + "stripe.stripe_create_refund", + "stripe.stripe_list_customers" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "kind": "paraphrase", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_refund", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_customer", + "stripe.stripe_create_invoice", + "stripe.stripe_create_customer", + "stripe.stripe_cancel_subscription", + "stripe.stripe_list_invoices", + "stripe.stripe_list_customers", + "stripe.stripe_create_price" + ], + "hit1": "stripe.stripe_create_refund", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 332, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 644, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "kind": "verb_product", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "ranking": [ + "gmail.gmail_send_email", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "gmail.gmail_modify_email", + "gmail.gmail_create_draft", + "gmail.gmail_search_emails", + "gmail.gmail_list_labels", + "twilio.twilio_send_sms", + "discord.discord_send_message", + "google_drive.gdrive_share_file" + ], + "hit1": "gmail.gmail_send_email", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 743, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "kind": "verb_product", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "ranking": [ + "gmail.gmail_search_emails", + "gmail.gmail_list_labels", + "gmail.gmail_get_email", + "gmail.gmail_create_draft", + "gmail.gmail_trash_email", + "gmail.gmail_send_email", + "gmail.gmail_modify_email", + "github.search_users", + "hubspot.hubspot_search_contacts", + "github.search_orgs" + ], + "hit1": "gmail.gmail_search_emails", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 661, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "kind": "paraphrase", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_update_file", + "google_drive.gdrive_share_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "slack.slack_upload_file", + "filesystem.write_file", + "filesystem.edit_file", + "gitlab.gitlab_get_file" + ], + "hit1": "google_drive.gdrive_update_file", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 307, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 810, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_create_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "github.create_milestone", + "slack.create_reminder", + "gitlab.create_milestone", + "linear.create_cycle", + "slack.create_canvase" + ], + "hit1": "google_calendar.gcal_create_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 945, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_list_pods", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_apply", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_delete", + "aws_s3.s3_list_buckets" + ], + "hit1": "kubernetes.k8s_list_pods", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_get_pod", + "docker.docker_logs", + "github.actions_get_job_logs", + "datadog.datadog_list_logs", + "git.git_log", + "kubernetes.k8s_get_events", + "slack.slack_get_channel_history", + "memory.delete_observations", + "redis.redis_get" + ], + "hit1": "kubernetes.k8s_pod_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 741, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "ranking": [ + "docker.docker_run", + "docker.docker_stop", + "docker.docker_exec", + "docker.docker_logs", + "docker.docker_rm", + "docker.docker_ps", + "docker.docker_images", + "docker.docker_build", + "github.create_check_run", + "github.get_check_run" + ], + "hit1": "docker.docker_run", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 644, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "kind": "verb_product", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "ranking": [ + "docker.docker_logs", + "docker.docker_run", + "docker.docker_stop", + "docker.docker_images", + "kubernetes.k8s_pod_logs", + "docker.docker_exec", + "docker.docker_ps", + "docker.docker_rm", + "git.git_log", + "docker.docker_build" + ], + "hit1": "docker.docker_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 649, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "kind": "ambiguous", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "ranking": [ + "playwright.browser_navigate", + "playwright.browser_tab_new", + "puppeteer.puppeteer_navigate", + "playwright.browser_click", + "playwright.browser_tab_list", + "playwright.browser_press_key", + "playwright.browser_take_screenshot", + "playwright.browser_evaluate", + "playwright.browser_snapshot", + "playwright.browser_tab_close" + ], + "hit1": "playwright.browser_navigate", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 279, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 543, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "kind": "ambiguous", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "ranking": [ + "playwright.browser_take_screenshot", + "puppeteer.puppeteer_screenshot", + "playwright.browser_snapshot", + "playwright.browser_evaluate", + "notion.notion_get_page", + "playwright.browser_click", + "playwright.browser_press_key", + "confluence.confluence_get_page", + "confluence.confluence_create_page", + "confluence.confluence_update_page" + ], + "hit1": "playwright.browser_take_screenshot", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_create_merge_request", + "gitlab.gitlab_get_merge_request", + "github.merge_pull_request", + "gitlab.gitlab_list_merge_requests", + "gitlab.gitlab_create_pipeline", + "github.create_pull_request", + "gitlab.gitlab_create_issue", + "gitlab.gitlab_get_file", + "gitlab.gitlab_get_issue" + ], + "hit1": "gitlab.gitlab_merge_merge_request", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 951, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "kind": "verb_product", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "ranking": [ + "notion.notion_search", + "notion.notion_get_page", + "notion.notion_query_database", + "notion.notion_update_page", + "notion.notion_get_block", + "notion.notion_create_database", + "notion.notion_create_page", + "notion.notion_list_users", + "notion.notion_get_comments", + "notion.get_users_token" + ], + "hit1": "notion.notion_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 671, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "kind": "verb_product", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "ranking": [ + "notion.notion_get_page", + "notion.notion_create_page", + "notion.notion_update_page", + "notion.notion_create_database", + "notion.notion_search", + "notion.notion_get_comments", + "notion.notion_list_users", + "notion.create_data_source", + "notion.notion_get_block", + "notion.create_users_token" + ], + "hit1": "notion.notion_get_page", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "kind": "verb_product", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "ranking": [ + "notion.notion_query_database", + "notion.notion_create_database", + "notion.notion_search", + "notion.notion_create_page", + "notion.notion_get_page", + "notion.notion_get_block", + "notion.notion_list_users", + "notion.notion_get_comments", + "notion.notion_update_page", + "notion.get_comment" + ], + "hit1": "notion.notion_query_database", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_search", + "jira.jira_create_issue", + "jira.create_board", + "jira.create_sprint", + "github.create_milestone", + "gitlab.create_milestone" + ], + "hit1": "zendesk.zendesk_create_ticket", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 891, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_create_deal", + "hubspot.hubspot_search_contacts", + "hubspot.hubspot_list_contacts", + "github.create_pull_request", + "github.get_discussion", + "github.pull_request_review_write", + "github.create_gist", + "slack.create_usergroup" + ], + "hit1": "hubspot.hubspot_create_contact", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 925, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "kind": "verb_product", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_convert_lead", + "salesforce.salesforce_create_lead", + "salesforce.salesforce_create_record", + "salesforce.salesforce_list_opportunities", + "salesforce.salesforce_get_record", + "salesforce.salesforce_query", + "salesforce.salesforce_update_record", + "github.create_hook", + "github.create_runner", + "linear.create_user" + ], + "hit1": "salesforce.salesforce_convert_lead", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 815, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "kind": "identifier", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_query", + "postgres.query", + "mysql.mysql_query", + "sqlite.sqlite_query", + "prometheus.prom_query", + "postgres.explain_query", + "prometheus.prom_query_range", + "datadog.datadog_query_metrics", + "notion.notion_query_database", + "datadog.datadog_list_monitors" + ], + "hit1": "salesforce.salesforce_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 187, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "kind": "verb_product", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "ranking": [ + "shopify.shopify_list_orders", + "shopify.shopify_list_products", + "shopify.shopify_list_customers", + "shopify.shopify_get_order", + "shopify.shopify_get_product", + "shopify.shopify_create_order", + "shopify.shopify_create_product", + "shopify.shopify_update_product", + "clickup.clickup_list_tasks", + "aws_s3.s3_list_buckets" + ], + "hit1": "shopify.shopify_list_orders", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "slack.create_bookmark", + "slack.create_reminder", + "slack.create_canvase", + "notion.create_comment" + ], + "hit1": "pagerduty.pagerduty_create_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 874, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "kind": "paraphrase", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_list_oncalls", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "twilio.twilio_make_call", + "playwright.browser_press_key", + "playwright.browser_click", + "filesystem.list_allowed_directories" + ], + "hit1": "pagerduty.pagerduty_list_oncalls", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "kind": "verb_product", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "ranking": [ + "datadog.datadog_query_metrics", + "datadog.datadog_list_monitors", + "datadog.datadog_create_monitor", + "datadog.datadog_list_logs", + "datadog.datadog_get_monitor", + "prometheus.prom_list_metrics", + "memory.search_nodes", + "prometheus.prom_query", + "github.search_pull_requests", + "prometheus.prom_query_range" + ], + "hit1": "datadog.datadog_query_metrics", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 769, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "kind": "verb_product", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "ranking": [ + "datadog.datadog_list_logs", + "datadog.datadog_list_monitors", + "datadog.datadog_get_monitor", + "datadog.datadog_create_monitor", + "datadog.datadog_query_metrics", + "docker.docker_logs", + "github.search_users", + "github.search_issues", + "github.actions_get_job_logs", + "github.search_orgs" + ], + "hit1": "datadog.datadog_list_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 802, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_list_buckets", + "aws_s3.s3_list_objects", + "kubernetes.k8s_list_services", + "gitlab.list_snippets", + "clickup.clickup_list_tasks", + "kubernetes.k8s_list_deployments", + "jira.list_webhooks", + "gitlab.list_pipelines", + "github.list_environments", + "github.list_milestones" + ], + "hit1": "aws_s3.s3_list_buckets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 185, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 318, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 690, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "kind": "paraphrase", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_put_object", + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "slack.slack_upload_file", + "aws_s3.s3_list_objects", + "twilio.twilio_send_sms", + "filesystem.write_file", + "filesystem.edit_file", + "github.assign_copilot_to_issue", + "slack.slack_reply_to_thread" + ], + "hit1": "aws_s3.s3_put_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 847, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "slack.delete_bookmark", + "slack.delete_canvase", + "memory.delete_observations", + "redis.redis_delete", + "aws_s3.s3_delete_object", + "linear.delete_attachment" + ], + "hit1": "cloudflare.cf_purge_cache", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 618, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "salesforce.salesforce_create_record", + "salesforce.salesforce_get_record", + "salesforce.salesforce_update_record", + "slack.create_file", + "github.create_environment", + "gitlab.create_pipeline" + ], + "hit1": "cloudflare.cf_create_dns", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 812, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_deployments", + "vercel.vercel_list_projects", + "vercel.vercel_get_deployment", + "vercel.vercel_create_project", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "github.list_packages", + "github.list_environments", + "gitlab.list_releases", + "github.list_artifacts" + ], + "hit1": "vercel.vercel_list_deployments", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 650, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "ranking": [ + "supabase.supabase_insert", + "supabase.supabase_select", + "supabase.supabase_update", + "supabase.supabase_invoke_fn", + "supabase.supabase_list_tables", + "mongodb.mongo_insert", + "playwright.browser_type", + "linear.update_cycle", + "linear.update_attachment", + "linear.update_user" + ], + "hit1": "supabase.supabase_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 701, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "kind": "paraphrase", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "ranking": [ + "figma.figma_get_images", + "figma.figma_get_file", + "figma.figma_get_file_nodes", + "figma.figma_get_comments", + "figma.figma_post_comment", + "docker.docker_images", + "filesystem.read_media_file", + "github.get_file_contents", + "elasticsearch.es_get_doc", + "linear.update_attachment" + ], + "hit1": "figma.figma_get_images", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 701, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "kind": "verb_product", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "ranking": [ + "twilio.twilio_send_sms", + "twilio.twilio_list_messages", + "twilio.twilio_make_call", + "gmail.gmail_send_email", + "discord.discord_send_message", + "slack.slack_add_reaction", + "aws_s3.s3_put_object", + "playwright.browser_type", + "filesystem.list_directory_with_sizes", + "slack.slack_reply_to_thread" + ], + "hit1": "twilio.twilio_send_sms", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 811, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "kind": "verb_product", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "ranking": [ + "okta.okta_get_user", + "okta.okta_create_user", + "okta.okta_deactivate_user", + "okta.okta_list_users", + "linear.create_user", + "asana.asana_create_task", + "asana.asana_create_project", + "linear.get_user", + "linear.update_user", + "slack.slack_get_user_profile" + ], + "hit1": "okta.okta_get_user", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 676, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query", + "prometheus.prom_query_range", + "prometheus.prom_list_metrics", + "datadog.datadog_query_metrics", + "memory.search_nodes", + "postgres.query", + "salesforce.salesforce_query", + "github.search_pull_requests", + "sqlite.sqlite_query", + "notion.notion_query_database" + ], + "hit1": "prometheus.prom_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "kind": "verb_product", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "ranking": [ + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "grafana.grafana_create_annotation", + "github.search_orgs", + "github.search_users", + "github.search_issues", + "brave_search.brave_local_search", + "confluence.confluence_search", + "brave_search.brave_web_search", + "github.search_pull_requests" + ], + "hit1": "grafana.grafana_search_dashboards", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 677, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "kind": "verb_product", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "ranking": [ + "circleci.circleci_trigger_pipeline", + "circleci.circleci_get_pipeline", + "circleci.circleci_list_pipelines", + "github.actions_run_trigger", + "gitlab.gitlab_create_pipeline", + "gitlab.update_pipeline", + "hubspot.hubspot_create_deal", + "gitlab.get_pipeline", + "gitlab.create_pipeline", + "gitlab.delete_pipeline" + ], + "hit1": "circleci.circleci_trigger_pipeline", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "kind": "identifier", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "ranking": [ + "terraform.tf_plan", + "terraform.tf_apply", + "terraform.tf_state_list", + "github.update_check_run", + "docker.docker_run", + "github.get_check_run", + "github.create_check_run", + "github.actions_run_trigger", + "docker.docker_exec", + "github.actions_get" + ], + "hit1": "terraform.tf_plan", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "kind": "verb_product", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "ranking": [ + "asana.asana_create_task", + "asana.asana_get_task", + "asana.asana_create_project", + "asana.asana_update_task", + "asana.asana_list_tasks", + "clickup.clickup_create_task", + "okta.okta_create_user", + "github.create_issue", + "clickup.clickup_get_task", + "gitlab.create_snippet" + ], + "hit1": "asana.asana_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 837, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "ranking": [ + "trello.trello_create_card", + "trello.trello_add_comment", + "trello.trello_create_board", + "trello.trello_list_cards", + "linear.create_label", + "linear.create_cycle", + "stripe.stripe_create_invoice", + "slack.create_reminder", + "linear.create_attachment", + "stripe.stripe_create_customer" + ], + "hit1": "trello.trello_create_card", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 411, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 819, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_create_task", + "clickup.clickup_get_task", + "clickup.clickup_list_tasks", + "clickup.clickup_update_task", + "puppeteer.puppeteer_click", + "playwright.browser_click", + "asana.asana_create_task", + "gitlab.create_snippet", + "asana.asana_get_task", + "jira.create_sprint" + ], + "hit1": "clickup.clickup_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 787, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "kind": "verb_product", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "ranking": [ + "confluence.confluence_create_page", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "confluence.confluence_search", + "notion.notion_create_page", + "notion.notion_get_page", + "github.create_artifact", + "playwright.browser_snapshot", + "github.create_environment", + "gitlab.create_wiki" + ], + "hit1": "confluence.confluence_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 791, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "kind": "verb_product", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "discord.discord_add_reaction", + "discord.discord_get_channel", + "slack.slack_post_message", + "discord.discord_list_channels", + "slack.slack_add_reaction", + "gmail.gmail_trash_email", + "twilio.twilio_send_sms", + "discord.discord_list_members", + "gmail.gmail_send_email" + ], + "hit1": "discord.discord_send_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_search", + "confluence.confluence_search", + "elasticsearch.es_get_doc", + "elasticsearch.es_index_doc", + "github.search_orgs", + "github.search_users", + "brave_search.brave_web_search", + "github.search_issues", + "hubspot.hubspot_search_contacts", + "github.search_pull_requests" + ], + "hit1": "elasticsearch.es_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "redis.redis_set", + "redis.redis_delete", + "redis.redis_keys", + "playwright.browser_press_key", + "figma.figma_get_file", + "jira.jira_get_issue", + "aws_s3.s3_get_object", + "github.get_milestone", + "gitlab.get_pipeline" + ], + "hit1": "redis.redis_get", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 566, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_find", + "mongodb.mongo_insert", + "mongodb.mongo_list_collections", + "mongodb.mongo_update", + "mongodb.mongo_delete", + "elasticsearch.es_get_doc", + "filesystem.search_files", + "confluence.confluence_search", + "datadog.datadog_list_logs", + "hubspot.hubspot_search_contacts" + ], + "hit1": "mongodb.mongo_find", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "kind": "verb_product", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "ranking": [ + "sqlite.sqlite_list_tables", + "mysql.mysql_list_tables", + "postgres.list_tables", + "supabase.supabase_list_tables", + "sqlite.sqlite_describe_table", + "postgres.list_schemas", + "trello.trello_list_cards", + "gitlab.list_wikis", + "gitlab.list_pipelines", + "gitlab.list_labels" + ], + "hit1": "sqlite.sqlite_list_tables", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 240, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 590, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "kind": "verb_product", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "ranking": [ + "postgres.describe_table", + "sqlite.sqlite_describe_table", + "postgres.explain_query", + "postgres.list_tables", + "postgres.list_schemas", + "postgres.query", + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update", + "mysql.mysql_list_tables" + ], + "hit1": "postgres.describe_table", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 274, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 590, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "github.issue_write", + "gitlab.gitlab_create_issue", + "bitbucket.bitbucket_create_issue", + "github.assign_copilot_to_issue", + "github.add_issue_comment", + "github.create_environment", + "github.create_package", + "github.update_issue_comment", + "github.issue_read" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 378, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 573, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1072, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "ranking": [ + "linear.create_user", + "linear.create_cycle", + "linear.create_comment", + "github.create_artifact", + "jira.create_board", + "linear.create_attachment", + "jira.create_filter", + "linear.create_label", + "github.create_gist", + "jira.create_attachment" + ], + "hit1": "linear.create_user", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 39, + "k3_calls": 13, + "k3_tokens": 3608, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 8, + "k5_tokens": 3654, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3582, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q071", + "query": "list-directory", + "kind": "identifier", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "ranking": [ + "filesystem.list_directory", + "filesystem.list_directory_with_sizes", + "filesystem.list_allowed_directories", + "clickup.clickup_list_tasks", + "github.list_environments", + "filesystem.directory_tree", + "aws_s3.s3_list_buckets", + "trello.trello_list_cards", + "github.list_runners", + "github.list_packages" + ], + "hit1": "filesystem.list_directory", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 623, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "kind": "paraphrase", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "ranking": [ + "sequential_thinking.sequentialthinking", + "memory.create_relations", + "memory.add_observations", + "memory.search_nodes", + "memory.create_entities", + "memory.delete_relations", + "notion.update_comment", + "notion.notion_search", + "memory.open_nodes", + "notion.notion_append_block" + ], + "hit1": "sequential_thinking.sequentialthinking", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 650, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "kind": "paraphrase", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "ranking": [ + "time.get_current_time", + "time.convert_time", + "google_calendar.gcal_list_events", + "playwright.browser_wait_for", + "grafana.grafana_create_annotation", + "jira.jira_add_worklog", + "zendesk.zendesk_list_tickets", + "filesystem.list_allowed_directories", + "slack.slack_create_channel", + "jira.jira_create_sprint" + ], + "hit1": "time.get_current_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "kind": "verb_product", + "gold": [ + "time.convert_time" + ], + "alts": [], + "ranking": [ + "time.convert_time", + "time.get_current_time", + "playwright.browser_wait_for", + "grafana.grafana_create_annotation", + "google_calendar.gcal_list_events", + "salesforce.salesforce_convert_lead", + "git.git_diff", + "datadog.datadog_query_metrics", + "memory.create_relations", + "cloudflare.cf_list_zones" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "kind": "verb_product", + "gold": [ + "github.star_repository" + ], + "alts": [], + "ranking": [ + "github.star_repository", + "github.unstar_repository", + "github.get_label", + "github.search_issues", + "github.get_file_blame", + "github.list_releases", + "git.git_show", + "github.create_gist", + "github.list_code_scanning_alerts", + "github.actions_list" + ], + "hit1": "github.star_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 425, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 801, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "kind": "resource", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "ranking": [ + "github.fork_repository", + "github.unstar_repository", + "github.star_repository", + "github.delete_repository", + "github.create_repository", + "github.get_repository_tree", + "github.list_dependabot_alerts", + "github.list_releases", + "github.list_pull_requests", + "github.create_or_update_file" + ], + "hit1": "github.fork_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 875, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "kind": "verb_product", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "ranking": [ + "github.get_latest_release", + "github.get_release_by_tag", + "gitlab.get_release", + "github.get_gist", + "github.get_notification_details", + "github.get_code_scanning_alert", + "github.get_secret_scanning_alert", + "github.get_commit", + "github.get_teams", + "gitlab.update_release" + ], + "hit1": "github.get_latest_release", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "kind": "resource", + "gold": [ + "github.create_gist" + ], + "alts": [], + "ranking": [ + "github.create_gist", + "github.update_gist", + "github.get_gist", + "github.create_milestone", + "github.create_hook", + "gitlab.create_snippet", + "github.create_artifact", + "github.create_package", + "gitlab.create_wiki", + "github.create_runner" + ], + "hit1": "github.create_gist", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 925, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "kind": "verb_product", + "gold": [ + "github.search_code" + ], + "alts": [], + "ranking": [ + "github.search_code", + "github.get_code_scanning_alert", + "github.list_code_scanning_alerts", + "github.search_orgs", + "github.search_users", + "github.search_pull_requests", + "github.search_commits", + "github.search_repositories", + "github.search_issues", + "github.get_label" + ], + "hit1": "github.search_code", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "kind": "verb_product", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "ranking": [ + "github.list_notifications", + "github.mark_all_notifications_read", + "github.list_code_scanning_alerts", + "github.list_secret_scanning_alerts", + "github.actions_list", + "github.list_gists", + "github.list_releases", + "github.list_starred_repositories", + "github.list_pull_requests", + "github.list_tags" + ], + "hit1": "github.list_notifications", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "kind": "verb_product", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "ranking": [ + "github.assign_copilot_to_issue", + "linear.linear_assign_issue", + "jira.jira_assign_issue", + "github.request_copilot_review", + "github.create_issue", + "jira.jira_update_issue", + "jira.jira_get_issue", + "jira.jira_transition_issue", + "sentry.sentry_get_issue", + "github.add_issue_comment" + ], + "hit1": "github.assign_copilot_to_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 500, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 862, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "kind": "resource", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "ranking": [ + "github.list_secret_scanning_alerts", + "github.get_secret_scanning_alert", + "github.list_code_scanning_alerts", + "github.get_code_scanning_alert", + "github.list_dependabot_alerts", + "github.get_dependabot_alert", + "github.get_hook", + "slack.get_reminder", + "gitlab.get_snippet", + "slack.get_pin" + ], + "hit1": "github.list_secret_scanning_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 776, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "ranking": [ + "github.create_or_update_file", + "github.update_gist", + "github.projects_write", + "git.git_add", + "github.create_gist", + "slack.update_file", + "github.get_file_contents", + "github.issue_write", + "github.create_pull_request", + "github.update_pull_request" + ], + "hit1": "github.create_or_update_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 476, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1078, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "kind": "resource", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "ranking": [ + "github.list_repository_collaborators", + "github.list_commits", + "github.list_labels", + "github.list_releases", + "github.actions_list", + "github.list_tags", + "github.list_discussions", + "github.list_branches", + "github.list_issues", + "github.list_dependabot_alerts" + ], + "hit1": "github.list_repository_collaborators", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 271, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 410, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 804, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "kind": "verb_product", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "ranking": [ + "slack.slack_reply_to_thread", + "slack.slack_get_thread_replies", + "slack.slack_post_message", + "slack.slack_search_messages", + "slack.slack_add_reaction", + "slack.slack_create_channel", + "slack.slack_upload_file", + "slack.get_reminder", + "slack.update_reminder", + "slack.slack_join_channel" + ], + "hit1": "slack.slack_reply_to_thread", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 748, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "kind": "verb_product", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "ranking": [ + "slack.slack_add_reaction", + "slack.slack_reply_to_thread", + "slack.slack_post_message", + "slack.slack_get_thread_replies", + "discord.discord_add_reaction", + "slack.slack_upload_file", + "slack.slack_create_channel", + "slack.get_reminder", + "slack.update_reminder", + "slack.slack_search_messages" + ], + "hit1": "slack.slack_add_reaction", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9072278740982787, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 271, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 436, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 785, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "kind": "verb_product", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "ranking": [ + "slack.slack_search_messages", + "slack.slack_get_channel_history", + "slack.slack_add_reaction", + "slack.slack_post_message", + "slack.slack_create_channel", + "slack.slack_get_thread_replies", + "slack.slack_reply_to_thread", + "slack.slack_list_channels", + "slack.get_reminder", + "slack.slack_get_users" + ], + "hit1": "slack.slack_search_messages", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 393, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 748, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "kind": "verb_product", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "ranking": [ + "slack.slack_join_channel", + "slack.slack_create_channel", + "slack.slack_leave_channel", + "slack.slack_post_message", + "slack.slack_set_channel_topic", + "slack.slack_add_reaction", + "slack.slack_get_channel_history", + "slack.slack_reply_to_thread", + "slack.slack_upload_file", + "slack.slack_get_thread_replies" + ], + "hit1": "slack.slack_join_channel", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "kind": "verb_product", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_transition_issue", + "jira.jira_assign_issue", + "jira.jira_update_issue", + "jira.jira_get_issue", + "jira.jira_add_comment", + "jira.jira_create_issue", + "jira.jira_search_issues", + "jira.jira_create_sprint", + "jira.update_sprint", + "jira.get_sprint" + ], + "hit1": "jira.jira_transition_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 222, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 760, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "kind": "identifier", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "ranking": [ + "jira.jira_search_issues", + "jira.jira_get_issue", + "jira.jira_list_projects", + "jira.get_filter", + "jira.create_filter", + "jira.jira_update_issue", + "jira.get_board", + "jira.jira_assign_issue", + "jira.update_filter", + "jira.get_sprint" + ], + "hit1": "jira.jira_search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 325, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 627, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "kind": "verb_product", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "ranking": [ + "jira.jira_add_worklog", + "jira.jira_add_comment", + "jira.jira_list_projects", + "jira.jira_create_issue", + "jira.jira_create_sprint", + "jira.jira_update_issue", + "jira.jira_get_issue", + "jira.create_board", + "jira.create_sprint", + "jira.jira_transition_issue" + ], + "hit1": "jira.jira_add_worklog", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 439, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 821, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "kind": "verb_product", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "ranking": [ + "linear.linear_list_teams", + "linear.linear_list_projects", + "linear.linear_list_issues", + "linear.linear_create_issue", + "linear.linear_create_project", + "linear.linear_get_issue", + "linear.list_cycles", + "linear.linear_search_issues", + "linear.list_users", + "linear.list_labels" + ], + "hit1": "linear.linear_list_teams", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 398, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "ranking": [ + "linear.linear_create_comment", + "linear.linear_get_issue", + "linear.linear_assign_issue", + "linear.linear_update_issue", + "linear.linear_create_issue", + "linear.get_comment", + "linear.update_comment", + "linear.create_comment", + "linear.linear_search_issues", + "linear.delete_comment" + ], + "hit1": "linear.linear_create_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 464, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "kind": "paraphrase", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "ranking": [ + "filesystem.move_file", + "filesystem.read_text_file", + "filesystem.read_file", + "filesystem.read_media_file", + "filesystem.write_file", + "slack.update_file", + "slack.get_file", + "slack.create_file", + "figma.figma_get_file", + "filesystem.edit_file" + ], + "hit1": "filesystem.move_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 715, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "kind": "paraphrase", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "ranking": [ + "filesystem.search_files", + "filesystem.directory_tree", + "redis.redis_keys", + "filesystem.list_directory", + "filesystem.list_directory_with_sizes", + "google_drive.gdrive_search", + "filesystem.get_file_info", + "filesystem.move_file", + "filesystem.read_text_file", + "linear.linear_search_issues" + ], + "hit1": "filesystem.search_files", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 305, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 617, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "kind": "paraphrase", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "ranking": [ + "filesystem.edit_file", + "filesystem.write_file", + "slack.update_file", + "slack.create_file", + "filesystem.move_file", + "google_drive.gdrive_update_file", + "slack.get_file", + "slack.delete_file", + "filesystem.read_text_file", + "slack.slack_upload_file" + ], + "hit1": "filesystem.edit_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 413, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 779, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "ranking": [ + "memory.read_graph", + "memory.delete_relations", + "memory.open_nodes", + "memory.add_observations", + "memory.create_entities", + "memory.search_nodes", + "memory.create_relations", + "memory.delete_entities", + "github.mark_all_notifications_read", + "memory.delete_observations" + ], + "hit1": "memory.read_graph", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 154, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 258, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 510, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "kind": "verb_product", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "memory.open_nodes", + "figma.figma_get_file_nodes", + "memory.read_graph", + "memory.delete_entities", + "github.search_orgs", + "memory.add_observations", + "filesystem.search_files", + "memory.create_relations", + "memory.delete_relations" + ], + "hit1": "memory.search_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 288, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 564, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "kind": "verb_product", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "ranking": [ + "git.git_diff_unstaged", + "git.git_diff", + "git.git_diff_staged", + "git.git_show", + "git.git_status", + "git.git_reset", + "git.git_commit", + "git.git_log", + "git.git_add", + "git.git_branch" + ], + "hit1": "git.git_diff_unstaged", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 588, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "kind": "identifier", + "gold": [ + "git.git_add" + ], + "alts": [], + "ranking": [ + "git.git_add", + "git.git_commit", + "git.git_show", + "git.git_diff_staged", + "git.git_diff", + "git.git_reset", + "git.git_branch", + "github.push_files", + "git.git_diff_unstaged", + "git.git_log" + ], + "hit1": "git.git_add", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 326, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 651, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "kind": "verb_product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "ranking": [ + "shopify.shopify_create_product", + "shopify.shopify_get_product", + "shopify.shopify_update_product", + "shopify.shopify_create_order", + "shopify.shopify_list_products", + "shopify.shopify_get_order", + "shopify.shopify_list_orders", + "stripe.stripe_create_product", + "shopify.shopify_list_customers", + "stripe.stripe_create_price" + ], + "hit1": "shopify.shopify_create_product", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "kind": "verb_product", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "ranking": [ + "stripe.stripe_cancel_subscription", + "stripe.stripe_list_invoices", + "stripe.stripe_create_invoice", + "stripe.stripe_list_subscriptions", + "stripe.stripe_list_customers", + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_create_refund", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_price" + ], + "hit1": "stripe.stripe_cancel_subscription", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 167, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 297, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 622, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "kind": "resource", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_refund", + "github.create_milestone", + "slack.create_reminder", + "stripe.stripe_create_invoice", + "stripe.stripe_create_price", + "github.create_hook", + "stripe.stripe_create_product", + "stripe.stripe_create_customer" + ], + "hit1": "stripe.stripe_create_payment_intent", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 441, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 800, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "sentry.sentry_list_events", + "slack.get_bookmark", + "slack.get_reminder", + "notion.notion_get_page" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "kind": "verb_product", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "ranking": [ + "datadog.datadog_get_monitor", + "datadog.datadog_create_monitor", + "datadog.datadog_list_monitors", + "datadog.datadog_list_logs", + "datadog.datadog_query_metrics", + "memory.create_entities", + "slack.create_bookmark", + "slack.create_reminder", + "memory.create_relations", + "github.create_gist" + ], + "hit1": "datadog.datadog_get_monitor", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 840, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "kind": "paraphrase", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "ranking": [ + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "sentry.sentry_list_issues", + "sentry.sentry_list_events", + "sentry.sentry_list_projects", + "pagerduty.pagerduty_resolve_incident", + "jira.jira_assign_issue", + "jira.jira_get_issue", + "github.issue_write", + "linear.linear_update_issue" + ], + "hit1": "sentry.sentry_get_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 274, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 832, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_apply", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_list_services", + "kubernetes.k8s_delete", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_get_events", + "vercel.vercel_get_deployment" + ], + "hit1": "kubernetes.k8s_scale_deployment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 241, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_apply", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_delete", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_get_events", + "kubernetes.k8s_pod_logs", + "terraform.tf_apply" + ], + "hit1": "kubernetes.k8s_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "ranking": [ + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_run", + "docker.docker_exec", + "docker.docker_images", + "docker.docker_ps", + "docker.docker_build", + "kubernetes.k8s_pod_logs", + "github.list_environments" + ], + "hit1": "docker.docker_stop", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 664, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "kind": "verb_product", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "ranking": [ + "docker.docker_build", + "docker.docker_run", + "docker.docker_images", + "docker.docker_stop", + "docker.docker_logs", + "docker.docker_ps", + "docker.docker_exec", + "docker.docker_rm", + "filesystem.read_media_file", + "github.create_environment" + ], + "hit1": "docker.docker_build", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 642, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "kind": "verb_product", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "ranking": [ + "playwright.browser_click", + "playwright.browser_type", + "puppeteer.puppeteer_click", + "playwright.browser_navigate", + "playwright.browser_take_screenshot", + "playwright.browser_press_key", + "playwright.browser_tab_list", + "playwright.browser_tab_new", + "playwright.browser_snapshot", + "playwright.browser_fill_form" + ], + "hit1": "playwright.browser_click", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 566, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "kind": "paraphrase", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "ranking": [ + "playwright.browser_type", + "playwright.browser_navigate", + "playwright.browser_click", + "playwright.browser_wait_for", + "playwright.browser_tab_new", + "playwright.browser_fill_form", + "puppeteer.puppeteer_fill", + "playwright.browser_press_key", + "playwright.browser_tab_list", + "playwright.browser_evaluate" + ], + "hit1": "playwright.browser_type", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8868854556705132, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 330, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 586, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_list_projects", + "gitlab.gitlab_list_issues", + "gitlab.gitlab_list_merge_requests", + "git.git_branch", + "vercel.vercel_list_projects", + "jira.jira_list_projects", + "github.projects_list", + "gitlab.gitlab_create_issue", + "gitlab.gitlab_get_issue", + "gitlab.gitlab_create_pipeline" + ], + "hit1": "gitlab.gitlab_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 635, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "kind": "verb_product", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "ranking": [ + "bitbucket.bitbucket_create_pr", + "bitbucket.bitbucket_list_prs", + "bitbucket.bitbucket_create_issue", + "github.create_pull_request", + "github.pull_request_read", + "github.merge_pull_request", + "bitbucket.bitbucket_list_repos", + "github.pull_request_review_write", + "github.update_pull_request_branch", + "github.search_pull_requests" + ], + "hit1": "bitbucket.bitbucket_create_pr", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 553, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1002, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "kind": "verb_product", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "ranking": [ + "gmail.gmail_create_draft", + "gmail.gmail_list_labels", + "gmail.gmail_get_email", + "gmail.gmail_modify_email", + "gmail.gmail_send_email", + "gmail.gmail_trash_email", + "gmail.gmail_search_emails", + "github.create_pull_request", + "github.create_artifact", + "gitlab.create_snippet" + ], + "hit1": "gmail.gmail_create_draft", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 855, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "kind": "verb_product", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_share_file", + "google_drive.gdrive_update_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "filesystem.write_file", + "gitlab.gitlab_get_file", + "figma.figma_get_file", + "slack.get_file" + ], + "hit1": "google_drive.gdrive_share_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 306, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 741, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "kind": "verb_product", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "kubernetes.k8s_get_events", + "sentry.sentry_list_events", + "gitlab.list_milestones", + "clickup.clickup_list_tasks", + "gitlab.list_wikis" + ], + "hit1": "google_calendar.gcal_list_events", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 769, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_update_event", + "github.delete_milestone", + "google_calendar.gcal_list_events", + "gitlab.delete_milestone", + "redis.redis_delete", + "google_calendar.gcal_list_calendars", + "slack.delete_reminder", + "slack.delete_canvase" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 444, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 697, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_projects", + "vercel.vercel_list_deployments", + "vercel.vercel_create_project", + "vercel.vercel_get_deployment", + "jira.jira_list_projects", + "sentry.sentry_list_projects", + "linear.linear_list_projects", + "gitlab.gitlab_list_projects", + "github.projects_list", + "kubernetes.k8s_list_deployments" + ], + "hit1": "vercel.vercel_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 175, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 269, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 539, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "ranking": [ + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update", + "supabase.supabase_invoke_fn", + "supabase.supabase_list_tables", + "datadog.datadog_list_logs", + "figma.figma_get_file_nodes", + "datadog.datadog_query_metrics", + "elasticsearch.es_get_doc", + "sqlite.sqlite_query" + ], + "hit1": "supabase.supabase_select", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 752, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "kind": "verb_product", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "figma.figma_get_file", + "figma.figma_get_comments", + "linear.get_comment", + "linear.create_comment", + "linear.update_comment", + "notion.get_comment", + "figma.figma_get_images", + "notion.create_comment", + "figma.figma_get_file_nodes" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "kind": "verb_product", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "ranking": [ + "okta.okta_list_users", + "okta.okta_get_user", + "okta.okta_create_user", + "okta.okta_deactivate_user", + "notion.notion_list_users", + "notion.list_users_tokens", + "linear.list_users", + "slack.slack_list_users", + "slack.slack_get_users", + "github.search_users" + ], + "hit1": "okta.okta_list_users", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 271, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query_range", + "prometheus.prom_query", + "prometheus.prom_list_metrics", + "datadog.datadog_query_metrics", + "memory.search_nodes", + "github.search_pull_requests", + "datadog.datadog_list_monitors", + "postgres.query", + "salesforce.salesforce_query", + "github.search_commits" + ], + "hit1": "prometheus.prom_query_range", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 661, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "kind": "verb_product", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "ranking": [ + "grafana.grafana_create_annotation", + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "github.create_artifact", + "gitlab.create_snippet", + "github.create_hook", + "gitlab.create_label", + "slack.create_canvase", + "gitlab.create_pipeline", + "linear.create_comment" + ], + "hit1": "grafana.grafana_create_annotation", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 421, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 915, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "kind": "identifier", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "ranking": [ + "terraform.tf_apply", + "terraform.tf_plan", + "terraform.tf_state_list", + "kubernetes.k8s_apply", + "github.update_environment", + "cloudflare.cf_create_dns", + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "playwright.browser_fill_form", + "hubspot.hubspot_create_deal" + ], + "hit1": "terraform.tf_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 653, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "kind": "verb_product", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "ranking": [ + "asana.asana_list_tasks", + "asana.asana_create_task", + "asana.asana_get_task", + "asana.asana_update_task", + "asana.asana_create_project", + "clickup.clickup_list_tasks", + "aws_s3.s3_list_buckets", + "gitlab.list_jobs", + "github.list_milestones", + "kubernetes.k8s_list_services" + ], + "hit1": "asana.asana_list_tasks", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "ranking": [ + "trello.trello_add_comment", + "trello.trello_create_card", + "trello.trello_list_cards", + "trello.trello_create_board", + "linear.get_comment", + "figma.figma_post_comment", + "linear.update_comment", + "linear.create_comment", + "notion.get_comment", + "notion.update_comment" + ], + "hit1": "trello.trello_add_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 721, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_update_task", + "clickup.clickup_get_task", + "clickup.clickup_list_tasks", + "clickup.clickup_create_task", + "puppeteer.puppeteer_click", + "asana.asana_update_task", + "playwright.browser_click", + "gitlab.update_snippet", + "github.update_milestone", + "jira.update_sprint" + ], + "hit1": "clickup.clickup_update_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "kind": "verb_product", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "ranking": [ + "confluence.confluence_search", + "confluence.confluence_get_page", + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "github.search_orgs", + "github.search_users", + "notion.notion_search", + "github.search_issues", + "elasticsearch.es_search", + "grafana.grafana_search_dashboards" + ], + "hit1": "confluence.confluence_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_deal", + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_search_contacts", + "github.create_pull_request", + "github.merge_pull_request", + "github.get_discussion", + "github.pull_request_review_write", + "github.create_gist" + ], + "hit1": "hubspot.hubspot_create_deal", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 927, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_search", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_create_ticket", + "github.list_milestones", + "github.list_packages", + "jira.list_boards", + "github.list_runners", + "jira.list_webhooks" + ], + "hit1": "zendesk.zendesk_list_tickets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "kind": "verb_product", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "ranking": [ + "discord.discord_list_channels", + "discord.discord_get_channel", + "discord.discord_list_members", + "slack.slack_list_channels", + "discord.discord_send_message", + "discord.discord_add_reaction", + "slack.list_usergroups", + "github.list_runners", + "slack.list_pins", + "gitlab.list_wikis" + ], + "hit1": "discord.discord_list_channels", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 166, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "ranking": [ + "redis.redis_set", + "redis.redis_get", + "redis.redis_delete", + "redis.redis_keys", + "playwright.browser_press_key", + "jira.jira_update_issue", + "aws_s3.s3_put_object", + "jira.jira_transition_issue", + "jira.jira_get_issue", + "slack.update_pin" + ], + "hit1": "redis.redis_set", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 621, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert", + "mongodb.mongo_update", + "mongodb.mongo_delete", + "supabase.supabase_insert", + "mongodb.mongo_find", + "mongodb.mongo_list_collections", + "aws_s3.s3_put_object", + "salesforce.salesforce_create_record", + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "kind": "verb_product", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "mysql.mysql_query", + "postgres.explain_query", + "sqlite.sqlite_query", + "salesforce.salesforce_query", + "mysql.mysql_list_tables", + "prometheus.prom_query", + "notion.notion_query_database", + "datadog.datadog_query_metrics", + "prometheus.prom_query_range" + ], + "hit1": "postgres.query", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 678, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_index_doc", + "elasticsearch.es_search", + "elasticsearch.es_get_doc", + "playwright.browser_tab_close", + "mongodb.mongo_find", + "github.create_milestone", + "confluence.confluence_search", + "github.get_milestone", + "google_drive.gdrive_search", + "github.list_milestones" + ], + "hit1": "elasticsearch.es_index_doc", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_put_object", + "aws_s3.s3_list_objects", + "asana.asana_get_task", + "okta.okta_get_user", + "redis.redis_get", + "gitlab.get_snippet", + "jira.get_sprint", + "jira.get_attachment" + ], + "hit1": "aws_s3.s3_get_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones", + "cloudflare.cf_list_dns", + "cloudflare.cf_purge_cache", + "cloudflare.cf_create_dns", + "aws_s3.s3_list_buckets", + "kubernetes.k8s_list_services", + "kubernetes.k8s_list_deployments", + "github.list_environments", + "gitlab.list_pipelines", + "slack.list_usergroups" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 326, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 671, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_create_customer", + "stripe.stripe_create_product", + "stripe.stripe_list_invoices", + "stripe.stripe_create_refund", + "stripe.stripe_create_price", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_customer", + "stripe.stripe_list_customers", + "stripe.stripe_list_subscriptions" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "kind": "verb_product", + "gold": [ + "github.actions_list" + ], + "alts": [], + "ranking": [ + "github.actions_list", + "github.actions_get", + "github.actions_run_trigger", + "github.actions_get_job_logs", + "github.list_notifications", + "github.list_issues", + "github.list_code_scanning_alerts", + "github.list_gists", + "git.git_branch", + "github.list_releases" + ], + "hit1": "github.actions_list", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 452, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 821, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "kind": "verb_product", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "ranking": [ + "github.actions_get_job_logs", + "github.actions_get", + "github.actions_list", + "github.actions_run_trigger", + "git.git_log", + "github.get_notification_details", + "git.git_status", + "github.get_code_scanning_alert", + "github.get_secret_scanning_alert", + "git.git_commit" + ], + "hit1": "github.actions_get_job_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 450, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 813, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "kind": "verb_product", + "gold": [ + "github.create_repository" + ], + "alts": [], + "ranking": [ + "github.create_repository", + "github.star_repository", + "github.create_branch", + "github.fork_repository", + "github.create_gist", + "github.unstar_repository", + "github.create_or_update_file", + "git.git_create_branch", + "github.delete_repository", + "github.create_issue" + ], + "hit1": "github.create_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 927, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "kind": "verb_product", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "ranking": [ + "git.git_checkout", + "git.git_branch", + "git.git_show", + "git.git_create_branch", + "git.git_status", + "github.create_branch", + "git.git_diff", + "github.update_pull_request_branch", + "git.git_log", + "github.list_branches" + ], + "hit1": "git.git_checkout", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 325, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "kind": "identifier", + "gold": [ + "git.git_log" + ], + "alts": [], + "ranking": [ + "git.git_log", + "git.git_show", + "git.git_status", + "git.git_diff", + "git.git_commit", + "git.git_branch", + "git.git_diff_unstaged", + "git.git_checkout", + "git.git_diff_staged", + "git.git_add" + ], + "hit1": "git.git_log", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "kind": "verb_product", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "ranking": [ + "postgres.list_schemas", + "postgres.list_tables", + "postgres.describe_table", + "aws_s3.s3_list_buckets", + "github.list_artifacts", + "kubernetes.k8s_list_deployments", + "postgres.explain_query", + "kubernetes.k8s_list_services", + "github.list_environments", + "github.list_packages" + ], + "hit1": "postgres.list_schemas", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 169, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 287, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "kind": "verb_product", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "ranking": [ + "memory.open_nodes", + "memory.read_graph", + "memory.search_nodes", + "memory.create_entities", + "memory.create_relations", + "memory.delete_relations", + "memory.add_observations", + "memory.delete_entities", + "figma.figma_get_file_nodes", + "memory.delete_observations" + ], + "hit1": "memory.open_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 549, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "kind": "paraphrase", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "ranking": [ + "filesystem.read_file", + "filesystem.get_file_info", + "filesystem.read_media_file", + "filesystem.read_text_file", + "google_drive.gdrive_get_file", + "figma.figma_get_file_nodes", + "filesystem.move_file", + "filesystem.write_file", + "figma.figma_get_file", + "filesystem.list_directory" + ], + "hit1": "filesystem.read_file", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 649, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "kind": "paraphrase", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "ranking": [ + "filesystem.write_file", + "filesystem.edit_file", + "filesystem.read_text_file", + "slack.create_file", + "filesystem.read_file", + "filesystem.read_media_file", + "playwright.browser_tab_new", + "slack.update_file", + "slack.get_file", + "google_drive.gdrive_create_file" + ], + "hit1": "filesystem.write_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 423, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "kind": "verb_product", + "gold": [ + "github.search_issues" + ], + "alts": [], + "ranking": [ + "github.search_issues", + "github.list_issues", + "github.search_users", + "github.search_orgs", + "github.search_commits", + "github.search_pull_requests", + "github.search_code", + "github.search_repositories", + "linear.linear_search_issues", + "jira.jira_search_issues" + ], + "hit1": "github.search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "kind": "paraphrase", + "gold": [ + "github.get_me" + ], + "alts": [], + "ranking": [ + "github.get_teams", + "github.get_me", + "github.list_notifications", + "slack.slack_get_user_profile", + "github.list_gists", + "github.get_latest_release", + "github.projects_get", + "github.get_secret_scanning_alert", + "github.get_code_scanning_alert", + "github.get_label" + ], + "hit1": "github.get_teams", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 285, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 700, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "cost": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.08, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 255.36, + "p95_tokens": 317.9499999999998, + "total_tokens": 38304, + "tokens_per_successful_query": 255.36, + "total_tokens_successful": 38304 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.0466666666666666, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 397.26, + "p95_tokens": 492.3499999999997, + "total_tokens": 59589, + "tokens_per_successful_query": 397.26, + "total_tokens_successful": 59589 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.02, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 758.62, + "p95_tokens": 1008.0499999999997, + "total_tokens": 113793, + "tokens_per_successful_query": 758.62, + "total_tokens_successful": 113793 + } + } +} diff --git a/experiments/mcp-tool-search/results/hybrid_rrf_weighted_hashed.csv b/experiments/mcp-tool-search/results/hybrid_rrf_weighted_hashed.csv new file mode 100644 index 00000000..490bce78 --- /dev/null +++ b/experiments/mcp-tool-search/results/hybrid_rrf_weighted_hashed.csv @@ -0,0 +1,151 @@ +id,query,kind,hit1,recall@1,recall@3,recall@5,mrr@10,ndcg@10,hit_rank,k3_calls,k3_tokens,k3_success1,k5_calls,k5_tokens,k5_success1,k10_calls,k10_tokens,k10_success1,ranking +q001,Create a GitHub issue,verb_product,github.issue_write,1.0,1.0,1.0,1.0000,0.7602,1,1,388,1.0,1,598,1.0,1,1024,1.0,github.issue_write github.issue_read github.create_issue github.update_issue_comment github.add_issue_comment github.create_gist gitlab.gitlab_create_issue github.assign_copilot_to_issue gitlab.gitlab_get_issue github.create_repository +q002,Create a Jira ticket,verb_product,jira.jira_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,312,1.0,1,398,1.0,1,854,1.0,jira.jira_create_issue jira.jira_create_sprint jira.jira_assign_issue jira.jira_get_issue jira.jira_list_projects jira.create_board jira.create_webhook jira.create_filter jira.create_sprint jira.jira_update_issue +q003,Send a Slack message,verb_product,slack.slack_post_message,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,373,1.0,1,737,1.0,slack.slack_post_message slack.slack_add_reaction slack.slack_search_messages slack.slack_join_channel slack.slack_create_channel slack.slack_leave_channel slack.slack_get_channel_history slack.slack_upload_file discord.discord_send_message slack.slack_reply_to_thread +q004,List open pull requests,resource,github.list_pull_requests,1.0,1.0,1.0,1.0000,0.9502,1,1,311,1.0,1,506,1.0,1,1060,1.0,github.list_pull_requests github.create_pull_request github.search_pull_requests github.merge_pull_request github.pull_request_read gitlab.gitlab_list_merge_requests github.request_copilot_review github.pull_request_review_write github.update_pull_request github.update_pull_request_branch +q005,Open a pull request on GitHub,verb_product,github.create_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,379,1.0,1,522,1.0,1,1144,1.0,github.create_pull_request github.merge_pull_request github.request_copilot_review github.pull_request_read github.search_pull_requests github.update_pull_request github.pull_request_review_write github.list_pull_requests github.update_pull_request_branch github.add_comment_to_pending_review +q006,Run a SQL query against postgres,verb_product,postgres.query,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,272,1.0,1,590,1.0,postgres.query postgres.explain_query mysql.mysql_query postgres.list_schemas sqlite.sqlite_query salesforce.salesforce_query postgres.describe_table postgres.list_tables prometheus.prom_query docker.docker_run +q007,Read a local file from disk,paraphrase,filesystem.read_file,1.0,1.0,1.0,1.0000,0.9502,1,1,223,1.0,1,385,1.0,1,745,1.0,filesystem.read_file filesystem.read_media_file filesystem.read_text_file filesystem.edit_file filesystem.write_file gitlab.gitlab_get_file slack.get_file figma.figma_get_file filesystem.get_file_info github.get_file_contents +q008,What files are in this folder,paraphrase,google_drive.gdrive_list_folder,0.0,0.0,0.0,0.1667,0.4275,6,2,478,0.0,2,719,0.0,1,697,1.0,google_drive.gdrive_list_folder github.push_files slack.list_files filesystem.move_file filesystem.search_files filesystem.list_directory_with_sizes filesystem.list_directory filesystem.directory_tree filesystem.list_allowed_directories filesystem.edit_file +q009,Remember this fact in the knowledge graph,paraphrase,memory.read_graph,0.0,1.0,1.0,0.5000,0.5672,2,1,155,1.0,1,260,1.0,1,510,1.0,memory.read_graph memory.create_entities memory.open_nodes memory.add_observations memory.delete_relations memory.search_nodes memory.create_relations memory.delete_entities sqlite.sqlite_list_tables github.get_me +q010,Search the web with Brave,verb_product,brave_search.brave_web_search,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,365,1.0,1,699,1.0,brave_search.brave_web_search brave_search.brave_local_search jira.jira_search_issues memory.search_nodes github.search_users elasticsearch.es_search github.search_orgs github.search_issues puppeteer.puppeteer_screenshot github.search_commits +q011,Fetch the contents of a URL,verb_product,fetch.fetch,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,389,1.0,1,681,1.0,fetch.fetch filesystem.read_file playwright.browser_navigate github.get_file_contents playwright.browser_take_screenshot git.git_add postgres.describe_table puppeteer.puppeteer_navigate github.get_me git.git_status +q012,Show git status,identifier,git.git_status,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,293,1.0,1,584,1.0,git.git_status git.git_show git.git_diff git.git_branch git.git_diff_staged git.git_diff_unstaged git.git_log git.git_checkout git.git_add git.git_reset +q013,Commit staged changes,paraphrase,git.git_commit,1.0,1.0,1.0,1.0000,1.0000,1,1,215,1.0,1,327,1.0,1,702,1.0,git.git_commit git.git_reset github.get_commit git.git_diff_staged git.git_show github.push_files git.git_log git.git_diff_unstaged filesystem.read_media_file filesystem.read_text_file +q014,Create a Linear issue,verb_product,linear.linear_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,333,1.0,1,463,1.0,1,875,1.0,linear.linear_create_issue linear.linear_update_issue linear.linear_create_comment linear.linear_get_issue linear.linear_assign_issue linear.linear_create_project linear.linear_list_issues linear.linear_search_issues linear.create_label linear.create_cycle +q015,Make a task in my project tracker Linear,paraphrase,linear.linear_create_project,0.0,1.0,1.0,0.3333,0.5000,3,1,284,1.0,1,415,1.0,1,758,1.0,linear.linear_create_project linear.linear_list_projects linear.linear_create_issue linear.list_cycles vercel.vercel_create_project linear.linear_create_comment linear.list_users linear.linear_search_issues linear.linear_list_teams linear.linear_update_issue +q016,Post a comment on a GitHub issue,verb_product,github.update_issue_comment,0.0,1.0,1.0,0.5000,0.6309,2,1,333,1.0,1,542,1.0,1,1063,1.0,github.update_issue_comment github.add_issue_comment github.issue_read github.issue_write figma.figma_post_comment github.add_comment_to_pending_review github.create_issue github.list_commits linear.linear_create_comment linear.update_comment +q017,Merge the pull request,ambiguous,github.merge_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,283,1.0,1,523,1.0,1,1031,1.0,github.merge_pull_request gitlab.gitlab_merge_merge_request gitlab.gitlab_get_merge_request github.pull_request_read github.create_pull_request github.request_copilot_review github.update_pull_request_branch github.search_pull_requests github.update_pull_request gitlab.gitlab_create_merge_request +q018,Trigger a GitHub Actions workflow,verb_product,github.actions_run_trigger,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,470,1.0,1,835,1.0,github.actions_run_trigger github.actions_get github.actions_list github.actions_get_job_logs github.fork_repository github.star_repository github.list_tags github.list_branches github.unstar_repository github.get_tag +q019,Get Dependabot alerts,resource,github.list_dependabot_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,287,1.0,1,418,1.0,1,730,1.0,github.list_dependabot_alerts github.get_dependabot_alert github.list_code_scanning_alerts github.list_secret_scanning_alerts jira.get_board jira.get_attachment gitlab.get_job jira.get_webhook sentry.sentry_get_issue github.get_commit +q020,List Sentry issues,verb_product,sentry.sentry_list_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,396,1.0,1,813,1.0,sentry.sentry_list_issues sentry.sentry_list_events sentry.sentry_list_projects sentry.sentry_get_issue sentry.sentry_update_issue github.list_issues gitlab.gitlab_list_issues linear.linear_list_issues github.search_issues jira.jira_search_issues +q021,Create a Stripe customer,verb_product,stripe.stripe_get_customer,0.0,1.0,1.0,0.5000,0.6309,2,1,186,1.0,1,315,1.0,1,669,1.0,stripe.stripe_get_customer stripe.stripe_create_customer stripe.stripe_create_invoice stripe.stripe_create_price stripe.stripe_list_invoices stripe.stripe_create_payment_intent stripe.stripe_create_product stripe.stripe_list_subscriptions stripe.stripe_create_refund stripe.stripe_list_customers +q022,Refund a Stripe payment,paraphrase,stripe.stripe_create_refund,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,352,1.0,1,662,1.0,stripe.stripe_create_refund stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_create_invoice stripe.stripe_list_customers stripe.stripe_create_price stripe.stripe_list_invoices stripe.stripe_create_product +q023,Send an email with Gmail,verb_product,gmail.gmail_send_email,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,328,1.0,1,735,1.0,gmail.gmail_send_email gmail.gmail_get_email gmail.gmail_search_emails gmail.gmail_trash_email gmail.gmail_list_labels gmail.gmail_modify_email gmail.gmail_create_draft filesystem.list_directory_with_sizes twilio.twilio_send_sms google_drive.gdrive_share_file +q024,Search my Gmail inbox,verb_product,gmail.gmail_search_emails,1.0,1.0,1.0,1.0000,1.0000,1,1,265,1.0,1,375,1.0,1,726,1.0,gmail.gmail_search_emails gmail.gmail_send_email gmail.gmail_create_draft gmail.gmail_get_email gmail.gmail_trash_email gmail.gmail_list_labels brave_search.brave_web_search gmail.gmail_modify_email elasticsearch.es_search notion.notion_search +q025,Upload a file to Google Drive,paraphrase,google_drive.gdrive_update_file,0.0,1.0,1.0,0.5000,0.6309,2,1,307,1.0,1,418,1.0,1,794,1.0,google_drive.gdrive_update_file google_drive.gdrive_create_file google_drive.gdrive_share_file google_drive.gdrive_get_file google_drive.gdrive_search google_drive.gdrive_list_folder slack.slack_upload_file gitlab.gitlab_get_file filesystem.edit_file figma.figma_get_file +q026,Create a calendar event,resource,google_calendar.gcal_create_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,427,1.0,1,885,1.0,google_calendar.gcal_create_event google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars github.create_gist jira.create_attachment github.create_environment linear.create_comment jira.create_board +q027,List Kubernetes pods,verb_product,kubernetes.k8s_list_pods,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,314,1.0,1,718,1.0,kubernetes.k8s_list_pods kubernetes.k8s_list_services kubernetes.k8s_list_deployments kubernetes.k8s_get_pod kubernetes.k8s_get_events kubernetes.k8s_apply kubernetes.k8s_pod_logs kubernetes.k8s_delete kubernetes.k8s_scale_deployment clickup.clickup_list_tasks +q028,Get logs from a k8s pod,verb_product,kubernetes.k8s_pod_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,275,1.0,1,447,1.0,1,911,1.0,kubernetes.k8s_pod_logs docker.docker_logs gitlab.gitlab_get_file github.actions_get_job_logs kubernetes.k8s_get_pod github.get_tag elasticsearch.es_get_doc github.get_file_contents figma.figma_get_images github.delete_file +q029,Run a docker container,verb_product,docker.docker_stop,0.0,1.0,1.0,0.5000,0.6309,2,1,223,1.0,1,345,1.0,1,640,1.0,docker.docker_stop docker.docker_run docker.docker_exec docker.docker_rm docker.docker_logs docker.docker_images docker.docker_ps docker.docker_build mysql.mysql_query kubernetes.k8s_pod_logs +q030,Show docker container logs,verb_product,docker.docker_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,345,1.0,1,649,1.0,docker.docker_logs docker.docker_stop docker.docker_rm docker.docker_exec docker.docker_run docker.docker_ps docker.docker_images docker.docker_build git.git_log kubernetes.k8s_pod_logs +q031,Navigate the browser to a URL,ambiguous,playwright.browser_navigate,1.0,1.0,1.0,1.0000,0.9502,1,1,180,1.0,1,265,1.0,1,550,1.0,playwright.browser_navigate playwright.browser_tab_new puppeteer.puppeteer_navigate playwright.browser_tab_close playwright.browser_tab_list playwright.browser_evaluate playwright.browser_wait_for playwright.browser_type playwright.browser_press_key playwright.browser_snapshot +q032,Take a screenshot of the page,ambiguous,playwright.browser_take_screenshot,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,338,1.0,1,671,1.0,playwright.browser_take_screenshot puppeteer.puppeteer_screenshot notion.notion_update_page playwright.browser_snapshot filesystem.read_file playwright.browser_press_key notion.notion_get_page confluence.confluence_update_page playwright.browser_evaluate notion.notion_create_page +q033,Create a GitLab merge request,verb_product,gitlab.gitlab_merge_merge_request,0.0,1.0,1.0,0.5000,0.6309,2,1,297,1.0,1,475,1.0,1,961,1.0,gitlab.gitlab_merge_merge_request gitlab.gitlab_create_merge_request gitlab.gitlab_get_merge_request github.merge_pull_request gitlab.gitlab_list_merge_requests github.request_copilot_review gitlab.gitlab_create_issue gitlab.gitlab_create_pipeline github.create_pull_request gitlab.gitlab_get_issue +q034,Search Notion pages,verb_product,notion.notion_search,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,336,1.0,1,704,1.0,notion.notion_search notion.notion_get_page notion.notion_update_page notion.notion_create_page notion.notion_list_users notion.notion_create_database notion.notion_get_comments notion.notion_get_block notion.notion_query_database brave_search.brave_web_search +q035,Create a Notion page,verb_product,notion.notion_get_page,0.0,1.0,1.0,0.5000,0.6309,2,1,230,1.0,1,371,1.0,1,796,1.0,notion.notion_get_page notion.notion_create_page notion.notion_update_page notion.notion_create_database notion.notion_get_comments notion.notion_search notion.create_comment notion.create_data_source notion.notion_get_block notion.create_users_token +q036,Query a Notion database,verb_product,notion.notion_query_database,1.0,1.0,1.0,1.0000,1.0000,1,1,261,1.0,1,367,1.0,1,674,1.0,notion.notion_query_database notion.notion_create_database notion.notion_search notion.notion_get_block notion.notion_get_page notion.notion_create_page notion.notion_list_users notion.notion_get_comments notion.get_data_source notion.notion_append_block +q037,Create a Zendesk ticket,verb_product,zendesk.zendesk_create_ticket,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,368,1.0,1,856,1.0,zendesk.zendesk_create_ticket zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_search zendesk.zendesk_list_tickets jira.jira_create_issue github.create_gist jira.create_sprint github.create_hook github.create_artifact +q038,Create a HubSpot contact,verb_product,hubspot.hubspot_create_contact,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,877,1.0,hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_create_deal hubspot.hubspot_search_contacts hubspot.hubspot_list_contacts github.create_gist github.create_or_update_file github.create_artifact github.create_package github.create_hook +q039,Create a Salesforce lead,verb_product,salesforce.salesforce_create_lead,1.0,1.0,1.0,1.0000,1.0000,1,1,252,1.0,1,402,1.0,1,782,1.0,salesforce.salesforce_create_lead salesforce.salesforce_convert_lead salesforce.salesforce_create_record salesforce.salesforce_update_record salesforce.salesforce_query salesforce.salesforce_delete_record salesforce.salesforce_get_record salesforce.salesforce_list_opportunities github.create_gist github.create_artifact +q040,Run a SOQL query,identifier,salesforce.salesforce_query,1.0,1.0,1.0,1.0000,1.0000,1,1,187,1.0,1,291,1.0,1,672,1.0,salesforce.salesforce_query postgres.query mysql.mysql_query sqlite.sqlite_query prometheus.prom_query postgres.explain_query memory.search_nodes github.update_check_run datadog.datadog_query_metrics prometheus.prom_query_range +q041,List Shopify orders,verb_product,shopify.shopify_list_orders,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,320,1.0,1,695,1.0,shopify.shopify_list_orders shopify.shopify_list_products shopify.shopify_list_customers shopify.shopify_get_order shopify.shopify_get_product shopify.shopify_create_order shopify.shopify_create_product shopify.shopify_update_product github.projects_list trello.trello_list_cards +q042,Create a PagerDuty incident,verb_product,pagerduty.pagerduty_create_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,403,1.0,1,836,1.0,pagerduty.pagerduty_create_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls github.create_gist github.create_package github.create_hook gitlab.create_wiki +q043,Who is on call in PagerDuty,paraphrase,pagerduty.pagerduty_list_oncalls,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,401,1.0,1,771,1.0,pagerduty.pagerduty_list_oncalls pagerduty.pagerduty_get_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_create_incident pagerduty.pagerduty_resolve_incident twilio.twilio_make_call notion.notion_get_comments memory.search_nodes github.assign_copilot_to_issue +q044,Query Datadog metrics,verb_product,datadog.datadog_query_metrics,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,458,1.0,1,797,1.0,datadog.datadog_query_metrics datadog.datadog_list_monitors datadog.datadog_create_monitor datadog.datadog_list_logs datadog.datadog_get_monitor prometheus.prom_query prometheus.prom_list_metrics notion.notion_query_database mysql.mysql_query prometheus.prom_query_range +q045,Search Datadog logs,verb_product,datadog.datadog_list_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,423,1.0,1,784,1.0,datadog.datadog_list_logs datadog.datadog_create_monitor datadog.datadog_get_monitor datadog.datadog_list_monitors elasticsearch.es_search brave_search.brave_web_search github.search_orgs datadog.datadog_query_metrics notion.notion_search github.search_users +q046,List S3 buckets,verb_product,aws_s3.s3_list_buckets,1.0,1.0,1.0,1.0000,1.0000,1,1,206,1.0,1,363,1.0,1,699,1.0,aws_s3.s3_list_buckets aws_s3.s3_list_objects github.list_packages gitlab.list_snippets github.list_environments github.list_runners github.list_hooks github.list_artifacts kubernetes.k8s_list_deployments clickup.clickup_list_tasks +q047,Upload an object to S3,paraphrase,aws_s3.s3_put_object,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,409,1.0,1,866,1.0,aws_s3.s3_put_object aws_s3.s3_delete_object aws_s3.s3_get_object aws_s3.s3_list_objects twilio.twilio_send_sms gmail.gmail_send_email slack.slack_upload_file asana.asana_create_project okta.okta_deactivate_user github.add_comment_to_pending_review +q048,Purge Cloudflare cache,verb_product,cloudflare.cf_purge_cache,1.0,1.0,1.0,1.0000,1.0000,1,1,231,1.0,1,338,1.0,1,732,1.0,cloudflare.cf_purge_cache cloudflare.cf_list_zones cloudflare.cf_create_dns cloudflare.cf_list_dns github.search_orgs gmail.gmail_search_emails slack.list_reminders gitlab.get_release gitlab.gitlab_create_merge_request gitlab.gitlab_list_merge_requests +q049,Create a Cloudflare DNS record,verb_product,cloudflare.cf_create_dns,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,356,1.0,1,819,1.0,cloudflare.cf_create_dns cloudflare.cf_list_dns cloudflare.cf_list_zones cloudflare.cf_purge_cache salesforce.salesforce_create_record github.create_gist jira.create_attachment jira.create_board github.create_hook jira.create_sprint +q050,List Vercel deployments,verb_product,vercel.vercel_list_deployments,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,291,1.0,1,615,1.0,vercel.vercel_list_deployments vercel.vercel_get_deployment vercel.vercel_list_projects kubernetes.k8s_list_deployments vercel.vercel_create_project clickup.clickup_list_tasks github.list_environments aws_s3.s3_list_buckets trello.trello_list_cards jira.list_attachments +q051,Insert a row into Supabase,verb_product,supabase.supabase_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,355,1.0,1,767,1.0,supabase.supabase_insert supabase.supabase_select supabase.supabase_update supabase.supabase_invoke_fn supabase.supabase_list_tables mongodb.mongo_insert playwright.browser_type jira.jira_create_issue jira.get_version jira.create_version +q052,Export images from Figma,paraphrase,figma.figma_get_images,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,328,1.0,1,660,1.0,figma.figma_get_images figma.figma_get_file_nodes docker.docker_images figma.figma_get_file figma.figma_get_comments figma.figma_post_comment gitlab.gitlab_get_file memory.delete_entities supabase.supabase_select memory.delete_relations +q053,Send an SMS with Twilio,verb_product,twilio.twilio_send_sms,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,398,1.0,1,748,1.0,twilio.twilio_send_sms twilio.twilio_list_messages twilio.twilio_make_call gmail.gmail_send_email okta.okta_deactivate_user discord.discord_send_message github.update_pull_request_branch filesystem.list_directory_with_sizes okta.okta_create_user playwright.browser_type +q054,Create an Okta user,verb_product,okta.okta_get_user,0.0,1.0,1.0,0.5000,0.6309,2,1,183,1.0,1,329,1.0,1,827,1.0,okta.okta_get_user okta.okta_create_user okta.okta_deactivate_user okta.okta_list_users linear.create_user asana.asana_create_project asana.asana_create_task github.create_issue github.create_environment github.create_hook +q055,Instant query Prometheus,verb_product,prometheus.prom_query,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,329,1.0,1,652,1.0,prometheus.prom_query prometheus.prom_query_range prometheus.prom_list_metrics postgres.query postgres.explain_query datadog.datadog_query_metrics salesforce.salesforce_query sqlite.sqlite_query mysql.mysql_query memory.search_nodes +q056,Search Grafana dashboards,verb_product,grafana.grafana_search_dashboards,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,382,1.0,1,705,1.0,grafana.grafana_search_dashboards grafana.grafana_get_dashboard grafana.grafana_create_annotation brave_search.brave_local_search brave_search.brave_web_search elasticsearch.es_search notion.notion_search github.search_orgs github.search_users github.search_issues +q057,Trigger a CircleCI pipeline,verb_product,circleci.circleci_trigger_pipeline,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,325,1.0,1,755,1.0,circleci.circleci_trigger_pipeline circleci.circleci_get_pipeline circleci.circleci_list_pipelines gitlab.gitlab_create_pipeline gitlab.delete_pipeline gitlab.create_pipeline gitlab.update_pipeline gitlab.get_pipeline hubspot.hubspot_create_deal github.actions_run_trigger +q058,Run terraform plan,identifier,terraform.tf_plan,1.0,1.0,1.0,1.0000,1.0000,1,1,172,1.0,1,287,1.0,1,618,1.0,terraform.tf_plan terraform.tf_apply terraform.tf_state_list postgres.explain_query salesforce.salesforce_query postgres.query docker.docker_exec playwright.browser_fill_form docker.docker_run github.list_releases +q059,Create an Asana task,verb_product,asana.asana_get_task,0.0,1.0,1.0,0.5000,0.6309,2,1,267,1.0,1,405,1.0,1,853,1.0,asana.asana_get_task asana.asana_create_task asana.asana_update_task asana.asana_create_project asana.asana_list_tasks clickup.clickup_create_task okta.okta_create_user github.create_gist github.create_issue github.create_package +q060,Create a Trello card,verb_product,trello.trello_create_card,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,394,1.0,1,874,1.0,trello.trello_create_card trello.trello_add_comment trello.trello_create_board trello.trello_list_cards github.create_gist jira.create_board github.create_runner github.create_package github.create_artifact jira.create_filter +q061,Create a ClickUp task,verb_product,clickup.clickup_get_task,0.0,1.0,1.0,0.5000,0.6309,2,1,257,1.0,1,410,1.0,1,871,1.0,clickup.clickup_get_task clickup.clickup_create_task clickup.clickup_update_task clickup.clickup_list_tasks github.create_artifact github.create_gist github.create_runner gitlab.create_release github.create_package jira.create_attachment +q062,Create a Confluence page,verb_product,confluence.confluence_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,401,1.0,1,876,1.0,confluence.confluence_create_page confluence.confluence_get_page confluence.confluence_update_page confluence.confluence_search notion.notion_create_page jira.create_board jira.create_attachment jira.create_filter github.create_package jira.create_version +q063,Send a Discord message,verb_product,discord.discord_send_message,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,322,1.0,1,698,1.0,discord.discord_send_message discord.discord_add_reaction discord.discord_get_channel discord.discord_list_channels discord.discord_list_members slack.slack_post_message git.git_commit datadog.datadog_create_monitor figma.figma_post_comment gmail.gmail_get_email +q064,Search Elasticsearch,verb_product,elasticsearch.es_search,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,404,1.0,1,686,1.0,elasticsearch.es_search elasticsearch.es_index_doc elasticsearch.es_get_doc brave_search.brave_web_search brave_search.brave_local_search github.search_users github.search_orgs notion.notion_search confluence.confluence_search zendesk.zendesk_search +q065,Get a Redis key,verb_product,redis.redis_get,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,312,1.0,1,631,1.0,redis.redis_get redis.redis_set redis.redis_delete redis.redis_keys github.get_label github.get_discussion aws_s3.s3_get_object github.get_artifact figma.figma_get_file gitlab.get_label +q066,Find MongoDB documents,verb_product,mongodb.mongo_find,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,381,1.0,1,788,1.0,mongodb.mongo_find mongodb.mongo_insert mongodb.mongo_update mongodb.mongo_delete mongodb.mongo_list_collections elasticsearch.es_index_doc github.list_environments zendesk.zendesk_create_ticket elasticsearch.es_get_doc stripe.stripe_create_payment_intent +q067,List SQLite tables,verb_product,sqlite.sqlite_list_tables,1.0,1.0,1.0,1.0000,1.0000,1,1,151,1.0,1,252,1.0,1,567,1.0,sqlite.sqlite_list_tables sqlite.sqlite_describe_table mysql.mysql_list_tables postgres.list_tables sqlite.sqlite_query supabase.supabase_list_tables github.list_milestones gitlab.list_milestones postgres.list_schemas github.list_hooks +q068,Describe a postgres table,verb_product,postgres.describe_table,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,272,1.0,1,643,1.0,postgres.describe_table sqlite.sqlite_describe_table postgres.query postgres.list_tables postgres.list_schemas postgres.explain_query supabase.supabase_select hubspot.hubspot_create_deal supabase.supabase_insert confluence.confluence_update_page +q069,github_create_issue,identifier,github.create_issue,1.0,1.0,1.0,1.0000,0.9072,1,1,361,1.0,1,588,1.0,1,1092,1.0,github.create_issue github.update_issue_comment github.add_issue_comment gitlab.gitlab_create_issue github.issue_write github.issue_read github.assign_copilot_to_issue bitbucket.bitbucket_create_issue jira.jira_create_issue github.create_hook +q070,createIssue,identifier,gitlab.gitlab_create_issue,0.0,1.0,1.0,0.3333,0.5531,3,1,332,1.0,1,520,1.0,1,988,1.0,gitlab.gitlab_create_issue bitbucket.bitbucket_create_issue github.create_issue jira.jira_create_issue gitlab.gitlab_get_issue github.issue_write sentry.sentry_get_issue jira.jira_get_issue linear.linear_create_issue github.create_gist +q071,list-directory,identifier,filesystem.create_directory,0.0,1.0,1.0,0.5000,0.6309,2,1,186,1.0,1,281,1.0,1,646,1.0,filesystem.create_directory filesystem.list_directory filesystem.list_directory_with_sizes filesystem.directory_tree filesystem.list_allowed_directories filesystem.get_file_info filesystem.move_file github.get_file_contents trello.trello_list_cards github.projects_list +q072,sequential thinking next thought,paraphrase,sequential_thinking.sequentialthinking,1.0,1.0,1.0,1.0000,1.0000,1,1,302,1.0,1,430,1.0,1,746,1.0,sequential_thinking.sequentialthinking filesystem.edit_file kubernetes.k8s_get_pod filesystem.list_allowed_directories github.list_pull_requests stripe.stripe_list_subscriptions playwright.browser_wait_for kubernetes.k8s_get_events notion.list_data_sources git.git_status +q073,What time is it in Tokyo,paraphrase,time.convert_time,1.0,1.0,1.0,1.0000,0.8597,1,1,217,1.0,1,380,1.0,1,674,1.0,time.convert_time time.get_current_time playwright.browser_wait_for jira.jira_add_worklog github.list_hooks github.delete_milestone gitlab.get_pipeline gitlab.list_pipelines github.delete_environment gitlab.delete_pipeline +q074,Convert time between timezones,verb_product,time.convert_time,1.0,1.0,1.0,1.0000,1.0000,1,1,213,1.0,1,369,1.0,1,743,1.0,time.convert_time time.get_current_time git.git_diff playwright.browser_wait_for google_calendar.gcal_list_events salesforce.salesforce_convert_lead docker.docker_logs notion.create_users_token memory.create_relations slack.create_usergroup +q075,Star a GitHub repo,verb_product,github.star_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,226,1.0,1,393,1.0,1,792,1.0,github.star_repository github.unstar_repository github.delete_repository github.get_tag github.fork_repository github.list_tags github.list_releases github.list_branches github.list_issues github.get_label +q076,Fork a repository,resource,github.fork_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,235,1.0,1,386,1.0,1,792,1.0,github.fork_repository github.delete_repository github.star_repository github.unstar_repository github.create_repository github.list_repository_collaborators github.get_repository_tree github.list_releases github.list_dependabot_alerts github.list_branches +q077,Get the latest GitHub release,verb_product,github.get_latest_release,1.0,1.0,1.0,1.0000,1.0000,1,1,268,1.0,1,412,1.0,1,709,1.0,github.get_latest_release github.get_release_by_tag github.get_label github.get_repository_tree github.get_teams gitlab.get_release github.get_gist github.get_me github.get_commit github.search_commits +q078,Create a gist,resource,github.create_gist,1.0,1.0,1.0,1.0000,1.0000,1,1,272,1.0,1,480,1.0,1,925,1.0,github.create_gist github.update_gist github.create_package slack.create_canvase github.create_artifact gitlab.create_wiki github.get_gist github.create_hook gitlab.create_pipeline slack.create_pin +q079,Search GitHub code,verb_product,github.search_code,1.0,1.0,1.0,1.0000,1.0000,1,1,283,1.0,1,385,1.0,1,748,1.0,github.search_code github.get_code_scanning_alert github.list_code_scanning_alerts github.search_users github.search_orgs github.search_issues github.search_commits github.search_repositories github.search_pull_requests github.update_gist +q080,List GitHub notifications,verb_product,github.list_notifications,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,311,1.0,1,687,1.0,github.list_notifications github.mark_all_notifications_read github.actions_list github.list_gists github.list_tags github.list_releases github.list_discussions github.list_issues github.search_orgs github.create_gist +q081,Assign Copilot to an issue,verb_product,github.assign_copilot_to_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,464,1.0,1,838,1.0,github.assign_copilot_to_issue jira.jira_assign_issue jira.jira_update_issue jira.jira_transition_issue github.issue_write gitlab.gitlab_get_issue github.add_issue_comment linear.linear_assign_issue jira.jira_get_issue sentry.sentry_get_issue +q082,Get secret scanning alerts,resource,github.list_secret_scanning_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,295,1.0,1,446,1.0,1,828,1.0,github.list_secret_scanning_alerts github.get_secret_scanning_alert github.list_code_scanning_alerts github.get_code_scanning_alert gitlab.get_release aws_s3.s3_get_object github.get_dependabot_alert github.actions_get shopify.shopify_get_product github.list_dependabot_alerts +q083,Create or update a file on GitHub,verb_product,github.create_or_update_file,1.0,1.0,1.0,1.0000,0.7602,1,1,316,1.0,1,478,1.0,1,1038,1.0,github.create_or_update_file slack.update_file github.update_gist github.create_gist github.projects_write slack.create_file github.update_pull_request github.update_issue_comment github.issue_write github.create_repository +q084,List collaborators on a repo,resource,github.list_repository_collaborators,1.0,1.0,1.0,1.0000,1.0000,1,1,278,1.0,1,420,1.0,1,893,1.0,github.list_repository_collaborators github.list_commits trello.trello_list_cards github.list_tags github.list_releases github.list_labels github.list_branches github.list_issues github.list_secret_scanning_alerts github.create_pull_request +q085,Reply in a Slack thread,verb_product,slack.slack_reply_to_thread,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,363,1.0,1,728,1.0,slack.slack_reply_to_thread slack.slack_get_thread_replies slack.slack_leave_channel slack.slack_join_channel slack.slack_create_channel slack.slack_get_user_profile slack.slack_add_reaction slack.slack_list_channels slack.slack_upload_file slack.slack_post_message +q086,Add an emoji reaction on Slack,verb_product,slack.slack_add_reaction,1.0,1.0,1.0,1.0000,0.9239,1,1,273,1.0,1,413,1.0,1,835,1.0,slack.slack_add_reaction slack.slack_reply_to_thread slack.slack_create_channel discord.discord_add_reaction slack.slack_leave_channel slack.slack_upload_file slack.slack_join_channel slack.slack_get_thread_replies slack.create_reminder slack.create_canvase +q087,Search Slack messages,verb_product,slack.slack_search_messages,1.0,1.0,1.0,1.0000,1.0000,1,1,231,1.0,1,386,1.0,1,696,1.0,slack.slack_search_messages slack.slack_get_channel_history slack.slack_post_message slack.slack_list_users slack.slack_add_reaction slack.slack_list_channels slack.slack_join_channel slack.slack_leave_channel slack.slack_get_users slack.slack_create_channel +q088,Create a Slack channel,verb_product,slack.slack_join_channel,0.0,1.0,1.0,0.5000,0.6309,2,1,196,1.0,1,339,1.0,1,777,1.0,slack.slack_join_channel slack.slack_create_channel slack.slack_leave_channel slack.slack_get_channel_history slack.slack_post_message slack.slack_set_channel_topic slack.slack_add_reaction slack.slack_reply_to_thread slack.slack_upload_file slack.create_usergroup +q089,Transition a Jira issue,verb_product,jira.jira_transition_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,389,1.0,1,763,1.0,jira.jira_transition_issue jira.jira_assign_issue jira.jira_get_issue jira.jira_create_issue jira.jira_update_issue jira.jira_add_comment jira.jira_add_worklog jira.jira_search_issues jira.jira_list_projects jira.jira_create_sprint +q090,Search Jira with JQL,identifier,jira.jira_search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,394,1.0,1,806,1.0,jira.jira_search_issues jira.jira_list_projects jira.jira_create_sprint jira.jira_assign_issue jira.jira_add_worklog jira.jira_update_issue jira.jira_create_issue jira.create_sprint jira.list_sprints jira.get_sprint +q091,Add a worklog to Jira,verb_product,jira.jira_add_worklog,1.0,1.0,1.0,1.0000,1.0000,1,1,214,1.0,1,391,1.0,1,791,1.0,jira.jira_add_worklog jira.jira_add_comment jira.jira_list_projects jira.jira_create_sprint jira.jira_assign_issue jira.jira_create_issue jira.jira_update_issue jira.jira_search_issues jira.list_sprints jira.jira_transition_issue +q092,List Linear teams,verb_product,linear.linear_list_teams,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,398,1.0,1,722,1.0,linear.linear_list_teams linear.linear_list_projects linear.linear_list_issues linear.linear_create_issue linear.linear_create_project linear.list_labels linear.list_users linear.linear_create_comment linear.linear_search_issues linear.linear_get_issue +q093,Comment on a Linear issue,verb_product,linear.linear_create_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,441,1.0,1,835,1.0,linear.linear_create_comment linear.linear_update_issue linear.linear_get_issue linear.get_comment linear.linear_create_issue linear.update_comment linear.linear_assign_issue github.update_issue_comment linear.linear_search_issues linear.create_comment +q094,Move a local file,paraphrase,filesystem.move_file,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,352,1.0,1,756,1.0,filesystem.move_file filesystem.read_file filesystem.write_file filesystem.read_media_file filesystem.read_text_file figma.figma_get_file slack.create_file google_drive.gdrive_share_file google_drive.gdrive_update_file brave_search.brave_local_search +q095,Recursively search files by pattern,paraphrase,filesystem.search_files,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,335,1.0,1,690,1.0,filesystem.search_files slack.list_files filesystem.list_directory filesystem.directory_tree filesystem.move_file filesystem.list_directory_with_sizes github.push_files google_drive.gdrive_search filesystem.read_file notion.notion_search +q096,Edit a file in place with old and new string,paraphrase,filesystem.edit_file,1.0,1.0,1.0,1.0000,1.0000,1,1,247,1.0,1,422,1.0,1,814,1.0,filesystem.edit_file filesystem.write_file filesystem.list_directory_with_sizes slack.update_file google_drive.gdrive_create_file google_drive.gdrive_update_file slack.delete_file slack.create_file figma.figma_get_file github.get_file_blame +q097,Read the whole knowledge graph,paraphrase,memory.read_graph,1.0,1.0,1.0,1.0000,1.0000,1,1,153,1.0,1,254,1.0,1,513,1.0,memory.read_graph memory.create_entities memory.open_nodes memory.delete_relations memory.delete_entities memory.search_nodes memory.add_observations memory.create_relations github.mark_all_notifications_read filesystem.read_file +q098,Search memory nodes,verb_product,memory.search_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,215,1.0,1,357,1.0,1,669,1.0,memory.search_nodes memory.open_nodes brave_search.brave_web_search elasticsearch.es_search brave_search.brave_local_search github.search_orgs notion.notion_search memory.add_observations memory.delete_relations github.search_issues +q099,Show unstaged git diff,verb_product,git.git_diff_unstaged,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,299,1.0,1,585,1.0,git.git_diff_unstaged git.git_diff_staged git.git_diff git.git_show git.git_reset git.git_checkout git.git_commit git.git_status git.git_branch git.git_log +q100,Stage files with git add,identifier,filesystem.list_directory_with_sizes,0.0,1.0,1.0,0.5000,0.6309,2,1,260,1.0,1,382,1.0,1,706,1.0,filesystem.list_directory_with_sizes git.git_add github.push_files filesystem.search_files filesystem.list_directory filesystem.move_file git.git_checkout filesystem.read_text_file git.git_commit git.git_diff_staged +q101,Create a Shopify product,verb_product,shopify.shopify_create_product,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,380,1.0,1,700,1.0,shopify.shopify_create_product shopify.shopify_get_product shopify.shopify_update_product shopify.shopify_create_order shopify.shopify_list_products shopify.shopify_get_order shopify.shopify_list_orders stripe.stripe_create_product shopify.shopify_list_customers github.create_gist +q102,Cancel a Stripe subscription,verb_product,stripe.stripe_cancel_subscription,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,316,1.0,1,632,1.0,stripe.stripe_cancel_subscription stripe.stripe_create_invoice stripe.stripe_create_product stripe.stripe_list_subscriptions stripe.stripe_list_customers stripe.stripe_list_invoices stripe.stripe_create_customer stripe.stripe_create_price stripe.stripe_get_customer stripe.stripe_create_refund +q103,Create a payment intent,resource,stripe.stripe_create_payment_intent,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,424,1.0,1,897,1.0,stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent stripe.stripe_create_refund github.create_environment jira.create_attachment jira.create_sprint gitlab.create_wiki github.create_gist linear.create_comment notion.create_comment +q104,Acknowledge a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,261,1.0,1,405,1.0,1,765,1.0,pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls github.update_package notion.notion_create_page github.delete_package github.create_package +q105,Create a Datadog monitor,verb_product,datadog.datadog_get_monitor,0.0,1.0,1.0,0.5000,0.6309,2,1,259,1.0,1,459,1.0,1,928,1.0,datadog.datadog_get_monitor datadog.datadog_create_monitor datadog.datadog_list_monitors datadog.datadog_list_logs datadog.datadog_query_metrics github.create_gist github.create_runner jira.create_board gitlab.create_job gitlab.create_snippet +q106,Resolve a Sentry issue,paraphrase,sentry.sentry_get_issue,0.0,1.0,1.0,0.5000,0.6309,2,1,251,1.0,1,397,1.0,1,837,1.0,sentry.sentry_get_issue sentry.sentry_update_issue sentry.sentry_list_events sentry.sentry_list_projects sentry.sentry_list_issues github.issue_write pagerduty.pagerduty_resolve_incident gitlab.gitlab_get_issue jira.jira_update_issue github.issue_read +q107,Scale a Kubernetes deployment,verb_product,kubernetes.k8s_scale_deployment,1.0,1.0,1.0,1.0000,1.0000,1,1,252,1.0,1,407,1.0,1,726,1.0,kubernetes.k8s_scale_deployment kubernetes.k8s_get_pod kubernetes.k8s_list_deployments kubernetes.k8s_delete kubernetes.k8s_apply kubernetes.k8s_pod_logs kubernetes.k8s_list_services kubernetes.k8s_get_events vercel.vercel_get_deployment kubernetes.k8s_list_pods +q108,Apply a Kubernetes manifest,verb_product,kubernetes.k8s_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,253,1.0,1,403,1.0,1,727,1.0,kubernetes.k8s_apply kubernetes.k8s_get_pod kubernetes.k8s_delete kubernetes.k8s_pod_logs kubernetes.k8s_list_services kubernetes.k8s_scale_deployment kubernetes.k8s_list_deployments kubernetes.k8s_get_events kubernetes.k8s_list_pods terraform.tf_apply +q109,Stop a docker container,verb_product,docker.docker_stop,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,345,1.0,1,645,1.0,docker.docker_stop docker.docker_rm docker.docker_logs docker.docker_run docker.docker_exec docker.docker_images docker.docker_ps docker.docker_build kubernetes.k8s_pod_logs hubspot.hubspot_get_contact +q110,Build a docker image,verb_product,docker.docker_build,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,317,1.0,1,614,1.0,docker.docker_build docker.docker_run docker.docker_images docker.docker_stop docker.docker_rm docker.docker_ps docker.docker_logs docker.docker_exec filesystem.read_media_file discord.discord_send_message +q111,Click an element in Playwright,verb_product,playwright.browser_click,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,339,1.0,1,706,1.0,playwright.browser_click puppeteer.puppeteer_click playwright.browser_type playwright.browser_take_screenshot playwright.browser_fill_form playwright.browser_tab_list github.create_issue clickup.clickup_list_tasks github.get_environment github.create_environment +q112,Type text into a browser input,paraphrase,playwright.browser_type,1.0,1.0,1.0,1.0000,0.7602,1,1,217,1.0,1,316,1.0,1,624,1.0,playwright.browser_type playwright.browser_navigate playwright.browser_wait_for playwright.browser_tab_close playwright.browser_tab_new playwright.browser_tab_list playwright.browser_evaluate playwright.browser_press_key supabase.supabase_insert cloudflare.cf_create_dns +q113,List GitLab projects,verb_product,gitlab.gitlab_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,209,1.0,1,316,1.0,1,676,1.0,gitlab.gitlab_list_projects github.projects_list gitlab.gitlab_list_issues jira.jira_list_projects gitlab.gitlab_list_merge_requests gitlab.gitlab_create_issue github.projects_get vercel.vercel_list_projects gitlab.gitlab_get_issue gitlab.gitlab_get_file +q114,Create a Bitbucket pull request,verb_product,bitbucket.bitbucket_create_pr,1.0,1.0,1.0,1.0000,1.0000,1,1,313,1.0,1,516,1.0,1,1097,1.0,bitbucket.bitbucket_create_pr bitbucket.bitbucket_list_prs bitbucket.bitbucket_create_issue github.create_pull_request bitbucket.bitbucket_list_repos github.pull_request_read github.request_copilot_review github.update_pull_request github.pull_request_review_write github.merge_pull_request +q115,Create a Gmail draft,verb_product,gmail.gmail_create_draft,1.0,1.0,1.0,1.0000,1.0000,1,1,228,1.0,1,375,1.0,1,836,1.0,gmail.gmail_create_draft gmail.gmail_trash_email gmail.gmail_search_emails gmail.gmail_send_email gmail.gmail_get_email github.create_gist github.create_pull_request github.create_artifact gmail.gmail_list_labels jira.create_sprint +q116,Share a Google Drive file,verb_product,google_drive.gdrive_share_file,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,418,1.0,1,802,1.0,google_drive.gdrive_share_file google_drive.gdrive_update_file google_drive.gdrive_get_file google_drive.gdrive_create_file google_drive.gdrive_list_folder google_drive.gdrive_search filesystem.write_file github.delete_file figma.figma_get_file gitlab.gitlab_get_file +q117,List Google Calendar events,verb_product,google_calendar.gcal_list_events,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,427,1.0,1,714,1.0,google_calendar.gcal_list_events google_calendar.gcal_delete_event google_calendar.gcal_list_calendars google_calendar.gcal_update_event google_calendar.gcal_create_event sentry.sentry_list_events kubernetes.k8s_get_events gitlab.gitlab_list_projects google_drive.gdrive_list_folder google_drive.gdrive_get_file +q118,Delete a calendar event,resource,google_calendar.gcal_delete_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,427,1.0,1,739,1.0,google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars github.delete_environment github.delete_repository github.delete_milestone kubernetes.k8s_delete linear.delete_comment +q119,List Vercel projects,verb_product,vercel.vercel_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,175,1.0,1,269,1.0,1,567,1.0,vercel.vercel_list_projects vercel.vercel_list_deployments vercel.vercel_create_project vercel.vercel_get_deployment jira.jira_list_projects github.projects_list github.projects_write gitlab.gitlab_list_projects linear.linear_list_projects sentry.sentry_list_projects +q120,Select rows from Supabase,verb_product,supabase.supabase_select,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,354,1.0,1,688,1.0,supabase.supabase_select supabase.supabase_insert supabase.supabase_update supabase.supabase_invoke_fn supabase.supabase_list_tables mongodb.mongo_delete figma.figma_get_images aws_s3.s3_get_object memory.delete_entities memory.delete_relations +q121,Post a Figma comment,verb_product,figma.figma_post_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,368,1.0,1,683,1.0,figma.figma_post_comment figma.figma_get_file figma.figma_get_comments figma.figma_get_images figma.figma_get_file_nodes linear.get_comment linear.update_comment linear.create_comment notion.get_comment linear.delete_comment +q122,List Okta users,verb_product,okta.okta_list_users,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,298,1.0,1,659,1.0,okta.okta_list_users okta.okta_get_user okta.okta_create_user okta.okta_deactivate_user slack.slack_list_users notion.create_users_token slack.slack_get_users notion.list_users_tokens notion.notion_list_users linear.list_users +q123,Range query Prometheus,verb_product,prometheus.prom_query_range,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,329,1.0,1,668,1.0,prometheus.prom_query_range prometheus.prom_query prometheus.prom_list_metrics postgres.explain_query postgres.query datadog.datadog_query_metrics salesforce.salesforce_query sqlite.sqlite_query google_drive.gdrive_search shopify.shopify_list_products +q124,Create a Grafana annotation,verb_product,grafana.grafana_create_annotation,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,391,1.0,1,884,1.0,grafana.grafana_create_annotation grafana.grafana_get_dashboard github.create_gist grafana.grafana_search_dashboards github.create_package gitlab.create_wiki jira.create_attachment github.create_artifact jira.create_board jira.create_sprint +q125,Apply terraform,identifier,terraform.tf_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,171,1.0,1,303,1.0,1,673,1.0,terraform.tf_apply terraform.tf_plan terraform.tf_state_list kubernetes.k8s_apply github.update_environment github.list_environments slack.slack_reply_to_thread memory.delete_entities github.update_gist github.list_releases +q126,List Asana tasks,verb_product,asana.asana_list_tasks,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,404,1.0,1,707,1.0,asana.asana_list_tasks asana.asana_get_task asana.asana_create_task asana.asana_update_task asana.asana_create_project clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments aws_s3.s3_list_buckets github.list_environments +q127,Comment on a Trello card,verb_product,trello.trello_add_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,422,1.0,1,759,1.0,trello.trello_add_comment trello.trello_create_card trello.trello_list_cards trello.trello_create_board github.update_issue_comment linear.create_comment figma.figma_post_comment linear.get_comment linear.update_comment notion.get_comment +q128,Update a ClickUp task,verb_product,clickup.clickup_update_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,399,1.0,1,739,1.0,clickup.clickup_update_task clickup.clickup_get_task clickup.clickup_create_task clickup.clickup_list_tasks asana.asana_update_task gitlab.update_wiki github.update_runner gitlab.update_snippet github.update_package github.update_artifact +q129,Search Confluence,verb_product,confluence.confluence_search,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,401,1.0,1,698,1.0,confluence.confluence_search confluence.confluence_get_page confluence.confluence_create_page confluence.confluence_update_page brave_search.brave_web_search brave_search.brave_local_search github.search_users elasticsearch.es_search github.search_orgs github.search_commits +q130,Create a HubSpot deal,verb_product,hubspot.hubspot_create_deal,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,862,1.0,hubspot.hubspot_create_deal hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_list_contacts hubspot.hubspot_search_contacts github.create_gist github.create_repository github.create_hook github.create_artifact github.create_or_update_file +q131,List Zendesk tickets,verb_product,zendesk.zendesk_list_tickets,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,367,1.0,1,697,1.0,zendesk.zendesk_list_tickets zendesk.zendesk_get_ticket zendesk.zendesk_search zendesk.zendesk_update_ticket zendesk.zendesk_create_ticket aws_s3.s3_list_buckets kubernetes.k8s_list_deployments jira.list_sprints gitlab.list_snippets notion.list_comments +q132,List Discord channels,verb_product,discord.discord_list_channels,1.0,1.0,1.0,1.0000,1.0000,1,1,166,1.0,1,321,1.0,1,669,1.0,discord.discord_list_channels discord.discord_get_channel discord.discord_list_members discord.discord_send_message discord.discord_add_reaction slack.slack_list_channels trello.trello_list_cards github.projects_list github.actions_list github.list_discussions +q133,Set a Redis key,verb_product,redis.redis_set,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,300,1.0,1,651,1.0,redis.redis_set redis.redis_get redis.redis_delete redis.redis_keys aws_s3.s3_get_object playwright.browser_press_key filesystem.edit_file jira.jira_get_issue github.get_label slack.slack_set_channel_topic +q134,Insert MongoDB documents,verb_product,mongodb.mongo_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,256,1.0,1,381,1.0,1,766,1.0,mongodb.mongo_insert mongodb.mongo_delete mongodb.mongo_update mongodb.mongo_find mongodb.mongo_list_collections supabase.supabase_insert elasticsearch.es_index_doc stripe.stripe_create_payment_intent github.create_environment okta.okta_list_users +q135,Run a MySQL query,verb_product,postgres.query,0.0,1.0,1.0,0.5000,0.6309,2,1,182,1.0,1,281,1.0,1,650,1.0,postgres.query mysql.mysql_query postgres.explain_query sqlite.sqlite_query mysql.mysql_list_tables prometheus.prom_query salesforce.salesforce_query datadog.datadog_query_metrics prometheus.prom_query_range memory.search_nodes +q136,Index a document in Elasticsearch,verb_product,elasticsearch.es_index_doc,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,386,1.0,1,810,1.0,elasticsearch.es_index_doc elasticsearch.es_search elasticsearch.es_get_doc playwright.browser_tab_close github.create_environment github.list_environments github.update_environment gitlab.create_wiki mongodb.mongo_insert github.create_milestone +q137,Get an S3 object,verb_product,aws_s3.s3_get_object,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,375,1.0,1,705,1.0,aws_s3.s3_get_object aws_s3.s3_delete_object aws_s3.s3_put_object aws_s3.s3_list_objects okta.okta_get_user github.get_label github.get_milestone twilio.twilio_send_sms gitlab.get_label github.get_environment +q138,List Cloudflare zones,verb_product,cloudflare.cf_list_zones,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,341,1.0,1,719,1.0,cloudflare.cf_list_zones cloudflare.cf_list_dns cloudflare.cf_purge_cache cloudflare.cf_create_dns kubernetes.k8s_list_deployments clickup.clickup_list_tasks gitlab.list_milestones gitlab.list_releases github.list_milestones gitlab.list_snippets +q139,Create a Stripe invoice,verb_product,stripe.stripe_create_invoice,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,333,1.0,1,654,1.0,stripe.stripe_create_invoice stripe.stripe_create_product stripe.stripe_create_customer stripe.stripe_create_price stripe.stripe_list_invoices stripe.stripe_create_refund stripe.stripe_create_payment_intent stripe.stripe_cancel_subscription stripe.stripe_list_subscriptions stripe.stripe_get_customer +q140,List GitHub Actions workflows,verb_product,github.actions_list,1.0,1.0,1.0,1.0000,1.0000,1,1,293,1.0,1,457,1.0,1,790,1.0,github.actions_list github.actions_get github.actions_run_trigger github.actions_get_job_logs github.list_tags github.list_discussions github.list_branches github.projects_list github.list_gists github.list_releases +q141,Get GitHub Actions job logs,verb_product,github.actions_get_job_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,471,1.0,1,884,1.0,github.actions_get_job_logs github.actions_get github.actions_list github.actions_run_trigger github.get_label github.get_discussion github.get_latest_release github.get_tag github.get_commit github.search_issues +q142,Create a GitHub repository,verb_product,github.create_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,388,1.0,1,949,1.0,github.create_repository github.delete_repository github.fork_repository github.star_repository github.unstar_repository github.list_repository_collaborators github.get_repository_tree github.create_branch github.create_issue github.create_or_update_file +q143,Checkout a git branch,verb_product,git.git_checkout,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,376,1.0,1,801,1.0,git.git_checkout git.git_create_branch git.git_branch github.create_branch git.git_diff git.git_show github.list_commits github.delete_file github.update_pull_request_branch git.git_status +q144,Show git log,identifier,git.git_log,1.0,1.0,1.0,1.0000,1.0000,1,1,206,1.0,1,304,1.0,1,619,1.0,git.git_log git.git_show git.git_diff git.git_branch git.git_status git.git_checkout git.git_diff_unstaged git.git_diff_staged git.git_commit git.git_create_branch +q145,List postgres schemas,verb_product,postgres.list_schemas,1.0,1.0,1.0,1.0000,1.0000,1,1,158,1.0,1,277,1.0,1,595,1.0,postgres.list_schemas postgres.list_tables postgres.query postgres.describe_table postgres.explain_query github.projects_list kubernetes.k8s_list_services kubernetes.k8s_list_deployments github.actions_list jira.list_attachments +q146,Open nodes in the knowledge graph,verb_product,memory.open_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,163,1.0,1,267,1.0,1,504,1.0,memory.open_nodes memory.read_graph memory.search_nodes memory.create_entities memory.add_observations memory.create_relations memory.delete_relations memory.delete_entities sqlite.sqlite_list_tables postgres.list_tables +q147,Get file metadata from the filesystem,paraphrase,filesystem.read_file,0.0,0.0,1.0,0.2500,0.4307,4,2,426,0.0,1,348,1.0,1,686,1.0,filesystem.read_file filesystem.read_media_file filesystem.read_text_file filesystem.get_file_info filesystem.write_file google_drive.gdrive_get_file filesystem.move_file figma.figma_get_file figma.figma_get_file_nodes gitlab.gitlab_get_file +q148,Write a new local file,paraphrase,filesystem.write_file,1.0,1.0,1.0,1.0000,1.0000,1,1,199,1.0,1,342,1.0,1,737,1.0,filesystem.write_file playwright.browser_tab_new filesystem.read_media_file filesystem.read_file filesystem.read_text_file figma.figma_get_file google_drive.gdrive_share_file google_drive.gdrive_update_file slack.update_file slack.create_file +q149,Search GitHub issues,verb_product,github.search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,363,1.0,1,683,1.0,github.search_issues github.list_issues github.search_orgs github.search_users github.search_pull_requests jira.jira_search_issues github.search_commits github.search_code linear.linear_search_issues gitlab.gitlab_list_issues +q150,Get the authenticated GitHub user,paraphrase,github.get_teams,0.0,1.0,1.0,0.5000,0.6309,2,1,179,1.0,1,301,1.0,1,693,1.0,github.get_teams github.get_me github.list_notifications github.get_latest_release github.list_gists github.get_gist github.get_label github.get_repository_tree github.get_team_members github.get_commit diff --git a/experiments/mcp-tool-search/results/hybrid_rrf_weighted_hashed.json b/experiments/mcp-tool-search/results/hybrid_rrf_weighted_hashed.json new file mode 100644 index 00000000..3e630c01 --- /dev/null +++ b/experiments/mcp-tool-search/results/hybrid_rrf_weighted_hashed.json @@ -0,0 +1,7013 @@ +{ + "summary": { + "n": 150, + "index_s": 0.024298550000139585, + "rss_mb": 0.03125, + "p50_ms": 4.685365999989699, + "p95_ms": 4.952740550021417, + "recall@1": 0.86, + "recall@1_lo": 0.8, + "recall@1_hi": 0.9133333333333333, + "recall@3": 0.9866666666666667, + "recall@3_lo": 0.9666666666666667, + "recall@3_hi": 1.0, + "recall@5": 0.9933333333333333, + "recall@5_lo": 0.98, + "recall@5_hi": 1.0, + "mrr@10": 0.923888888888889, + "mrr@10_lo": 0.8927777777777777, + "mrr@10_hi": 0.9533333333333334, + "ndcg@10": 0.9359694088979469, + "ndcg@10_lo": 0.9119244041164476, + "ndcg@10_hi": 0.9580275511730296 + }, + "meta": { + "name": "hybrid_rrf_weighted_hashed", + "needs_model": false, + "needs_api": false, + "notes": "RRF of weighted BM25 + hashed embedding" + }, + "rows": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.issue_write", + "github.issue_read", + "github.create_issue", + "github.update_issue_comment", + "github.add_issue_comment", + "github.create_gist", + "gitlab.gitlab_create_issue", + "github.assign_copilot_to_issue", + "gitlab.gitlab_get_issue", + "github.create_repository" + ], + "hit1": "github.issue_write", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 388, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 598, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1024, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "kind": "verb_product", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_create_issue", + "jira.jira_create_sprint", + "jira.jira_assign_issue", + "jira.jira_get_issue", + "jira.jira_list_projects", + "jira.create_board", + "jira.create_webhook", + "jira.create_filter", + "jira.create_sprint", + "jira.jira_update_issue" + ], + "hit1": "jira.jira_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 312, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 398, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 854, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "kind": "verb_product", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "ranking": [ + "slack.slack_post_message", + "slack.slack_add_reaction", + "slack.slack_search_messages", + "slack.slack_join_channel", + "slack.slack_create_channel", + "slack.slack_leave_channel", + "slack.slack_get_channel_history", + "slack.slack_upload_file", + "discord.discord_send_message", + "slack.slack_reply_to_thread" + ], + "hit1": "slack.slack_post_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "kind": "resource", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "ranking": [ + "github.list_pull_requests", + "github.create_pull_request", + "github.search_pull_requests", + "github.merge_pull_request", + "github.pull_request_read", + "gitlab.gitlab_list_merge_requests", + "github.request_copilot_review", + "github.pull_request_review_write", + "github.update_pull_request", + "github.update_pull_request_branch" + ], + "hit1": "github.list_pull_requests", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 311, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 506, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1060, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "ranking": [ + "github.create_pull_request", + "github.merge_pull_request", + "github.request_copilot_review", + "github.pull_request_read", + "github.search_pull_requests", + "github.update_pull_request", + "github.pull_request_review_write", + "github.list_pull_requests", + "github.update_pull_request_branch", + "github.add_comment_to_pending_review" + ], + "hit1": "github.create_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 379, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 522, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1144, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "kind": "verb_product", + "gold": [ + "postgres.query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "postgres.explain_query", + "mysql.mysql_query", + "postgres.list_schemas", + "sqlite.sqlite_query", + "salesforce.salesforce_query", + "postgres.describe_table", + "postgres.list_tables", + "prometheus.prom_query", + "docker.docker_run" + ], + "hit1": "postgres.query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 272, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 590, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "kind": "paraphrase", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "ranking": [ + "filesystem.read_file", + "filesystem.read_media_file", + "filesystem.read_text_file", + "filesystem.edit_file", + "filesystem.write_file", + "gitlab.gitlab_get_file", + "slack.get_file", + "figma.figma_get_file", + "filesystem.get_file_info", + "github.get_file_contents" + ], + "hit1": "filesystem.read_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 745, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "kind": "paraphrase", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "ranking": [ + "google_drive.gdrive_list_folder", + "github.push_files", + "slack.list_files", + "filesystem.move_file", + "filesystem.search_files", + "filesystem.list_directory_with_sizes", + "filesystem.list_directory", + "filesystem.directory_tree", + "filesystem.list_allowed_directories", + "filesystem.edit_file" + ], + "hit1": "google_drive.gdrive_list_folder", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.16666666666666666, + "ndcg@10": 0.42745728454423865, + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 478, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 719, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 697, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "ranking": [ + "memory.read_graph", + "memory.create_entities", + "memory.open_nodes", + "memory.add_observations", + "memory.delete_relations", + "memory.search_nodes", + "memory.create_relations", + "memory.delete_entities", + "sqlite.sqlite_list_tables", + "github.get_me" + ], + "hit1": "memory.read_graph", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.5672074169568709, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 155, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 260, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 510, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "kind": "verb_product", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "ranking": [ + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "jira.jira_search_issues", + "memory.search_nodes", + "github.search_users", + "elasticsearch.es_search", + "github.search_orgs", + "github.search_issues", + "puppeteer.puppeteer_screenshot", + "github.search_commits" + ], + "hit1": "brave_search.brave_web_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 365, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "kind": "verb_product", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "ranking": [ + "fetch.fetch", + "filesystem.read_file", + "playwright.browser_navigate", + "github.get_file_contents", + "playwright.browser_take_screenshot", + "git.git_add", + "postgres.describe_table", + "puppeteer.puppeteer_navigate", + "github.get_me", + "git.git_status" + ], + "hit1": "fetch.fetch", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 681, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "kind": "identifier", + "gold": [ + "git.git_status" + ], + "alts": [], + "ranking": [ + "git.git_status", + "git.git_show", + "git.git_diff", + "git.git_branch", + "git.git_diff_staged", + "git.git_diff_unstaged", + "git.git_log", + "git.git_checkout", + "git.git_add", + "git.git_reset" + ], + "hit1": "git.git_status", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 293, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 584, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "kind": "paraphrase", + "gold": [ + "git.git_commit" + ], + "alts": [], + "ranking": [ + "git.git_commit", + "git.git_reset", + "github.get_commit", + "git.git_diff_staged", + "git.git_show", + "github.push_files", + "git.git_log", + "git.git_diff_unstaged", + "filesystem.read_media_file", + "filesystem.read_text_file" + ], + "hit1": "git.git_commit", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_issue", + "linear.linear_update_issue", + "linear.linear_create_comment", + "linear.linear_get_issue", + "linear.linear_assign_issue", + "linear.linear_create_project", + "linear.linear_list_issues", + "linear.linear_search_issues", + "linear.create_label", + "linear.create_cycle" + ], + "hit1": "linear.linear_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 875, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "kind": "paraphrase", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_project", + "linear.linear_list_projects", + "linear.linear_create_issue", + "linear.list_cycles", + "vercel.vercel_create_project", + "linear.linear_create_comment", + "linear.list_users", + "linear.linear_search_issues", + "linear.linear_list_teams", + "linear.linear_update_issue" + ], + "hit1": "linear.linear_create_project", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 284, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 415, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "ranking": [ + "github.update_issue_comment", + "github.add_issue_comment", + "github.issue_read", + "github.issue_write", + "figma.figma_post_comment", + "github.add_comment_to_pending_review", + "github.create_issue", + "github.list_commits", + "linear.linear_create_comment", + "linear.update_comment" + ], + "hit1": "github.update_issue_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 542, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1063, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "kind": "ambiguous", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "ranking": [ + "github.merge_pull_request", + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_get_merge_request", + "github.pull_request_read", + "github.create_pull_request", + "github.request_copilot_review", + "github.update_pull_request_branch", + "github.search_pull_requests", + "github.update_pull_request", + "gitlab.gitlab_create_merge_request" + ], + "hit1": "github.merge_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 523, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1031, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "kind": "verb_product", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "ranking": [ + "github.actions_run_trigger", + "github.actions_get", + "github.actions_list", + "github.actions_get_job_logs", + "github.fork_repository", + "github.star_repository", + "github.list_tags", + "github.list_branches", + "github.unstar_repository", + "github.get_tag" + ], + "hit1": "github.actions_run_trigger", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 835, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "kind": "resource", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "ranking": [ + "github.list_dependabot_alerts", + "github.get_dependabot_alert", + "github.list_code_scanning_alerts", + "github.list_secret_scanning_alerts", + "jira.get_board", + "jira.get_attachment", + "gitlab.get_job", + "jira.get_webhook", + "sentry.sentry_get_issue", + "github.get_commit" + ], + "hit1": "github.list_dependabot_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 287, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 730, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "kind": "verb_product", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_issues", + "sentry.sentry_list_events", + "sentry.sentry_list_projects", + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "github.list_issues", + "gitlab.gitlab_list_issues", + "linear.linear_list_issues", + "github.search_issues", + "jira.jira_search_issues" + ], + "hit1": "sentry.sentry_list_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 813, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "ranking": [ + "stripe.stripe_get_customer", + "stripe.stripe_create_customer", + "stripe.stripe_create_invoice", + "stripe.stripe_create_price", + "stripe.stripe_list_invoices", + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_product", + "stripe.stripe_list_subscriptions", + "stripe.stripe_create_refund", + "stripe.stripe_list_customers" + ], + "hit1": "stripe.stripe_get_customer", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "kind": "paraphrase", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_refund", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_create_invoice", + "stripe.stripe_list_customers", + "stripe.stripe_create_price", + "stripe.stripe_list_invoices", + "stripe.stripe_create_product" + ], + "hit1": "stripe.stripe_create_refund", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "kind": "verb_product", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "ranking": [ + "gmail.gmail_send_email", + "gmail.gmail_get_email", + "gmail.gmail_search_emails", + "gmail.gmail_trash_email", + "gmail.gmail_list_labels", + "gmail.gmail_modify_email", + "gmail.gmail_create_draft", + "filesystem.list_directory_with_sizes", + "twilio.twilio_send_sms", + "google_drive.gdrive_share_file" + ], + "hit1": "gmail.gmail_send_email", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 735, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "kind": "verb_product", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "ranking": [ + "gmail.gmail_search_emails", + "gmail.gmail_send_email", + "gmail.gmail_create_draft", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "gmail.gmail_list_labels", + "brave_search.brave_web_search", + "gmail.gmail_modify_email", + "elasticsearch.es_search", + "notion.notion_search" + ], + "hit1": "gmail.gmail_search_emails", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 265, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "kind": "paraphrase", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_update_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_share_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "slack.slack_upload_file", + "gitlab.gitlab_get_file", + "filesystem.edit_file", + "figma.figma_get_file" + ], + "hit1": "google_drive.gdrive_update_file", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 307, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 794, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_create_event", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "github.create_gist", + "jira.create_attachment", + "github.create_environment", + "linear.create_comment", + "jira.create_board" + ], + "hit1": "google_calendar.gcal_create_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 885, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_list_pods", + "kubernetes.k8s_list_services", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_get_events", + "kubernetes.k8s_apply", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_delete", + "kubernetes.k8s_scale_deployment", + "clickup.clickup_list_tasks" + ], + "hit1": "kubernetes.k8s_list_pods", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_pod_logs", + "docker.docker_logs", + "gitlab.gitlab_get_file", + "github.actions_get_job_logs", + "kubernetes.k8s_get_pod", + "github.get_tag", + "elasticsearch.es_get_doc", + "github.get_file_contents", + "figma.figma_get_images", + "github.delete_file" + ], + "hit1": "kubernetes.k8s_pod_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 275, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 447, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 911, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "ranking": [ + "docker.docker_stop", + "docker.docker_run", + "docker.docker_exec", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_images", + "docker.docker_ps", + "docker.docker_build", + "mysql.mysql_query", + "kubernetes.k8s_pod_logs" + ], + "hit1": "docker.docker_stop", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 640, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "kind": "verb_product", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "ranking": [ + "docker.docker_logs", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_exec", + "docker.docker_run", + "docker.docker_ps", + "docker.docker_images", + "docker.docker_build", + "git.git_log", + "kubernetes.k8s_pod_logs" + ], + "hit1": "docker.docker_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 649, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "kind": "ambiguous", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "ranking": [ + "playwright.browser_navigate", + "playwright.browser_tab_new", + "puppeteer.puppeteer_navigate", + "playwright.browser_tab_close", + "playwright.browser_tab_list", + "playwright.browser_evaluate", + "playwright.browser_wait_for", + "playwright.browser_type", + "playwright.browser_press_key", + "playwright.browser_snapshot" + ], + "hit1": "playwright.browser_navigate", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 265, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "kind": "ambiguous", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "ranking": [ + "playwright.browser_take_screenshot", + "puppeteer.puppeteer_screenshot", + "notion.notion_update_page", + "playwright.browser_snapshot", + "filesystem.read_file", + "playwright.browser_press_key", + "notion.notion_get_page", + "confluence.confluence_update_page", + "playwright.browser_evaluate", + "notion.notion_create_page" + ], + "hit1": "playwright.browser_take_screenshot", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 671, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_create_merge_request", + "gitlab.gitlab_get_merge_request", + "github.merge_pull_request", + "gitlab.gitlab_list_merge_requests", + "github.request_copilot_review", + "gitlab.gitlab_create_issue", + "gitlab.gitlab_create_pipeline", + "github.create_pull_request", + "gitlab.gitlab_get_issue" + ], + "hit1": "gitlab.gitlab_merge_merge_request", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 961, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "kind": "verb_product", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "ranking": [ + "notion.notion_search", + "notion.notion_get_page", + "notion.notion_update_page", + "notion.notion_create_page", + "notion.notion_list_users", + "notion.notion_create_database", + "notion.notion_get_comments", + "notion.notion_get_block", + "notion.notion_query_database", + "brave_search.brave_web_search" + ], + "hit1": "notion.notion_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 336, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "kind": "verb_product", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "ranking": [ + "notion.notion_get_page", + "notion.notion_create_page", + "notion.notion_update_page", + "notion.notion_create_database", + "notion.notion_get_comments", + "notion.notion_search", + "notion.create_comment", + "notion.create_data_source", + "notion.notion_get_block", + "notion.create_users_token" + ], + "hit1": "notion.notion_get_page", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "kind": "verb_product", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "ranking": [ + "notion.notion_query_database", + "notion.notion_create_database", + "notion.notion_search", + "notion.notion_get_block", + "notion.notion_get_page", + "notion.notion_create_page", + "notion.notion_list_users", + "notion.notion_get_comments", + "notion.get_data_source", + "notion.notion_append_block" + ], + "hit1": "notion.notion_query_database", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 674, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_search", + "zendesk.zendesk_list_tickets", + "jira.jira_create_issue", + "github.create_gist", + "jira.create_sprint", + "github.create_hook", + "github.create_artifact" + ], + "hit1": "zendesk.zendesk_create_ticket", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 856, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_create_deal", + "hubspot.hubspot_search_contacts", + "hubspot.hubspot_list_contacts", + "github.create_gist", + "github.create_or_update_file", + "github.create_artifact", + "github.create_package", + "github.create_hook" + ], + "hit1": "hubspot.hubspot_create_contact", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 877, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "kind": "verb_product", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_create_lead", + "salesforce.salesforce_convert_lead", + "salesforce.salesforce_create_record", + "salesforce.salesforce_update_record", + "salesforce.salesforce_query", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_get_record", + "salesforce.salesforce_list_opportunities", + "github.create_gist", + "github.create_artifact" + ], + "hit1": "salesforce.salesforce_create_lead", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 782, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "kind": "identifier", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_query", + "postgres.query", + "mysql.mysql_query", + "sqlite.sqlite_query", + "prometheus.prom_query", + "postgres.explain_query", + "memory.search_nodes", + "github.update_check_run", + "datadog.datadog_query_metrics", + "prometheus.prom_query_range" + ], + "hit1": "salesforce.salesforce_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 187, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 672, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "kind": "verb_product", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "ranking": [ + "shopify.shopify_list_orders", + "shopify.shopify_list_products", + "shopify.shopify_list_customers", + "shopify.shopify_get_order", + "shopify.shopify_get_product", + "shopify.shopify_create_order", + "shopify.shopify_create_product", + "shopify.shopify_update_product", + "github.projects_list", + "trello.trello_list_cards" + ], + "hit1": "shopify.shopify_list_orders", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 695, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "github.create_gist", + "github.create_package", + "github.create_hook", + "gitlab.create_wiki" + ], + "hit1": "pagerduty.pagerduty_create_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 836, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "kind": "paraphrase", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_list_oncalls", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_resolve_incident", + "twilio.twilio_make_call", + "notion.notion_get_comments", + "memory.search_nodes", + "github.assign_copilot_to_issue" + ], + "hit1": "pagerduty.pagerduty_list_oncalls", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 771, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "kind": "verb_product", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "ranking": [ + "datadog.datadog_query_metrics", + "datadog.datadog_list_monitors", + "datadog.datadog_create_monitor", + "datadog.datadog_list_logs", + "datadog.datadog_get_monitor", + "prometheus.prom_query", + "prometheus.prom_list_metrics", + "notion.notion_query_database", + "mysql.mysql_query", + "prometheus.prom_query_range" + ], + "hit1": "datadog.datadog_query_metrics", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "kind": "verb_product", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "ranking": [ + "datadog.datadog_list_logs", + "datadog.datadog_create_monitor", + "datadog.datadog_get_monitor", + "datadog.datadog_list_monitors", + "elasticsearch.es_search", + "brave_search.brave_web_search", + "github.search_orgs", + "datadog.datadog_query_metrics", + "notion.notion_search", + "github.search_users" + ], + "hit1": "datadog.datadog_list_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 423, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 784, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_list_buckets", + "aws_s3.s3_list_objects", + "github.list_packages", + "gitlab.list_snippets", + "github.list_environments", + "github.list_runners", + "github.list_hooks", + "github.list_artifacts", + "kubernetes.k8s_list_deployments", + "clickup.clickup_list_tasks" + ], + "hit1": "aws_s3.s3_list_buckets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "kind": "paraphrase", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_put_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_get_object", + "aws_s3.s3_list_objects", + "twilio.twilio_send_sms", + "gmail.gmail_send_email", + "slack.slack_upload_file", + "asana.asana_create_project", + "okta.okta_deactivate_user", + "github.add_comment_to_pending_review" + ], + "hit1": "aws_s3.s3_put_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 866, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "github.search_orgs", + "gmail.gmail_search_emails", + "slack.list_reminders", + "gitlab.get_release", + "gitlab.gitlab_create_merge_request", + "gitlab.gitlab_list_merge_requests" + ], + "hit1": "cloudflare.cf_purge_cache", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 732, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "cloudflare.cf_list_zones", + "cloudflare.cf_purge_cache", + "salesforce.salesforce_create_record", + "github.create_gist", + "jira.create_attachment", + "jira.create_board", + "github.create_hook", + "jira.create_sprint" + ], + "hit1": "cloudflare.cf_create_dns", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 819, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_deployments", + "vercel.vercel_get_deployment", + "vercel.vercel_list_projects", + "kubernetes.k8s_list_deployments", + "vercel.vercel_create_project", + "clickup.clickup_list_tasks", + "github.list_environments", + "aws_s3.s3_list_buckets", + "trello.trello_list_cards", + "jira.list_attachments" + ], + "hit1": "vercel.vercel_list_deployments", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 615, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "ranking": [ + "supabase.supabase_insert", + "supabase.supabase_select", + "supabase.supabase_update", + "supabase.supabase_invoke_fn", + "supabase.supabase_list_tables", + "mongodb.mongo_insert", + "playwright.browser_type", + "jira.jira_create_issue", + "jira.get_version", + "jira.create_version" + ], + "hit1": "supabase.supabase_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 767, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "kind": "paraphrase", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "ranking": [ + "figma.figma_get_images", + "figma.figma_get_file_nodes", + "docker.docker_images", + "figma.figma_get_file", + "figma.figma_get_comments", + "figma.figma_post_comment", + "gitlab.gitlab_get_file", + "memory.delete_entities", + "supabase.supabase_select", + "memory.delete_relations" + ], + "hit1": "figma.figma_get_images", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "kind": "verb_product", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "ranking": [ + "twilio.twilio_send_sms", + "twilio.twilio_list_messages", + "twilio.twilio_make_call", + "gmail.gmail_send_email", + "okta.okta_deactivate_user", + "discord.discord_send_message", + "github.update_pull_request_branch", + "filesystem.list_directory_with_sizes", + "okta.okta_create_user", + "playwright.browser_type" + ], + "hit1": "twilio.twilio_send_sms", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 398, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 748, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "kind": "verb_product", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "ranking": [ + "okta.okta_get_user", + "okta.okta_create_user", + "okta.okta_deactivate_user", + "okta.okta_list_users", + "linear.create_user", + "asana.asana_create_project", + "asana.asana_create_task", + "github.create_issue", + "github.create_environment", + "github.create_hook" + ], + "hit1": "okta.okta_get_user", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 827, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query", + "prometheus.prom_query_range", + "prometheus.prom_list_metrics", + "postgres.query", + "postgres.explain_query", + "datadog.datadog_query_metrics", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "memory.search_nodes" + ], + "hit1": "prometheus.prom_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 652, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "kind": "verb_product", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "ranking": [ + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "grafana.grafana_create_annotation", + "brave_search.brave_local_search", + "brave_search.brave_web_search", + "elasticsearch.es_search", + "notion.notion_search", + "github.search_orgs", + "github.search_users", + "github.search_issues" + ], + "hit1": "grafana.grafana_search_dashboards", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "kind": "verb_product", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "ranking": [ + "circleci.circleci_trigger_pipeline", + "circleci.circleci_get_pipeline", + "circleci.circleci_list_pipelines", + "gitlab.gitlab_create_pipeline", + "gitlab.delete_pipeline", + "gitlab.create_pipeline", + "gitlab.update_pipeline", + "gitlab.get_pipeline", + "hubspot.hubspot_create_deal", + "github.actions_run_trigger" + ], + "hit1": "circleci.circleci_trigger_pipeline", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 325, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "kind": "identifier", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "ranking": [ + "terraform.tf_plan", + "terraform.tf_apply", + "terraform.tf_state_list", + "postgres.explain_query", + "salesforce.salesforce_query", + "postgres.query", + "docker.docker_exec", + "playwright.browser_fill_form", + "docker.docker_run", + "github.list_releases" + ], + "hit1": "terraform.tf_plan", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 287, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 618, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "kind": "verb_product", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "ranking": [ + "asana.asana_get_task", + "asana.asana_create_task", + "asana.asana_update_task", + "asana.asana_create_project", + "asana.asana_list_tasks", + "clickup.clickup_create_task", + "okta.okta_create_user", + "github.create_gist", + "github.create_issue", + "github.create_package" + ], + "hit1": "asana.asana_get_task", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 853, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "ranking": [ + "trello.trello_create_card", + "trello.trello_add_comment", + "trello.trello_create_board", + "trello.trello_list_cards", + "github.create_gist", + "jira.create_board", + "github.create_runner", + "github.create_package", + "github.create_artifact", + "jira.create_filter" + ], + "hit1": "trello.trello_create_card", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 394, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 874, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_get_task", + "clickup.clickup_create_task", + "clickup.clickup_update_task", + "clickup.clickup_list_tasks", + "github.create_artifact", + "github.create_gist", + "github.create_runner", + "gitlab.create_release", + "github.create_package", + "jira.create_attachment" + ], + "hit1": "clickup.clickup_get_task", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 410, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 871, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "kind": "verb_product", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "ranking": [ + "confluence.confluence_create_page", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "confluence.confluence_search", + "notion.notion_create_page", + "jira.create_board", + "jira.create_attachment", + "jira.create_filter", + "github.create_package", + "jira.create_version" + ], + "hit1": "confluence.confluence_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 876, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "kind": "verb_product", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "discord.discord_add_reaction", + "discord.discord_get_channel", + "discord.discord_list_channels", + "discord.discord_list_members", + "slack.slack_post_message", + "git.git_commit", + "datadog.datadog_create_monitor", + "figma.figma_post_comment", + "gmail.gmail_get_email" + ], + "hit1": "discord.discord_send_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_search", + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "github.search_users", + "github.search_orgs", + "notion.notion_search", + "confluence.confluence_search", + "zendesk.zendesk_search" + ], + "hit1": "elasticsearch.es_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "redis.redis_set", + "redis.redis_delete", + "redis.redis_keys", + "github.get_label", + "github.get_discussion", + "aws_s3.s3_get_object", + "github.get_artifact", + "figma.figma_get_file", + "gitlab.get_label" + ], + "hit1": "redis.redis_get", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 631, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_find", + "mongodb.mongo_insert", + "mongodb.mongo_update", + "mongodb.mongo_delete", + "mongodb.mongo_list_collections", + "elasticsearch.es_index_doc", + "github.list_environments", + "zendesk.zendesk_create_ticket", + "elasticsearch.es_get_doc", + "stripe.stripe_create_payment_intent" + ], + "hit1": "mongodb.mongo_find", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 788, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "kind": "verb_product", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "ranking": [ + "sqlite.sqlite_list_tables", + "sqlite.sqlite_describe_table", + "mysql.mysql_list_tables", + "postgres.list_tables", + "sqlite.sqlite_query", + "supabase.supabase_list_tables", + "github.list_milestones", + "gitlab.list_milestones", + "postgres.list_schemas", + "github.list_hooks" + ], + "hit1": "sqlite.sqlite_list_tables", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 252, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 567, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "kind": "verb_product", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "ranking": [ + "postgres.describe_table", + "sqlite.sqlite_describe_table", + "postgres.query", + "postgres.list_tables", + "postgres.list_schemas", + "postgres.explain_query", + "supabase.supabase_select", + "hubspot.hubspot_create_deal", + "supabase.supabase_insert", + "confluence.confluence_update_page" + ], + "hit1": "postgres.describe_table", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 272, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 643, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "github.update_issue_comment", + "github.add_issue_comment", + "gitlab.gitlab_create_issue", + "github.issue_write", + "github.issue_read", + "github.assign_copilot_to_issue", + "bitbucket.bitbucket_create_issue", + "jira.jira_create_issue", + "github.create_hook" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9072278740982787, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 361, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 588, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1092, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "ranking": [ + "gitlab.gitlab_create_issue", + "bitbucket.bitbucket_create_issue", + "github.create_issue", + "jira.jira_create_issue", + "gitlab.gitlab_get_issue", + "github.issue_write", + "sentry.sentry_get_issue", + "jira.jira_get_issue", + "linear.linear_create_issue", + "github.create_gist" + ], + "hit1": "gitlab.gitlab_create_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5530965847323828, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 332, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 520, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 988, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "kind": "identifier", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "ranking": [ + "filesystem.create_directory", + "filesystem.list_directory", + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree", + "filesystem.list_allowed_directories", + "filesystem.get_file_info", + "filesystem.move_file", + "github.get_file_contents", + "trello.trello_list_cards", + "github.projects_list" + ], + "hit1": "filesystem.create_directory", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 646, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "kind": "paraphrase", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "ranking": [ + "sequential_thinking.sequentialthinking", + "filesystem.edit_file", + "kubernetes.k8s_get_pod", + "filesystem.list_allowed_directories", + "github.list_pull_requests", + "stripe.stripe_list_subscriptions", + "playwright.browser_wait_for", + "kubernetes.k8s_get_events", + "notion.list_data_sources", + "git.git_status" + ], + "hit1": "sequential_thinking.sequentialthinking", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 302, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 430, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "kind": "paraphrase", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "ranking": [ + "time.convert_time", + "time.get_current_time", + "playwright.browser_wait_for", + "jira.jira_add_worklog", + "github.list_hooks", + "github.delete_milestone", + "gitlab.get_pipeline", + "gitlab.list_pipelines", + "github.delete_environment", + "gitlab.delete_pipeline" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8597186998521972, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 674, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "kind": "verb_product", + "gold": [ + "time.convert_time" + ], + "alts": [], + "ranking": [ + "time.convert_time", + "time.get_current_time", + "git.git_diff", + "playwright.browser_wait_for", + "google_calendar.gcal_list_events", + "salesforce.salesforce_convert_lead", + "docker.docker_logs", + "notion.create_users_token", + "memory.create_relations", + "slack.create_usergroup" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 369, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 743, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "kind": "verb_product", + "gold": [ + "github.star_repository" + ], + "alts": [], + "ranking": [ + "github.star_repository", + "github.unstar_repository", + "github.delete_repository", + "github.get_tag", + "github.fork_repository", + "github.list_tags", + "github.list_releases", + "github.list_branches", + "github.list_issues", + "github.get_label" + ], + "hit1": "github.star_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 393, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 792, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "kind": "resource", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "ranking": [ + "github.fork_repository", + "github.delete_repository", + "github.star_repository", + "github.unstar_repository", + "github.create_repository", + "github.list_repository_collaborators", + "github.get_repository_tree", + "github.list_releases", + "github.list_dependabot_alerts", + "github.list_branches" + ], + "hit1": "github.fork_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 792, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "kind": "verb_product", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "ranking": [ + "github.get_latest_release", + "github.get_release_by_tag", + "github.get_label", + "github.get_repository_tree", + "github.get_teams", + "gitlab.get_release", + "github.get_gist", + "github.get_me", + "github.get_commit", + "github.search_commits" + ], + "hit1": "github.get_latest_release", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "kind": "resource", + "gold": [ + "github.create_gist" + ], + "alts": [], + "ranking": [ + "github.create_gist", + "github.update_gist", + "github.create_package", + "slack.create_canvase", + "github.create_artifact", + "gitlab.create_wiki", + "github.get_gist", + "github.create_hook", + "gitlab.create_pipeline", + "slack.create_pin" + ], + "hit1": "github.create_gist", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 272, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 480, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 925, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "kind": "verb_product", + "gold": [ + "github.search_code" + ], + "alts": [], + "ranking": [ + "github.search_code", + "github.get_code_scanning_alert", + "github.list_code_scanning_alerts", + "github.search_users", + "github.search_orgs", + "github.search_issues", + "github.search_commits", + "github.search_repositories", + "github.search_pull_requests", + "github.update_gist" + ], + "hit1": "github.search_code", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 748, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "kind": "verb_product", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "ranking": [ + "github.list_notifications", + "github.mark_all_notifications_read", + "github.actions_list", + "github.list_gists", + "github.list_tags", + "github.list_releases", + "github.list_discussions", + "github.list_issues", + "github.search_orgs", + "github.create_gist" + ], + "hit1": "github.list_notifications", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 311, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "kind": "verb_product", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "ranking": [ + "github.assign_copilot_to_issue", + "jira.jira_assign_issue", + "jira.jira_update_issue", + "jira.jira_transition_issue", + "github.issue_write", + "gitlab.gitlab_get_issue", + "github.add_issue_comment", + "linear.linear_assign_issue", + "jira.jira_get_issue", + "sentry.sentry_get_issue" + ], + "hit1": "github.assign_copilot_to_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 464, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 838, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "kind": "resource", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "ranking": [ + "github.list_secret_scanning_alerts", + "github.get_secret_scanning_alert", + "github.list_code_scanning_alerts", + "github.get_code_scanning_alert", + "gitlab.get_release", + "aws_s3.s3_get_object", + "github.get_dependabot_alert", + "github.actions_get", + "shopify.shopify_get_product", + "github.list_dependabot_alerts" + ], + "hit1": "github.list_secret_scanning_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 446, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 828, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "ranking": [ + "github.create_or_update_file", + "slack.update_file", + "github.update_gist", + "github.create_gist", + "github.projects_write", + "slack.create_file", + "github.update_pull_request", + "github.update_issue_comment", + "github.issue_write", + "github.create_repository" + ], + "hit1": "github.create_or_update_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 316, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 478, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1038, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "kind": "resource", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "ranking": [ + "github.list_repository_collaborators", + "github.list_commits", + "trello.trello_list_cards", + "github.list_tags", + "github.list_releases", + "github.list_labels", + "github.list_branches", + "github.list_issues", + "github.list_secret_scanning_alerts", + "github.create_pull_request" + ], + "hit1": "github.list_repository_collaborators", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 278, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 420, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 893, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "kind": "verb_product", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "ranking": [ + "slack.slack_reply_to_thread", + "slack.slack_get_thread_replies", + "slack.slack_leave_channel", + "slack.slack_join_channel", + "slack.slack_create_channel", + "slack.slack_get_user_profile", + "slack.slack_add_reaction", + "slack.slack_list_channels", + "slack.slack_upload_file", + "slack.slack_post_message" + ], + "hit1": "slack.slack_reply_to_thread", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "kind": "verb_product", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "ranking": [ + "slack.slack_add_reaction", + "slack.slack_reply_to_thread", + "slack.slack_create_channel", + "discord.discord_add_reaction", + "slack.slack_leave_channel", + "slack.slack_upload_file", + "slack.slack_join_channel", + "slack.slack_get_thread_replies", + "slack.create_reminder", + "slack.create_canvase" + ], + "hit1": "slack.slack_add_reaction", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9238850086262381, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 273, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 413, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 835, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "kind": "verb_product", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "ranking": [ + "slack.slack_search_messages", + "slack.slack_get_channel_history", + "slack.slack_post_message", + "slack.slack_list_users", + "slack.slack_add_reaction", + "slack.slack_list_channels", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_get_users", + "slack.slack_create_channel" + ], + "hit1": "slack.slack_search_messages", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 696, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "kind": "verb_product", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "ranking": [ + "slack.slack_join_channel", + "slack.slack_create_channel", + "slack.slack_leave_channel", + "slack.slack_get_channel_history", + "slack.slack_post_message", + "slack.slack_set_channel_topic", + "slack.slack_add_reaction", + "slack.slack_reply_to_thread", + "slack.slack_upload_file", + "slack.create_usergroup" + ], + "hit1": "slack.slack_join_channel", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "kind": "verb_product", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_transition_issue", + "jira.jira_assign_issue", + "jira.jira_get_issue", + "jira.jira_create_issue", + "jira.jira_update_issue", + "jira.jira_add_comment", + "jira.jira_add_worklog", + "jira.jira_search_issues", + "jira.jira_list_projects", + "jira.jira_create_sprint" + ], + "hit1": "jira.jira_transition_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "kind": "identifier", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "ranking": [ + "jira.jira_search_issues", + "jira.jira_list_projects", + "jira.jira_create_sprint", + "jira.jira_assign_issue", + "jira.jira_add_worklog", + "jira.jira_update_issue", + "jira.jira_create_issue", + "jira.create_sprint", + "jira.list_sprints", + "jira.get_sprint" + ], + "hit1": "jira.jira_search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 394, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 806, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "kind": "verb_product", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "ranking": [ + "jira.jira_add_worklog", + "jira.jira_add_comment", + "jira.jira_list_projects", + "jira.jira_create_sprint", + "jira.jira_assign_issue", + "jira.jira_create_issue", + "jira.jira_update_issue", + "jira.jira_search_issues", + "jira.list_sprints", + "jira.jira_transition_issue" + ], + "hit1": "jira.jira_add_worklog", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 791, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "kind": "verb_product", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "ranking": [ + "linear.linear_list_teams", + "linear.linear_list_projects", + "linear.linear_list_issues", + "linear.linear_create_issue", + "linear.linear_create_project", + "linear.list_labels", + "linear.list_users", + "linear.linear_create_comment", + "linear.linear_search_issues", + "linear.linear_get_issue" + ], + "hit1": "linear.linear_list_teams", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 398, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 722, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "ranking": [ + "linear.linear_create_comment", + "linear.linear_update_issue", + "linear.linear_get_issue", + "linear.get_comment", + "linear.linear_create_issue", + "linear.update_comment", + "linear.linear_assign_issue", + "github.update_issue_comment", + "linear.linear_search_issues", + "linear.create_comment" + ], + "hit1": "linear.linear_create_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 441, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 835, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "kind": "paraphrase", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "ranking": [ + "filesystem.move_file", + "filesystem.read_file", + "filesystem.write_file", + "filesystem.read_media_file", + "filesystem.read_text_file", + "figma.figma_get_file", + "slack.create_file", + "google_drive.gdrive_share_file", + "google_drive.gdrive_update_file", + "brave_search.brave_local_search" + ], + "hit1": "filesystem.move_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 756, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "kind": "paraphrase", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "ranking": [ + "filesystem.search_files", + "slack.list_files", + "filesystem.list_directory", + "filesystem.directory_tree", + "filesystem.move_file", + "filesystem.list_directory_with_sizes", + "github.push_files", + "google_drive.gdrive_search", + "filesystem.read_file", + "notion.notion_search" + ], + "hit1": "filesystem.search_files", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 690, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "kind": "paraphrase", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "ranking": [ + "filesystem.edit_file", + "filesystem.write_file", + "filesystem.list_directory_with_sizes", + "slack.update_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_update_file", + "slack.delete_file", + "slack.create_file", + "figma.figma_get_file", + "github.get_file_blame" + ], + "hit1": "filesystem.edit_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 422, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 814, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "ranking": [ + "memory.read_graph", + "memory.create_entities", + "memory.open_nodes", + "memory.delete_relations", + "memory.delete_entities", + "memory.search_nodes", + "memory.add_observations", + "memory.create_relations", + "github.mark_all_notifications_read", + "filesystem.read_file" + ], + "hit1": "memory.read_graph", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 153, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 254, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 513, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "kind": "verb_product", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "memory.open_nodes", + "brave_search.brave_web_search", + "elasticsearch.es_search", + "brave_search.brave_local_search", + "github.search_orgs", + "notion.notion_search", + "memory.add_observations", + "memory.delete_relations", + "github.search_issues" + ], + "hit1": "memory.search_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 357, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "kind": "verb_product", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "ranking": [ + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_diff", + "git.git_show", + "git.git_reset", + "git.git_checkout", + "git.git_commit", + "git.git_status", + "git.git_branch", + "git.git_log" + ], + "hit1": "git.git_diff_unstaged", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 585, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "kind": "identifier", + "gold": [ + "git.git_add" + ], + "alts": [], + "ranking": [ + "filesystem.list_directory_with_sizes", + "git.git_add", + "github.push_files", + "filesystem.search_files", + "filesystem.list_directory", + "filesystem.move_file", + "git.git_checkout", + "filesystem.read_text_file", + "git.git_commit", + "git.git_diff_staged" + ], + "hit1": "filesystem.list_directory_with_sizes", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "kind": "verb_product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "ranking": [ + "shopify.shopify_create_product", + "shopify.shopify_get_product", + "shopify.shopify_update_product", + "shopify.shopify_create_order", + "shopify.shopify_list_products", + "shopify.shopify_get_order", + "shopify.shopify_list_orders", + "stripe.stripe_create_product", + "shopify.shopify_list_customers", + "github.create_gist" + ], + "hit1": "shopify.shopify_create_product", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 700, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "kind": "verb_product", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "ranking": [ + "stripe.stripe_cancel_subscription", + "stripe.stripe_create_invoice", + "stripe.stripe_create_product", + "stripe.stripe_list_subscriptions", + "stripe.stripe_list_customers", + "stripe.stripe_list_invoices", + "stripe.stripe_create_customer", + "stripe.stripe_create_price", + "stripe.stripe_get_customer", + "stripe.stripe_create_refund" + ], + "hit1": "stripe.stripe_cancel_subscription", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "kind": "resource", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_refund", + "github.create_environment", + "jira.create_attachment", + "jira.create_sprint", + "gitlab.create_wiki", + "github.create_gist", + "linear.create_comment", + "notion.create_comment" + ], + "hit1": "stripe.stripe_create_payment_intent", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 424, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 897, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "github.update_package", + "notion.notion_create_page", + "github.delete_package", + "github.create_package" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "kind": "verb_product", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "ranking": [ + "datadog.datadog_get_monitor", + "datadog.datadog_create_monitor", + "datadog.datadog_list_monitors", + "datadog.datadog_list_logs", + "datadog.datadog_query_metrics", + "github.create_gist", + "github.create_runner", + "jira.create_board", + "gitlab.create_job", + "gitlab.create_snippet" + ], + "hit1": "datadog.datadog_get_monitor", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 928, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "kind": "paraphrase", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "ranking": [ + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "sentry.sentry_list_events", + "sentry.sentry_list_projects", + "sentry.sentry_list_issues", + "github.issue_write", + "pagerduty.pagerduty_resolve_incident", + "gitlab.gitlab_get_issue", + "jira.jira_update_issue", + "github.issue_read" + ], + "hit1": "sentry.sentry_get_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 837, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_delete", + "kubernetes.k8s_apply", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "vercel.vercel_get_deployment", + "kubernetes.k8s_list_pods" + ], + "hit1": "kubernetes.k8s_scale_deployment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_apply", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_delete", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_list_services", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_get_events", + "kubernetes.k8s_list_pods", + "terraform.tf_apply" + ], + "hit1": "kubernetes.k8s_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 253, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "ranking": [ + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_run", + "docker.docker_exec", + "docker.docker_images", + "docker.docker_ps", + "docker.docker_build", + "kubernetes.k8s_pod_logs", + "hubspot.hubspot_get_contact" + ], + "hit1": "docker.docker_stop", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "kind": "verb_product", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "ranking": [ + "docker.docker_build", + "docker.docker_run", + "docker.docker_images", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_ps", + "docker.docker_logs", + "docker.docker_exec", + "filesystem.read_media_file", + "discord.discord_send_message" + ], + "hit1": "docker.docker_build", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 614, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "kind": "verb_product", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "ranking": [ + "playwright.browser_click", + "puppeteer.puppeteer_click", + "playwright.browser_type", + "playwright.browser_take_screenshot", + "playwright.browser_fill_form", + "playwright.browser_tab_list", + "github.create_issue", + "clickup.clickup_list_tasks", + "github.get_environment", + "github.create_environment" + ], + "hit1": "playwright.browser_click", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "kind": "paraphrase", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "ranking": [ + "playwright.browser_type", + "playwright.browser_navigate", + "playwright.browser_wait_for", + "playwright.browser_tab_close", + "playwright.browser_tab_new", + "playwright.browser_tab_list", + "playwright.browser_evaluate", + "playwright.browser_press_key", + "supabase.supabase_insert", + "cloudflare.cf_create_dns" + ], + "hit1": "playwright.browser_type", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 624, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_list_projects", + "github.projects_list", + "gitlab.gitlab_list_issues", + "jira.jira_list_projects", + "gitlab.gitlab_list_merge_requests", + "gitlab.gitlab_create_issue", + "github.projects_get", + "vercel.vercel_list_projects", + "gitlab.gitlab_get_issue", + "gitlab.gitlab_get_file" + ], + "hit1": "gitlab.gitlab_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 209, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 676, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "kind": "verb_product", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "ranking": [ + "bitbucket.bitbucket_create_pr", + "bitbucket.bitbucket_list_prs", + "bitbucket.bitbucket_create_issue", + "github.create_pull_request", + "bitbucket.bitbucket_list_repos", + "github.pull_request_read", + "github.request_copilot_review", + "github.update_pull_request", + "github.pull_request_review_write", + "github.merge_pull_request" + ], + "hit1": "bitbucket.bitbucket_create_pr", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 516, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1097, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "kind": "verb_product", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "ranking": [ + "gmail.gmail_create_draft", + "gmail.gmail_trash_email", + "gmail.gmail_search_emails", + "gmail.gmail_send_email", + "gmail.gmail_get_email", + "github.create_gist", + "github.create_pull_request", + "github.create_artifact", + "gmail.gmail_list_labels", + "jira.create_sprint" + ], + "hit1": "gmail.gmail_create_draft", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 228, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 836, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "kind": "verb_product", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_share_file", + "google_drive.gdrive_update_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_list_folder", + "google_drive.gdrive_search", + "filesystem.write_file", + "github.delete_file", + "figma.figma_get_file", + "gitlab.gitlab_get_file" + ], + "hit1": "google_drive.gdrive_share_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 802, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "kind": "verb_product", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_list_events", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_list_calendars", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "sentry.sentry_list_events", + "kubernetes.k8s_get_events", + "gitlab.gitlab_list_projects", + "google_drive.gdrive_list_folder", + "google_drive.gdrive_get_file" + ], + "hit1": "google_calendar.gcal_list_events", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "github.delete_environment", + "github.delete_repository", + "github.delete_milestone", + "kubernetes.k8s_delete", + "linear.delete_comment" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 739, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_projects", + "vercel.vercel_list_deployments", + "vercel.vercel_create_project", + "vercel.vercel_get_deployment", + "jira.jira_list_projects", + "github.projects_list", + "github.projects_write", + "gitlab.gitlab_list_projects", + "linear.linear_list_projects", + "sentry.sentry_list_projects" + ], + "hit1": "vercel.vercel_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 175, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 269, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 567, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "ranking": [ + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update", + "supabase.supabase_invoke_fn", + "supabase.supabase_list_tables", + "mongodb.mongo_delete", + "figma.figma_get_images", + "aws_s3.s3_get_object", + "memory.delete_entities", + "memory.delete_relations" + ], + "hit1": "supabase.supabase_select", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 688, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "kind": "verb_product", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "figma.figma_get_file", + "figma.figma_get_comments", + "figma.figma_get_images", + "figma.figma_get_file_nodes", + "linear.get_comment", + "linear.update_comment", + "linear.create_comment", + "notion.get_comment", + "linear.delete_comment" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 683, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "kind": "verb_product", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "ranking": [ + "okta.okta_list_users", + "okta.okta_get_user", + "okta.okta_create_user", + "okta.okta_deactivate_user", + "slack.slack_list_users", + "notion.create_users_token", + "slack.slack_get_users", + "notion.list_users_tokens", + "notion.notion_list_users", + "linear.list_users" + ], + "hit1": "okta.okta_list_users", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 298, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 659, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query_range", + "prometheus.prom_query", + "prometheus.prom_list_metrics", + "postgres.explain_query", + "postgres.query", + "datadog.datadog_query_metrics", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "google_drive.gdrive_search", + "shopify.shopify_list_products" + ], + "hit1": "prometheus.prom_query_range", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 668, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "kind": "verb_product", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "ranking": [ + "grafana.grafana_create_annotation", + "grafana.grafana_get_dashboard", + "github.create_gist", + "grafana.grafana_search_dashboards", + "github.create_package", + "gitlab.create_wiki", + "jira.create_attachment", + "github.create_artifact", + "jira.create_board", + "jira.create_sprint" + ], + "hit1": "grafana.grafana_create_annotation", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 391, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 884, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "kind": "identifier", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "ranking": [ + "terraform.tf_apply", + "terraform.tf_plan", + "terraform.tf_state_list", + "kubernetes.k8s_apply", + "github.update_environment", + "github.list_environments", + "slack.slack_reply_to_thread", + "memory.delete_entities", + "github.update_gist", + "github.list_releases" + ], + "hit1": "terraform.tf_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 673, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "kind": "verb_product", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "ranking": [ + "asana.asana_list_tasks", + "asana.asana_get_task", + "asana.asana_create_task", + "asana.asana_update_task", + "asana.asana_create_project", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments", + "aws_s3.s3_list_buckets", + "github.list_environments" + ], + "hit1": "asana.asana_list_tasks", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "ranking": [ + "trello.trello_add_comment", + "trello.trello_create_card", + "trello.trello_list_cards", + "trello.trello_create_board", + "github.update_issue_comment", + "linear.create_comment", + "figma.figma_post_comment", + "linear.get_comment", + "linear.update_comment", + "notion.get_comment" + ], + "hit1": "trello.trello_add_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 422, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 759, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_update_task", + "clickup.clickup_get_task", + "clickup.clickup_create_task", + "clickup.clickup_list_tasks", + "asana.asana_update_task", + "gitlab.update_wiki", + "github.update_runner", + "gitlab.update_snippet", + "github.update_package", + "github.update_artifact" + ], + "hit1": "clickup.clickup_update_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 739, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "kind": "verb_product", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "ranking": [ + "confluence.confluence_search", + "confluence.confluence_get_page", + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "github.search_users", + "elasticsearch.es_search", + "github.search_orgs", + "github.search_commits" + ], + "hit1": "confluence.confluence_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_deal", + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_search_contacts", + "github.create_gist", + "github.create_repository", + "github.create_hook", + "github.create_artifact", + "github.create_or_update_file" + ], + "hit1": "hubspot.hubspot_create_deal", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 862, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_search", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_create_ticket", + "aws_s3.s3_list_buckets", + "kubernetes.k8s_list_deployments", + "jira.list_sprints", + "gitlab.list_snippets", + "notion.list_comments" + ], + "hit1": "zendesk.zendesk_list_tickets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 697, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "kind": "verb_product", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "ranking": [ + "discord.discord_list_channels", + "discord.discord_get_channel", + "discord.discord_list_members", + "discord.discord_send_message", + "discord.discord_add_reaction", + "slack.slack_list_channels", + "trello.trello_list_cards", + "github.projects_list", + "github.actions_list", + "github.list_discussions" + ], + "hit1": "discord.discord_list_channels", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 166, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "ranking": [ + "redis.redis_set", + "redis.redis_get", + "redis.redis_delete", + "redis.redis_keys", + "aws_s3.s3_get_object", + "playwright.browser_press_key", + "filesystem.edit_file", + "jira.jira_get_issue", + "github.get_label", + "slack.slack_set_channel_topic" + ], + "hit1": "redis.redis_set", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 651, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert", + "mongodb.mongo_delete", + "mongodb.mongo_update", + "mongodb.mongo_find", + "mongodb.mongo_list_collections", + "supabase.supabase_insert", + "elasticsearch.es_index_doc", + "stripe.stripe_create_payment_intent", + "github.create_environment", + "okta.okta_list_users" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 766, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "kind": "verb_product", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "mysql.mysql_query", + "postgres.explain_query", + "sqlite.sqlite_query", + "mysql.mysql_list_tables", + "prometheus.prom_query", + "salesforce.salesforce_query", + "datadog.datadog_query_metrics", + "prometheus.prom_query_range", + "memory.search_nodes" + ], + "hit1": "postgres.query", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 650, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_index_doc", + "elasticsearch.es_search", + "elasticsearch.es_get_doc", + "playwright.browser_tab_close", + "github.create_environment", + "github.list_environments", + "github.update_environment", + "gitlab.create_wiki", + "mongodb.mongo_insert", + "github.create_milestone" + ], + "hit1": "elasticsearch.es_index_doc", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 810, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_put_object", + "aws_s3.s3_list_objects", + "okta.okta_get_user", + "github.get_label", + "github.get_milestone", + "twilio.twilio_send_sms", + "gitlab.get_label", + "github.get_environment" + ], + "hit1": "aws_s3.s3_get_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones", + "cloudflare.cf_list_dns", + "cloudflare.cf_purge_cache", + "cloudflare.cf_create_dns", + "kubernetes.k8s_list_deployments", + "clickup.clickup_list_tasks", + "gitlab.list_milestones", + "gitlab.list_releases", + "github.list_milestones", + "gitlab.list_snippets" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 341, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 719, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_create_product", + "stripe.stripe_create_customer", + "stripe.stripe_create_price", + "stripe.stripe_list_invoices", + "stripe.stripe_create_refund", + "stripe.stripe_create_payment_intent", + "stripe.stripe_cancel_subscription", + "stripe.stripe_list_subscriptions", + "stripe.stripe_get_customer" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 333, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 654, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "kind": "verb_product", + "gold": [ + "github.actions_list" + ], + "alts": [], + "ranking": [ + "github.actions_list", + "github.actions_get", + "github.actions_run_trigger", + "github.actions_get_job_logs", + "github.list_tags", + "github.list_discussions", + "github.list_branches", + "github.projects_list", + "github.list_gists", + "github.list_releases" + ], + "hit1": "github.actions_list", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 457, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 790, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "kind": "verb_product", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "ranking": [ + "github.actions_get_job_logs", + "github.actions_get", + "github.actions_list", + "github.actions_run_trigger", + "github.get_label", + "github.get_discussion", + "github.get_latest_release", + "github.get_tag", + "github.get_commit", + "github.search_issues" + ], + "hit1": "github.actions_get_job_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 471, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 884, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "kind": "verb_product", + "gold": [ + "github.create_repository" + ], + "alts": [], + "ranking": [ + "github.create_repository", + "github.delete_repository", + "github.fork_repository", + "github.star_repository", + "github.unstar_repository", + "github.list_repository_collaborators", + "github.get_repository_tree", + "github.create_branch", + "github.create_issue", + "github.create_or_update_file" + ], + "hit1": "github.create_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 949, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "kind": "verb_product", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "ranking": [ + "git.git_checkout", + "git.git_create_branch", + "git.git_branch", + "github.create_branch", + "git.git_diff", + "git.git_show", + "github.list_commits", + "github.delete_file", + "github.update_pull_request_branch", + "git.git_status" + ], + "hit1": "git.git_checkout", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 376, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 801, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "kind": "identifier", + "gold": [ + "git.git_log" + ], + "alts": [], + "ranking": [ + "git.git_log", + "git.git_show", + "git.git_diff", + "git.git_branch", + "git.git_status", + "git.git_checkout", + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_commit", + "git.git_create_branch" + ], + "hit1": "git.git_log", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 304, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 619, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "kind": "verb_product", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "ranking": [ + "postgres.list_schemas", + "postgres.list_tables", + "postgres.query", + "postgres.describe_table", + "postgres.explain_query", + "github.projects_list", + "kubernetes.k8s_list_services", + "kubernetes.k8s_list_deployments", + "github.actions_list", + "jira.list_attachments" + ], + "hit1": "postgres.list_schemas", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 158, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 595, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "kind": "verb_product", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "ranking": [ + "memory.open_nodes", + "memory.read_graph", + "memory.search_nodes", + "memory.create_entities", + "memory.add_observations", + "memory.create_relations", + "memory.delete_relations", + "memory.delete_entities", + "sqlite.sqlite_list_tables", + "postgres.list_tables" + ], + "hit1": "memory.open_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 267, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 504, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "kind": "paraphrase", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "ranking": [ + "filesystem.read_file", + "filesystem.read_media_file", + "filesystem.read_text_file", + "filesystem.get_file_info", + "filesystem.write_file", + "google_drive.gdrive_get_file", + "filesystem.move_file", + "figma.figma_get_file", + "figma.figma_get_file_nodes", + "gitlab.gitlab_get_file" + ], + "hit1": "filesystem.read_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 426, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "kind": "paraphrase", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "ranking": [ + "filesystem.write_file", + "playwright.browser_tab_new", + "filesystem.read_media_file", + "filesystem.read_file", + "filesystem.read_text_file", + "figma.figma_get_file", + "google_drive.gdrive_share_file", + "google_drive.gdrive_update_file", + "slack.update_file", + "slack.create_file" + ], + "hit1": "filesystem.write_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 199, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "kind": "verb_product", + "gold": [ + "github.search_issues" + ], + "alts": [], + "ranking": [ + "github.search_issues", + "github.list_issues", + "github.search_orgs", + "github.search_users", + "github.search_pull_requests", + "jira.jira_search_issues", + "github.search_commits", + "github.search_code", + "linear.linear_search_issues", + "gitlab.gitlab_list_issues" + ], + "hit1": "github.search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 683, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "kind": "paraphrase", + "gold": [ + "github.get_me" + ], + "alts": [], + "ranking": [ + "github.get_teams", + "github.get_me", + "github.list_notifications", + "github.get_latest_release", + "github.list_gists", + "github.get_gist", + "github.get_label", + "github.get_repository_tree", + "github.get_team_members", + "github.get_commit" + ], + "hit1": "github.get_teams", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 693, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "cost": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.0133333333333334, + "p95_calls": 1.0, + "success_within_1": 0.9866666666666667, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 239.23333333333332, + "p95_tokens": 324.7999999999997, + "total_tokens": 35885, + "tokens_per_successful_query": 239.23333333333332, + "total_tokens_successful": 35885 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.0066666666666666, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 379.00666666666666, + "p95_tokens": 511.49999999999983, + "total_tokens": 56851, + "tokens_per_successful_query": 379.00666666666666, + "total_tokens_successful": 56851 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.0, + "p95_calls": 1.0, + "success_within_1": 1.0, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 754.9066666666666, + "p95_tokens": 1007.7999999999994, + "total_tokens": 113236, + "tokens_per_successful_query": 754.9066666666666, + "total_tokens_successful": 113236 + } + } +} diff --git a/experiments/mcp-tool-search/results/hybrid_rrf_weighted_trigram.csv b/experiments/mcp-tool-search/results/hybrid_rrf_weighted_trigram.csv new file mode 100644 index 00000000..2a6d2fd8 --- /dev/null +++ b/experiments/mcp-tool-search/results/hybrid_rrf_weighted_trigram.csv @@ -0,0 +1,151 @@ +id,query,kind,hit1,recall@1,recall@3,recall@5,mrr@10,ndcg@10,hit_rank,k3_calls,k3_tokens,k3_success1,k5_calls,k5_tokens,k5_success1,k10_calls,k10_tokens,k10_success1,ranking +q001,Create a GitHub issue,verb_product,github.create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,388,1.0,1,584,1.0,1,1038,1.0,github.create_issue github.issue_write github.issue_read github.update_issue_comment gitlab.gitlab_create_issue github.add_issue_comment github.create_gist github.create_branch github.create_repository bitbucket.bitbucket_create_issue +q002,Create a Jira ticket,verb_product,jira.jira_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,333,1.0,1,528,1.0,1,1003,1.0,jira.jira_create_issue jira.jira_create_sprint zendesk.zendesk_create_ticket jira.create_board jira.create_sprint jira.create_filter jira.create_version jira.create_webhook jira.create_attachment zendesk.zendesk_update_ticket +q003,Send a Slack message,verb_product,slack.slack_post_message,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,353,1.0,1,693,1.0,slack.slack_post_message slack.slack_add_reaction slack.slack_search_messages slack.slack_join_channel slack.slack_leave_channel slack.slack_get_channel_history discord.discord_send_message slack.slack_create_channel slack.slack_set_channel_topic slack.slack_get_user_profile +q004,List open pull requests,resource,github.list_pull_requests,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,474,1.0,1,993,1.0,github.list_pull_requests github.search_pull_requests gitlab.gitlab_list_merge_requests github.create_pull_request github.pull_request_read bitbucket.bitbucket_list_prs github.merge_pull_request github.update_pull_request github.update_pull_request_branch github.request_copilot_review +q005,Open a pull request on GitHub,verb_product,github.create_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,368,1.0,1,522,1.0,1,1144,1.0,github.create_pull_request github.pull_request_read github.merge_pull_request github.request_copilot_review github.search_pull_requests github.update_pull_request github.update_pull_request_branch github.list_pull_requests github.pull_request_review_write github.add_comment_to_pending_review +q006,Run a SQL query against postgres,verb_product,postgres.query,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,272,1.0,1,590,1.0,postgres.query postgres.explain_query mysql.mysql_query postgres.list_schemas sqlite.sqlite_query postgres.list_tables salesforce.salesforce_query postgres.describe_table prometheus.prom_query docker.docker_run +q007,Read a local file from disk,paraphrase,filesystem.read_file,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,428,1.0,1,781,1.0,filesystem.read_file filesystem.read_text_file filesystem.read_media_file gitlab.gitlab_get_file github.delete_file figma.figma_get_file_nodes figma.figma_get_file filesystem.get_file_info filesystem.move_file github.get_file_contents +q008,What files are in this folder,paraphrase,google_drive.gdrive_list_folder,0.0,0.0,0.0,0.1111,0.1923,9,3,664,0.0,2,703,0.0,1,681,1.0,google_drive.gdrive_list_folder slack.list_files github.push_files filesystem.list_allowed_directories google_drive.gdrive_search cloudflare.cf_purge_cache filesystem.search_files git.git_add filesystem.list_directory filesystem.move_file +q009,Remember this fact in the knowledge graph,paraphrase,memory.read_graph,0.0,1.0,1.0,0.5000,0.5106,2,1,154,1.0,1,255,1.0,1,506,1.0,memory.read_graph memory.create_entities memory.delete_relations memory.open_nodes memory.create_relations memory.add_observations memory.search_nodes memory.delete_entities time.get_current_time sqlite.sqlite_list_tables +q010,Search the web with Brave,verb_product,brave_search.brave_web_search,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,410,1.0,1,705,1.0,brave_search.brave_web_search brave_search.brave_local_search jira.jira_search_issues github.update_pull_request_branch memory.search_nodes github.search_users github.search_orgs github.search_commits github.search_issues confluence.confluence_search +q011,Fetch the contents of a URL,verb_product,fetch.fetch,1.0,1.0,1.0,1.0000,1.0000,1,1,222,1.0,1,380,1.0,1,722,1.0,fetch.fetch filesystem.read_file git.git_add playwright.browser_navigate github.get_file_contents github.get_me git.git_status slack.slack_set_channel_topic github.get_repository_tree puppeteer.puppeteer_navigate +q012,Show git status,identifier,git.git_status,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,301,1.0,1,662,1.0,git.git_status git.git_diff_staged git.git_show git.git_diff_unstaged git.git_diff git.git_log git.git_add sentry.sentry_update_issue shopify.shopify_list_orders gitlab.list_snippets +q013,Commit staged changes,paraphrase,git.git_commit,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,305,1.0,1,725,1.0,git.git_commit git.git_reset git.git_diff_staged git.git_show git.git_log github.get_commit github.push_files linear.list_comments notion.list_comments github.list_packages +q014,Create a Linear issue,verb_product,linear.linear_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,340,1.0,1,463,1.0,1,925,1.0,linear.linear_create_issue linear.linear_update_issue linear.linear_assign_issue linear.linear_create_comment linear.linear_get_issue linear.linear_create_project linear.create_user linear.create_label linear.create_cycle linear.create_comment +q015,Make a task in my project tracker Linear,paraphrase,linear.linear_create_project,0.0,1.0,1.0,0.3333,0.5000,3,1,284,1.0,1,386,1.0,1,638,1.0,linear.linear_create_project linear.linear_list_projects linear.linear_create_issue linear.delete_cycle linear.get_cycle linear.delete_user linear.get_user linear.linear_list_teams linear.delete_label linear.update_user +q016,Post a comment on a GitHub issue,verb_product,github.update_issue_comment,0.0,1.0,1.0,0.5000,0.6309,2,1,333,1.0,1,510,1.0,1,1006,1.0,github.update_issue_comment github.add_issue_comment github.issue_read github.assign_copilot_to_issue figma.figma_post_comment github.create_issue jira.jira_add_comment github.get_discussion_comments linear.linear_create_comment github.issue_write +q017,Merge the pull request,ambiguous,github.merge_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,299,1.0,1,468,1.0,1,987,1.0,github.merge_pull_request gitlab.gitlab_merge_merge_request github.update_pull_request_branch gitlab.gitlab_get_merge_request github.pull_request_read github.search_pull_requests github.update_pull_request gitlab.gitlab_list_merge_requests github.create_pull_request github.request_copilot_review +q018,Trigger a GitHub Actions workflow,verb_product,github.actions_run_trigger,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,470,1.0,1,842,1.0,github.actions_run_trigger github.actions_get github.actions_list github.actions_get_job_logs github.fork_repository github.star_repository github.unstar_repository github.get_discussion github.delete_repository github.update_gist +q019,Get Dependabot alerts,resource,github.list_dependabot_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,418,1.0,1,675,1.0,github.list_dependabot_alerts github.get_dependabot_alert github.list_secret_scanning_alerts github.list_code_scanning_alerts github.get_hook gitlab.get_job gitlab.get_wiki redis.redis_get github.get_package github.get_runner +q020,List Sentry issues,verb_product,sentry.sentry_list_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,396,1.0,1,796,1.0,sentry.sentry_list_issues sentry.sentry_list_projects sentry.sentry_list_events sentry.sentry_get_issue sentry.sentry_update_issue gitlab.gitlab_list_issues linear.linear_search_issues github.list_issues github.search_issues linear.linear_list_issues +q021,Create a Stripe customer,verb_product,stripe.stripe_create_invoice,0.0,1.0,1.0,0.5000,0.6309,2,1,186,1.0,1,322,1.0,1,669,1.0,stripe.stripe_create_invoice stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_list_invoices stripe.stripe_create_payment_intent stripe.stripe_create_price stripe.stripe_create_product stripe.stripe_list_subscriptions stripe.stripe_list_customers stripe.stripe_create_refund +q022,Refund a Stripe payment,paraphrase,stripe.stripe_create_refund,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,352,1.0,1,662,1.0,stripe.stripe_create_refund stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent stripe.stripe_get_customer stripe.stripe_create_customer stripe.stripe_create_price stripe.stripe_create_product stripe.stripe_create_invoice stripe.stripe_list_customers stripe.stripe_list_invoices +q023,Send an email with Gmail,verb_product,gmail.gmail_send_email,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,349,1.0,1,713,1.0,gmail.gmail_send_email gmail.gmail_get_email gmail.gmail_search_emails gmail.gmail_trash_email puppeteer.puppeteer_click twilio.twilio_send_sms gmail.gmail_list_labels gmail.gmail_modify_email puppeteer.puppeteer_fill gmail.gmail_create_draft +q024,Search my Gmail inbox,verb_product,gmail.gmail_search_emails,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,294,1.0,1,617,1.0,gmail.gmail_search_emails gmail.gmail_list_labels elasticsearch.es_search github.search_users confluence.confluence_search gmail.gmail_send_email google_drive.gdrive_search slack.slack_search_messages gmail.gmail_get_email github.search_orgs +q025,Upload a file to Google Drive,paraphrase,google_drive.gdrive_update_file,0.0,0.0,1.0,0.2000,0.3869,5,2,496,0.0,1,418,1.0,1,702,1.0,google_drive.gdrive_update_file google_drive.gdrive_share_file google_drive.gdrive_search google_drive.gdrive_get_file google_drive.gdrive_create_file google_drive.gdrive_list_folder slack.slack_upload_file figma.figma_get_file google_calendar.gcal_list_calendars slack.delete_file +q026,Create a calendar event,resource,google_calendar.gcal_delete_event,0.0,1.0,1.0,0.5000,0.6309,2,1,300,1.0,1,427,1.0,1,909,1.0,google_calendar.gcal_delete_event google_calendar.gcal_create_event google_calendar.gcal_update_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars linear.create_comment jira.create_attachment github.create_environment jira.create_sprint notion.create_comment +q027,List Kubernetes pods,verb_product,kubernetes.k8s_list_pods,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,314,1.0,1,738,1.0,kubernetes.k8s_list_pods kubernetes.k8s_list_services kubernetes.k8s_list_deployments kubernetes.k8s_get_events kubernetes.k8s_get_pod kubernetes.k8s_apply kubernetes.k8s_pod_logs kubernetes.k8s_delete kubernetes.k8s_scale_deployment linear.list_cycles +q028,Get logs from a k8s pod,verb_product,kubernetes.k8s_pod_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,262,1.0,1,441,1.0,1,848,1.0,kubernetes.k8s_pod_logs kubernetes.k8s_get_pod docker.docker_logs github.actions_get_job_logs github.get_tag gitlab.gitlab_get_file datadog.datadog_list_logs elasticsearch.es_get_doc figma.figma_get_file_nodes aws_s3.s3_get_object +q029,Run a docker container,verb_product,docker.docker_run,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,345,1.0,1,640,1.0,docker.docker_run docker.docker_stop docker.docker_rm docker.docker_exec docker.docker_logs docker.docker_ps docker.docker_build docker.docker_images kubernetes.k8s_pod_logs mysql.mysql_query +q030,Show docker container logs,verb_product,docker.docker_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,345,1.0,1,649,1.0,docker.docker_logs docker.docker_stop docker.docker_rm docker.docker_run docker.docker_exec docker.docker_ps docker.docker_images kubernetes.k8s_pod_logs docker.docker_build git.git_log +q031,Navigate the browser to a URL,ambiguous,playwright.browser_navigate,1.0,1.0,1.0,1.0000,0.9502,1,1,180,1.0,1,280,1.0,1,574,1.0,playwright.browser_navigate playwright.browser_tab_new puppeteer.puppeteer_navigate playwright.browser_tab_close playwright.browser_evaluate playwright.browser_press_key playwright.browser_tab_list playwright.browser_click playwright.browser_wait_for playwright.browser_type +q032,Take a screenshot of the page,ambiguous,playwright.browser_take_screenshot,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,372,1.0,1,675,1.0,playwright.browser_take_screenshot puppeteer.puppeteer_screenshot playwright.browser_snapshot confluence.confluence_create_page notion.notion_update_page confluence.confluence_get_page playwright.browser_press_key notion.notion_get_page playwright.browser_evaluate confluence.confluence_update_page +q033,Create a GitLab merge request,verb_product,gitlab.gitlab_merge_merge_request,0.0,1.0,1.0,0.5000,0.6309,2,1,297,1.0,1,475,1.0,1,1054,1.0,gitlab.gitlab_merge_merge_request gitlab.gitlab_create_merge_request gitlab.gitlab_get_merge_request github.merge_pull_request gitlab.gitlab_list_merge_requests gitlab.gitlab_create_pipeline gitlab.gitlab_create_issue github.pull_request_review_write github.create_pull_request bitbucket.bitbucket_create_pr +q034,Search Notion pages,verb_product,notion.notion_search,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,353,1.0,1,635,1.0,notion.notion_search notion.notion_get_page notion.notion_update_page notion.notion_get_comments notion.notion_create_page notion.notion_get_block notion.notion_create_database notion.get_comment notion.notion_list_users github.search_orgs +q035,Create a Notion page,verb_product,notion.notion_get_page,0.0,1.0,1.0,0.5000,0.6309,2,1,230,1.0,1,371,1.0,1,832,1.0,notion.notion_get_page notion.notion_create_page notion.notion_update_page notion.notion_create_database notion.notion_get_comments notion.create_comment notion.create_users_token notion.create_data_source confluence.confluence_create_page notion.update_comment +q036,Query a Notion database,verb_product,notion.notion_query_database,1.0,1.0,1.0,1.0000,1.0000,1,1,261,1.0,1,401,1.0,1,688,1.0,notion.notion_query_database notion.notion_create_database notion.notion_search notion.notion_create_page notion.notion_get_block notion.notion_get_page notion.notion_update_page notion.delete_data_source notion.get_data_source notion.delete_comment +q037,Create a Zendesk ticket,verb_product,zendesk.zendesk_create_ticket,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,368,1.0,1,882,1.0,zendesk.zendesk_create_ticket zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_search zendesk.zendesk_list_tickets jira.jira_create_issue gitlab.create_snippet gitlab.create_job github.create_hook gitlab.create_wiki +q038,Create a HubSpot contact,verb_product,hubspot.hubspot_create_contact,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,364,1.0,1,867,1.0,hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_list_contacts hubspot.hubspot_create_deal hubspot.hubspot_search_contacts github.create_gist github.create_artifact github.create_branch github.create_or_update_file github.create_repository +q039,Create a Salesforce lead,verb_product,salesforce.salesforce_convert_lead,0.0,1.0,1.0,0.5000,0.6309,2,1,252,1.0,1,402,1.0,1,787,1.0,salesforce.salesforce_convert_lead salesforce.salesforce_create_lead salesforce.salesforce_create_record salesforce.salesforce_update_record salesforce.salesforce_query salesforce.salesforce_delete_record salesforce.salesforce_get_record salesforce.salesforce_list_opportunities github.create_gist github.create_milestone +q040,Run a SOQL query,identifier,salesforce.salesforce_query,1.0,1.0,1.0,1.0000,1.0000,1,1,187,1.0,1,294,1.0,1,631,1.0,salesforce.salesforce_query postgres.query mysql.mysql_query sqlite.sqlite_query postgres.explain_query prometheus.prom_query notion.notion_query_database memory.search_nodes github.search_users docker.docker_run +q041,List Shopify orders,verb_product,shopify.shopify_list_orders,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,320,1.0,1,703,1.0,shopify.shopify_list_orders shopify.shopify_list_customers shopify.shopify_list_products shopify.shopify_get_order shopify.shopify_get_product shopify.shopify_create_order shopify.shopify_update_product shopify.shopify_create_product github.list_runners linear.list_users +q042,Create a PagerDuty incident,verb_product,pagerduty.pagerduty_create_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,403,1.0,1,848,1.0,pagerduty.pagerduty_create_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents github.create_package github.create_environment pagerduty.pagerduty_list_oncalls gitlab.create_pipeline jira.create_attachment +q043,Who is on call in PagerDuty,paraphrase,pagerduty.pagerduty_list_oncalls,1.0,1.0,1.0,1.0000,1.0000,1,1,227,1.0,1,363,1.0,1,764,1.0,pagerduty.pagerduty_list_oncalls pagerduty.pagerduty_get_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_create_incident notion.notion_get_comments playwright.browser_press_key twilio.twilio_make_call github.list_commits +q044,Query Datadog metrics,verb_product,datadog.datadog_query_metrics,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,458,1.0,1,799,1.0,datadog.datadog_query_metrics datadog.datadog_list_monitors datadog.datadog_create_monitor datadog.datadog_list_logs datadog.datadog_get_monitor prometheus.prom_list_metrics prometheus.prom_query notion.notion_query_database prometheus.prom_query_range postgres.explain_query +q045,Search Datadog logs,verb_product,datadog.datadog_list_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,412,1.0,1,749,1.0,datadog.datadog_list_logs datadog.datadog_list_monitors datadog.datadog_get_monitor datadog.datadog_query_metrics notion.notion_search github.search_orgs datadog.datadog_create_monitor confluence.confluence_search github.search_users elasticsearch.es_search +q046,List S3 buckets,verb_product,aws_s3.s3_list_buckets,1.0,1.0,1.0,1.0000,1.0000,1,1,213,1.0,1,322,1.0,1,704,1.0,aws_s3.s3_list_buckets aws_s3.s3_list_objects gitlab.list_snippets clickup.clickup_list_tasks kubernetes.k8s_list_deployments jira.list_sprints linear.list_comments github.list_artifacts github.list_environments linear.list_attachments +q047,Upload an object to S3,paraphrase,aws_s3.s3_put_object,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,426,1.0,1,850,1.0,aws_s3.s3_put_object aws_s3.s3_delete_object aws_s3.s3_get_object slack.slack_upload_file gmail.gmail_send_email aws_s3.s3_list_objects twilio.twilio_send_sms github.assign_copilot_to_issue slack.slack_add_reaction asana.asana_create_project +q048,Purge Cloudflare cache,verb_product,cloudflare.cf_purge_cache,1.0,1.0,1.0,1.0000,1.0000,1,1,231,1.0,1,335,1.0,1,548,1.0,cloudflare.cf_purge_cache cloudflare.cf_list_zones cloudflare.cf_create_dns cloudflare.cf_list_dns playwright.browser_tab_close playwright.browser_snapshot postgres.list_tables memory.read_graph postgres.list_schemas puppeteer.puppeteer_click +q049,Create a Cloudflare DNS record,verb_product,cloudflare.cf_create_dns,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,356,1.0,1,810,1.0,cloudflare.cf_create_dns cloudflare.cf_list_dns cloudflare.cf_list_zones salesforce.salesforce_create_record cloudflare.cf_purge_cache salesforce.salesforce_update_record salesforce.salesforce_delete_record jira.create_board jira.create_sprint jira.create_filter +q050,List Vercel deployments,verb_product,vercel.vercel_list_deployments,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,291,1.0,1,636,1.0,vercel.vercel_list_deployments vercel.vercel_list_projects vercel.vercel_get_deployment kubernetes.k8s_list_deployments vercel.vercel_create_project github.list_environments aws_s3.s3_list_buckets linear.list_comments jira.list_attachments linear.list_attachments +q051,Insert a row into Supabase,verb_product,supabase.supabase_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,389,1.0,1,672,1.0,supabase.supabase_insert supabase.supabase_select supabase.supabase_update mongodb.mongo_insert supabase.supabase_invoke_fn supabase.supabase_list_tables playwright.browser_type gitlab.get_release slack.get_canvase gitlab.delete_release +q052,Export images from Figma,paraphrase,figma.figma_get_images,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,328,1.0,1,701,1.0,figma.figma_get_images figma.figma_get_file_nodes figma.figma_get_file docker.docker_images figma.figma_get_comments figma.figma_post_comment supabase.supabase_select slack.slack_get_channel_history docker.docker_logs kubernetes.k8s_pod_logs +q053,Send an SMS with Twilio,verb_product,twilio.twilio_send_sms,1.0,1.0,1.0,1.0000,1.0000,1,1,215,1.0,1,396,1.0,1,740,1.0,twilio.twilio_send_sms twilio.twilio_list_messages puppeteer.puppeteer_click gmail.gmail_send_email twilio.twilio_make_call filesystem.list_directory_with_sizes puppeteer.puppeteer_fill jira.jira_search_issues discord.discord_send_message puppeteer.puppeteer_screenshot +q054,Create an Okta user,verb_product,okta.okta_get_user,0.0,1.0,1.0,0.5000,0.6309,2,1,183,1.0,1,329,1.0,1,753,1.0,okta.okta_get_user okta.okta_create_user okta.okta_deactivate_user okta.okta_list_users linear.create_user linear.update_user linear.delete_user asana.asana_create_project asana.asana_create_task github.create_issue +q055,Instant query Prometheus,verb_product,prometheus.prom_query,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,327,1.0,1,666,1.0,prometheus.prom_query prometheus.prom_query_range prometheus.prom_list_metrics postgres.query mysql.mysql_query salesforce.salesforce_query sqlite.sqlite_query postgres.explain_query datadog.datadog_query_metrics shopify.shopify_list_products +q056,Search Grafana dashboards,verb_product,grafana.grafana_search_dashboards,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,288,1.0,1,663,1.0,grafana.grafana_search_dashboards grafana.grafana_get_dashboard github.search_users github.search_orgs confluence.confluence_search github.search_issues grafana.grafana_create_annotation slack.slack_search_messages elasticsearch.es_search notion.notion_search +q057,Trigger a CircleCI pipeline,verb_product,circleci.circleci_trigger_pipeline,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,325,1.0,1,755,1.0,circleci.circleci_trigger_pipeline circleci.circleci_get_pipeline circleci.circleci_list_pipelines gitlab.gitlab_create_pipeline gitlab.get_pipeline github.actions_run_trigger gitlab.create_pipeline gitlab.delete_pipeline gitlab.update_pipeline hubspot.hubspot_create_deal +q058,Run terraform plan,identifier,terraform.tf_plan,1.0,1.0,1.0,1.0000,1.0000,1,1,172,1.0,1,287,1.0,1,593,1.0,terraform.tf_plan terraform.tf_apply terraform.tf_state_list postgres.explain_query salesforce.salesforce_query github.get_check_run docker.docker_run github.delete_check_run mysql.mysql_query sqlite.sqlite_query +q059,Create an Asana task,verb_product,asana.asana_get_task,0.0,1.0,1.0,0.5000,0.6309,2,1,267,1.0,1,429,1.0,1,841,1.0,asana.asana_get_task asana.asana_create_task asana.asana_update_task asana.asana_create_project clickup.clickup_create_task asana.asana_list_tasks okta.okta_create_user clickup.clickup_update_task github.create_issue filesystem.write_file +q060,Create a Trello card,verb_product,trello.trello_create_card,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,411,1.0,1,888,1.0,trello.trello_create_card trello.trello_add_comment trello.trello_create_board trello.trello_list_cards jira.create_board github.create_gist gitlab.create_release gitlab.create_job github.create_hook slack.create_canvase +q061,Create a ClickUp task,verb_product,clickup.clickup_get_task,0.0,1.0,1.0,0.5000,0.6309,2,1,257,1.0,1,415,1.0,1,878,1.0,clickup.clickup_get_task clickup.clickup_create_task clickup.clickup_update_task clickup.clickup_list_tasks asana.asana_create_task github.create_gist jira.create_board jira.create_sprint jira.create_filter jira.create_version +q062,Create a Confluence page,verb_product,confluence.confluence_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,401,1.0,1,833,1.0,confluence.confluence_create_page confluence.confluence_get_page confluence.confluence_update_page confluence.confluence_search notion.notion_create_page notion.notion_update_page github.create_package github.create_gist jira.create_board linear.create_comment +q063,Send a Discord message,verb_product,discord.discord_send_message,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,343,1.0,1,656,1.0,discord.discord_send_message discord.discord_add_reaction discord.discord_get_channel discord.discord_list_members slack.slack_post_message gmail.gmail_get_email discord.discord_list_channels gmail.gmail_trash_email git.git_commit slack.slack_add_reaction +q064,Search Elasticsearch,verb_product,elasticsearch.es_search,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,343,1.0,1,660,1.0,elasticsearch.es_search elasticsearch.es_index_doc elasticsearch.es_get_doc github.search_users confluence.confluence_search zendesk.zendesk_search github.search_orgs github.search_commits slack.slack_search_messages github.search_issues +q065,Get a Redis key,verb_product,redis.redis_get,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,281,1.0,1,627,1.0,redis.redis_get redis.redis_set redis.redis_delete redis.redis_keys jira.jira_get_issue figma.figma_get_file github.get_discussion aws_s3.s3_get_object github.get_label github.get_runner +q066,Find MongoDB documents,verb_product,mongodb.mongo_find,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,381,1.0,1,742,1.0,mongodb.mongo_find mongodb.mongo_update mongodb.mongo_insert mongodb.mongo_delete mongodb.mongo_list_collections elasticsearch.es_index_doc elasticsearch.es_get_doc figma.figma_get_comments github.get_discussion_comments filesystem.read_file +q067,List SQLite tables,verb_product,sqlite.sqlite_list_tables,1.0,1.0,1.0,1.0000,1.0000,1,1,151,1.0,1,240,1.0,1,536,1.0,sqlite.sqlite_list_tables mysql.mysql_list_tables postgres.list_tables sqlite.sqlite_describe_table supabase.supabase_list_tables sqlite.sqlite_query postgres.list_schemas clickup.clickup_list_tasks github.list_milestones linear.list_cycles +q068,Describe a postgres table,verb_product,postgres.describe_table,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,290,1.0,1,640,1.0,postgres.describe_table sqlite.sqlite_describe_table postgres.list_tables postgres.query postgres.explain_query postgres.list_schemas supabase.supabase_insert supabase.supabase_update supabase.supabase_select slack.create_pin +q069,github_create_issue,identifier,github.create_issue,1.0,1.0,1.0,1.0000,0.9072,1,1,355,1.0,1,582,1.0,1,1050,1.0,github.create_issue github.update_issue_comment github.issue_read gitlab.gitlab_create_issue github.issue_write bitbucket.bitbucket_create_issue github.add_issue_comment github.assign_copilot_to_issue github.create_hook gitlab.gitlab_get_issue +q070,createIssue,identifier,bitbucket.bitbucket_create_issue,0.0,1.0,1.0,0.5000,0.6404,2,1,332,1.0,1,600,1.0,1,999,1.0,bitbucket.bitbucket_create_issue github.create_issue gitlab.gitlab_create_issue github.issue_write linear.linear_create_issue jira.jira_create_issue linear.linear_create_comment github.issue_read jira.jira_update_issue jira.jira_get_issue +q071,list-directory,identifier,filesystem.list_directory,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,281,1.0,1,643,1.0,filesystem.list_directory filesystem.create_directory filesystem.list_directory_with_sizes filesystem.directory_tree filesystem.list_allowed_directories filesystem.get_file_info github.get_file_contents filesystem.move_file github.projects_list linear.list_cycles +q072,sequential thinking next thought,paraphrase,sequential_thinking.sequentialthinking,1.0,1.0,1.0,1.0000,1.0000,1,1,278,1.0,1,434,1.0,1,862,1.0,sequential_thinking.sequentialthinking filesystem.edit_file confluence.confluence_search playwright.browser_wait_for brave_search.brave_web_search playwright.browser_type brave_search.brave_local_search github.add_comment_to_pending_review github.list_notifications memory.add_observations +q073,What time is it in Tokyo,paraphrase,time.get_current_time,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,392,1.0,1,876,1.0,time.get_current_time time.convert_time playwright.browser_wait_for grafana.grafana_create_annotation jira.jira_add_worklog google_calendar.gcal_list_events github.create_milestone github.create_package github.create_environment github.create_artifact +q074,Convert time between timezones,verb_product,time.convert_time,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,319,1.0,1,687,1.0,time.convert_time time.get_current_time salesforce.salesforce_convert_lead git.git_diff memory.create_relations playwright.browser_wait_for grafana.grafana_create_annotation google_calendar.gcal_list_events jira.jira_add_worklog cloudflare.cf_list_zones +q075,Star a GitHub repo,verb_product,github.star_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,226,1.0,1,379,1.0,1,789,1.0,github.star_repository github.unstar_repository github.delete_repository github.list_tags github.get_label github.fork_repository github.get_tag github.list_issues github.list_labels github.list_branches +q076,Fork a repository,resource,github.fork_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,235,1.0,1,378,1.0,1,758,1.0,github.fork_repository github.delete_repository github.star_repository github.unstar_repository github.list_releases github.list_dependabot_alerts github.create_repository github.list_discussions github.list_branches github.list_repository_collaborators +q077,Get the latest GitHub release,verb_product,github.get_latest_release,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,383,1.0,1,725,1.0,github.get_latest_release github.get_release_by_tag gitlab.get_release github.get_teams github.get_repository_tree github.get_gist github.get_label github.get_tag gitlab.update_release gitlab.delete_release +q078,Create a gist,resource,github.create_gist,1.0,1.0,1.0,1.0000,1.0000,1,1,272,1.0,1,473,1.0,1,966,1.0,github.create_gist github.update_gist github.create_hook gitlab.create_job gitlab.create_wiki github.create_runner github.create_package github.create_milestone gitlab.create_label github.create_artifact +q079,Search GitHub code,verb_product,github.search_code,1.0,1.0,1.0,1.0000,1.0000,1,1,283,1.0,1,386,1.0,1,746,1.0,github.search_code github.get_code_scanning_alert github.list_code_scanning_alerts github.search_users github.search_commits github.search_orgs github.search_issues github.search_pull_requests github.search_repositories github.create_gist +q080,List GitHub notifications,verb_product,github.list_notifications,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,326,1.0,1,639,1.0,github.list_notifications github.mark_all_notifications_read github.actions_list github.projects_list github.list_tags github.list_discussions github.list_gists github.list_releases git.git_branch github.list_branches +q081,Assign Copilot to an issue,verb_product,github.assign_copilot_to_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,468,1.0,1,878,1.0,github.assign_copilot_to_issue jira.jira_assign_issue linear.linear_assign_issue github.create_issue jira.jira_transition_issue github.add_issue_comment jira.jira_update_issue github.request_copilot_review sentry.sentry_get_issue jira.jira_add_comment +q082,Get secret scanning alerts,resource,github.list_secret_scanning_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,295,1.0,1,470,1.0,1,733,1.0,github.list_secret_scanning_alerts github.get_secret_scanning_alert github.list_code_scanning_alerts github.get_code_scanning_alert github.list_dependabot_alerts github.get_hook gitlab.get_job gitlab.get_wiki github.get_runner github.get_package +q083,Create or update a file on GitHub,verb_product,github.create_or_update_file,1.0,1.0,1.0,1.0000,0.7602,1,1,333,1.0,1,527,1.0,1,1127,1.0,github.create_or_update_file github.projects_write github.update_gist github.create_gist github.label_write github.update_issue_comment slack.update_file github.issue_write github.update_pull_request github.create_pull_request +q084,List collaborators on a repo,resource,github.list_repository_collaborators,1.0,1.0,1.0,1.0000,1.0000,1,1,278,1.0,1,472,1.0,1,860,1.0,github.list_repository_collaborators github.list_commits trello.trello_list_cards github.get_discussion_comments github.request_copilot_review github.list_labels github.list_discussions github.list_branches github.update_issue_comment github.list_tags +q085,Reply in a Slack thread,verb_product,slack.slack_reply_to_thread,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,349,1.0,1,668,1.0,slack.slack_reply_to_thread slack.slack_get_thread_replies slack.slack_join_channel slack.get_reminder slack.delete_reminder slack.slack_create_channel slack.get_canvase slack.slack_leave_channel slack.update_reminder slack.slack_list_channels +q086,Add an emoji reaction on Slack,verb_product,slack.slack_add_reaction,1.0,1.0,1.0,1.0000,1.0000,1,1,302,1.0,1,465,1.0,1,816,1.0,slack.slack_add_reaction discord.discord_add_reaction slack.create_reminder slack.get_reminder slack.create_canvase slack.create_bookmark slack.delete_reminder slack.slack_join_channel slack.slack_create_channel slack.update_reminder +q087,Search Slack messages,verb_product,slack.slack_search_messages,1.0,1.0,1.0,1.0000,1.0000,1,1,231,1.0,1,373,1.0,1,661,1.0,slack.slack_search_messages slack.slack_get_channel_history slack.slack_post_message slack.slack_add_reaction twilio.twilio_list_messages slack.slack_join_channel slack.slack_leave_channel gmail.gmail_search_emails slack.get_canvase slack.get_reminder +q088,Create a Slack channel,verb_product,slack.slack_join_channel,0.0,1.0,1.0,0.5000,0.6309,2,1,196,1.0,1,339,1.0,1,751,1.0,slack.slack_join_channel slack.slack_create_channel slack.slack_leave_channel slack.slack_set_channel_topic slack.slack_post_message slack.slack_get_channel_history slack.slack_get_thread_replies slack.slack_reply_to_thread slack.slack_add_reaction slack.slack_upload_file +q089,Transition a Jira issue,verb_product,jira.jira_transition_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,389,1.0,1,748,1.0,jira.jira_transition_issue jira.jira_assign_issue jira.jira_get_issue jira.jira_create_issue jira.jira_update_issue jira.jira_add_comment jira.jira_add_worklog jira.jira_search_issues jira.create_board jira.jira_list_projects +q090,Search Jira with JQL,identifier,jira.jira_search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,178,1.0,1,283,1.0,1,617,1.0,jira.jira_search_issues jira.jira_list_projects jira.get_board jira.jira_get_issue jira.get_webhook jira.jira_add_worklog jira.get_sprint github.search_issues confluence.confluence_search jira.delete_board +q091,Add a worklog to Jira,verb_product,jira.jira_add_worklog,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,352,1.0,1,610,1.0,jira.jira_add_worklog jira.jira_add_comment jira.jira_transition_issue jira.get_board jira.jira_get_issue jira.get_sprint jira.get_webhook jira.jira_assign_issue jira.delete_board jira.jira_list_projects +q092,List Linear teams,verb_product,linear.linear_list_teams,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,339,1.0,1,690,1.0,linear.linear_list_teams linear.linear_list_projects linear.linear_list_issues linear.linear_create_project linear.list_labels linear.list_cycles linear.list_users linear.linear_search_issues linear.list_comments linear.list_attachments +q093,Comment on a Linear issue,verb_product,linear.linear_create_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,213,1.0,1,383,1.0,1,831,1.0,linear.linear_create_comment linear.get_comment linear.linear_assign_issue linear.linear_get_issue linear.linear_update_issue linear.delete_comment linear.linear_create_issue linear.create_comment linear.update_comment github.update_issue_comment +q094,Move a local file,paraphrase,filesystem.read_text_file,0.0,1.0,1.0,0.3333,0.5000,3,1,224,1.0,1,335,1.0,1,686,1.0,filesystem.read_text_file filesystem.read_file filesystem.move_file filesystem.read_media_file slack.delete_file figma.figma_get_file slack.update_file google_drive.gdrive_share_file slack.get_file google_drive.gdrive_update_file +q095,Recursively search files by pattern,paraphrase,filesystem.search_files,1.0,1.0,1.0,1.0000,1.0000,1,1,192,1.0,1,322,1.0,1,696,1.0,filesystem.search_files filesystem.directory_tree redis.redis_keys slack.list_files google_drive.gdrive_search linear.linear_search_issues filesystem.list_directory github.search_repositories filesystem.list_directory_with_sizes github.push_files +q096,Edit a file in place with old and new string,paraphrase,filesystem.edit_file,1.0,1.0,1.0,1.0000,1.0000,1,1,230,1.0,1,353,1.0,1,776,1.0,filesystem.edit_file filesystem.list_directory_with_sizes slack.get_file slack.delete_file filesystem.write_file slack.update_file slack.create_file figma.figma_get_file github.get_file_blame google_drive.gdrive_create_file +q097,Read the whole knowledge graph,paraphrase,memory.read_graph,1.0,1.0,1.0,1.0000,1.0000,1,1,152,1.0,1,253,1.0,1,561,1.0,memory.read_graph memory.delete_relations memory.create_entities memory.create_relations memory.open_nodes memory.add_observations memory.delete_entities memory.search_nodes filesystem.read_file filesystem.read_text_file +q098,Search memory nodes,verb_product,memory.search_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,176,1.0,1,331,1.0,1,625,1.0,memory.search_nodes memory.open_nodes github.search_users github.search_issues notion.notion_search confluence.confluence_search slack.slack_search_messages memory.create_entities zendesk.zendesk_search elasticsearch.es_search +q099,Show unstaged git diff,verb_product,git.git_diff_unstaged,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,303,1.0,1,613,1.0,git.git_diff_unstaged git.git_diff_staged git.git_diff git.git_show git.git_status git.git_reset git.git_log git.git_commit git.git_add github.unstar_repository +q100,Stage files with git add,identifier,git.git_add,1.0,1.0,1.0,1.0000,1.0000,1,1,193,1.0,1,332,1.0,1,694,1.0,git.git_add filesystem.list_directory_with_sizes git.git_reset git.git_commit slack.list_files github.push_files git.git_diff_staged git.git_checkout git.git_diff_unstaged github.update_gist +q101,Create a Shopify product,verb_product,shopify.shopify_create_product,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,380,1.0,1,702,1.0,shopify.shopify_create_product shopify.shopify_get_product shopify.shopify_update_product shopify.shopify_create_order shopify.shopify_list_products stripe.stripe_create_product shopify.shopify_get_order stripe.stripe_create_price shopify.shopify_list_customers shopify.shopify_list_orders +q102,Cancel a Stripe subscription,verb_product,stripe.stripe_cancel_subscription,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,321,1.0,1,663,1.0,stripe.stripe_cancel_subscription stripe.stripe_get_customer stripe.stripe_create_customer stripe.stripe_create_product stripe.stripe_list_subscriptions stripe.stripe_create_invoice stripe.stripe_get_payment_intent stripe.stripe_create_refund stripe.stripe_create_price stripe.stripe_create_payment_intent +q103,Create a payment intent,resource,stripe.stripe_create_payment_intent,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,424,1.0,1,906,1.0,stripe.stripe_create_payment_intent stripe.stripe_create_refund stripe.stripe_get_payment_intent github.create_environment github.create_package jira.create_attachment jira.create_sprint linear.create_comment notion.create_comment linear.create_attachment +q104,Acknowledge a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,261,1.0,1,405,1.0,1,665,1.0,pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls memory.read_graph github.delete_package memory.create_entities github.update_package +q105,Create a Datadog monitor,verb_product,datadog.datadog_get_monitor,0.0,1.0,1.0,0.5000,0.6309,2,1,259,1.0,1,392,1.0,1,851,1.0,datadog.datadog_get_monitor datadog.datadog_create_monitor datadog.datadog_list_monitors github.create_gist filesystem.create_directory notion.notion_create_database github.create_branch github.create_repository github.create_hook gitlab.create_job +q106,Resolve a Sentry issue,paraphrase,sentry.sentry_get_issue,0.0,1.0,1.0,0.5000,0.6309,2,1,251,1.0,1,412,1.0,1,817,1.0,sentry.sentry_get_issue sentry.sentry_update_issue sentry.sentry_list_events pagerduty.pagerduty_resolve_incident sentry.sentry_list_issues jira.jira_assign_issue sentry.sentry_list_projects github.issue_write github.issue_read jira.jira_get_issue +q107,Scale a Kubernetes deployment,verb_product,kubernetes.k8s_scale_deployment,1.0,1.0,1.0,1.0000,1.0000,1,1,252,1.0,1,407,1.0,1,726,1.0,kubernetes.k8s_scale_deployment kubernetes.k8s_get_pod kubernetes.k8s_list_deployments kubernetes.k8s_delete kubernetes.k8s_apply vercel.vercel_get_deployment kubernetes.k8s_get_events kubernetes.k8s_list_services kubernetes.k8s_list_pods kubernetes.k8s_pod_logs +q108,Apply a Kubernetes manifest,verb_product,kubernetes.k8s_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,253,1.0,1,366,1.0,1,727,1.0,kubernetes.k8s_apply kubernetes.k8s_get_pod kubernetes.k8s_delete kubernetes.k8s_list_pods kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_scale_deployment kubernetes.k8s_get_events kubernetes.k8s_pod_logs terraform.tf_apply +q109,Stop a docker container,verb_product,docker.docker_stop,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,345,1.0,1,655,1.0,docker.docker_stop docker.docker_rm docker.docker_logs docker.docker_run docker.docker_exec docker.docker_ps docker.docker_build docker.docker_images kubernetes.k8s_pod_logs stripe.stripe_create_customer +q110,Build a docker image,verb_product,docker.docker_build,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,317,1.0,1,598,1.0,docker.docker_build docker.docker_run docker.docker_images docker.docker_stop docker.docker_rm docker.docker_logs docker.docker_ps docker.docker_exec filesystem.read_media_file jira.delete_filter +q111,Click an element in Playwright,verb_product,playwright.browser_click,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,329,1.0,1,564,1.0,playwright.browser_click puppeteer.puppeteer_click playwright.browser_type playwright.browser_take_screenshot playwright.browser_snapshot playwright.browser_navigate playwright.browser_tab_close playwright.browser_tab_list playwright.browser_fill_form playwright.browser_tab_new +q112,Type text into a browser input,paraphrase,playwright.browser_type,1.0,1.0,1.0,1.0000,0.7602,1,1,217,1.0,1,316,1.0,1,589,1.0,playwright.browser_type playwright.browser_navigate playwright.browser_wait_for playwright.browser_tab_close playwright.browser_tab_new playwright.browser_tab_list supabase.supabase_insert playwright.browser_press_key playwright.browser_fill_form slack.slack_post_message +q113,List GitLab projects,verb_product,gitlab.gitlab_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,289,1.0,1,564,1.0,gitlab.gitlab_list_projects github.projects_list linear.linear_list_projects gitlab.gitlab_list_issues jira.jira_list_projects gitlab.gitlab_list_merge_requests vercel.vercel_list_projects sentry.sentry_list_projects git.git_branch github.projects_get +q114,Create a Bitbucket pull request,verb_product,bitbucket.bitbucket_create_pr,1.0,1.0,1.0,1.0000,1.0000,1,1,313,1.0,1,553,1.0,1,1037,1.0,bitbucket.bitbucket_create_pr bitbucket.bitbucket_list_prs bitbucket.bitbucket_create_issue github.pull_request_read github.create_pull_request github.merge_pull_request github.pull_request_review_write bitbucket.bitbucket_list_repos github.update_pull_request_branch github.list_pull_requests +q115,Create a Gmail draft,verb_product,gmail.gmail_create_draft,1.0,1.0,1.0,1.0000,1.0000,1,1,235,1.0,1,439,1.0,1,903,1.0,gmail.gmail_create_draft gmail.gmail_trash_email github.create_gist github.create_pull_request gmail.gmail_get_email stripe.stripe_create_customer gitlab.create_job github.create_hook gitlab.create_wiki github.create_branch +q116,Share a Google Drive file,verb_product,google_drive.gdrive_share_file,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,418,1.0,1,762,1.0,google_drive.gdrive_share_file google_drive.gdrive_update_file google_drive.gdrive_get_file google_drive.gdrive_list_folder google_drive.gdrive_create_file google_drive.gdrive_search google_calendar.gcal_delete_event google_calendar.gcal_update_event gitlab.gitlab_get_file google_calendar.gcal_list_calendars +q117,List Google Calendar events,verb_product,google_calendar.gcal_list_events,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,427,1.0,1,754,1.0,google_calendar.gcal_list_events google_calendar.gcal_delete_event google_calendar.gcal_list_calendars google_calendar.gcal_update_event google_calendar.gcal_create_event kubernetes.k8s_get_events sentry.sentry_list_events google_drive.gdrive_list_folder google_drive.gdrive_get_file google_drive.gdrive_update_file +q118,Delete a calendar event,resource,google_calendar.gcal_delete_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,440,1.0,1,685,1.0,google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event google_calendar.gcal_list_events github.delete_environment linear.delete_comment github.delete_hook linear.delete_cycle gitlab.delete_job google_calendar.gcal_list_calendars +q119,List Vercel projects,verb_product,vercel.vercel_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,163,1.0,1,255,1.0,1,550,1.0,vercel.vercel_list_projects vercel.vercel_list_deployments linear.linear_list_projects vercel.vercel_create_project jira.jira_list_projects github.projects_list gitlab.gitlab_list_projects sentry.sentry_list_projects vercel.vercel_get_deployment github.projects_get +q120,Select rows from Supabase,verb_product,supabase.supabase_select,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,354,1.0,1,688,1.0,supabase.supabase_select supabase.supabase_insert supabase.supabase_update supabase.supabase_invoke_fn supabase.supabase_list_tables aws_s3.s3_get_object mongodb.mongo_delete memory.delete_observations docker.docker_logs slack.slack_get_channel_history +q121,Post a Figma comment,verb_product,figma.figma_post_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,201,1.0,1,320,1.0,1,639,1.0,figma.figma_post_comment figma.figma_get_comments linear.get_comment figma.figma_get_file linear.update_comment notion.get_comment linear.delete_comment notion.update_comment linear.create_comment notion.delete_comment +q122,List Okta users,verb_product,okta.okta_list_users,1.0,1.0,1.0,1.0000,1.0000,1,1,198,1.0,1,288,1.0,1,618,1.0,okta.okta_list_users okta.okta_get_user linear.list_users okta.okta_create_user notion.notion_list_users slack.slack_get_users okta.okta_deactivate_user slack.slack_list_users notion.list_users_tokens pagerduty.pagerduty_list_oncalls +q123,Range query Prometheus,verb_product,prometheus.prom_query_range,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,377,1.0,1,674,1.0,prometheus.prom_query_range prometheus.prom_query prometheus.prom_list_metrics postgres.explain_query datadog.datadog_query_metrics mysql.mysql_query sqlite.sqlite_query github.search_users memory.search_nodes notion.notion_query_database +q124,Create a Grafana annotation,verb_product,grafana.grafana_create_annotation,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,435,1.0,1,923,1.0,grafana.grafana_create_annotation github.create_gist grafana.grafana_get_dashboard github.create_artifact gitlab.create_job github.create_hook jira.create_attachment gitlab.create_wiki notion.create_comment gitlab.create_label +q125,Apply terraform,identifier,terraform.tf_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,171,1.0,1,327,1.0,1,644,1.0,terraform.tf_apply terraform.tf_plan terraform.tf_state_list kubernetes.k8s_apply slack.slack_reply_to_thread playwright.browser_fill_form filesystem.search_files linear.linear_list_issues docker.docker_rm linear.linear_search_issues +q126,List Asana tasks,verb_product,asana.asana_list_tasks,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,386,1.0,1,724,1.0,asana.asana_list_tasks asana.asana_get_task asana.asana_create_task asana.asana_update_task clickup.clickup_list_tasks aws_s3.s3_list_buckets asana.asana_create_project trello.trello_list_cards github.list_hooks linear.list_attachments +q127,Comment on a Trello card,verb_product,trello.trello_add_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,390,1.0,1,703,1.0,trello.trello_add_comment trello.trello_create_card trello.trello_list_cards trello.trello_create_board figma.figma_post_comment github.update_issue_comment linear.get_comment notion.get_comment notion.delete_comment linear.delete_comment +q128,Update a ClickUp task,verb_product,clickup.clickup_update_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,399,1.0,1,736,1.0,clickup.clickup_update_task clickup.clickup_get_task clickup.clickup_create_task clickup.clickup_list_tasks asana.asana_update_task github.update_hook linear.update_cycle gitlab.update_job linear.update_comment slack.update_canvase +q129,Search Confluence,verb_product,confluence.confluence_search,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,362,1.0,1,708,1.0,confluence.confluence_search confluence.confluence_get_page confluence.confluence_create_page confluence.confluence_update_page github.search_users github.search_commits brave_search.brave_web_search github.search_orgs elasticsearch.es_search github.search_issues +q130,Create a HubSpot deal,verb_product,hubspot.hubspot_create_deal,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,413,1.0,1,860,1.0,hubspot.hubspot_create_deal hubspot.hubspot_create_contact hubspot.hubspot_get_contact github.create_gist github.create_repository github.create_hook hubspot.hubspot_list_contacts github.create_branch github.create_runner github.create_milestone +q131,List Zendesk tickets,verb_product,zendesk.zendesk_list_tickets,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,367,1.0,1,669,1.0,zendesk.zendesk_list_tickets zendesk.zendesk_search zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_create_ticket aws_s3.s3_list_buckets clickup.clickup_list_tasks gitlab.list_snippets kubernetes.k8s_list_deployments aws_s3.s3_list_objects +q132,List Discord channels,verb_product,discord.discord_list_channels,1.0,1.0,1.0,1.0000,1.0000,1,1,166,1.0,1,321,1.0,1,666,1.0,discord.discord_list_channels discord.discord_get_channel discord.discord_list_members discord.discord_send_message discord.discord_add_reaction slack.slack_list_channels linear.list_labels trello.trello_list_cards gitlab.list_labels clickup.clickup_list_tasks +q133,Set a Redis key,verb_product,redis.redis_set,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,281,1.0,1,586,1.0,redis.redis_set redis.redis_get redis.redis_delete redis.redis_keys jira.jira_get_issue figma.figma_get_file aws_s3.s3_get_object slack.slack_set_channel_topic playwright.browser_press_key figma.figma_get_comments +q134,Insert MongoDB documents,verb_product,mongodb.mongo_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,381,1.0,1,751,1.0,mongodb.mongo_insert mongodb.mongo_update mongodb.mongo_find mongodb.mongo_delete mongodb.mongo_list_collections supabase.supabase_insert elasticsearch.es_index_doc elasticsearch.es_get_doc linear.list_comments notion.list_comments +q135,Run a MySQL query,verb_product,postgres.query,0.0,1.0,1.0,0.5000,0.6309,2,1,182,1.0,1,294,1.0,1,585,1.0,postgres.query mysql.mysql_query postgres.explain_query sqlite.sqlite_query salesforce.salesforce_query mysql.mysql_list_tables prometheus.prom_query notion.notion_query_database memory.search_nodes github.search_users +q136,Index a document in Elasticsearch,verb_product,elasticsearch.es_index_doc,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,342,1.0,1,734,1.0,elasticsearch.es_index_doc elasticsearch.es_search elasticsearch.es_get_doc playwright.browser_tab_close github.get_environment github.delete_environment github.update_environment github.create_environment mongodb.mongo_find github.create_milestone +q137,Get an S3 object,verb_product,aws_s3.s3_get_object,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,375,1.0,1,691,1.0,aws_s3.s3_get_object aws_s3.s3_delete_object aws_s3.s3_put_object aws_s3.s3_list_objects okta.okta_get_user asana.asana_get_task gmail.gmail_send_email github.projects_get github.get_artifact github.get_hook +q138,List Cloudflare zones,verb_product,cloudflare.cf_list_zones,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,338,1.0,1,729,1.0,cloudflare.cf_list_zones cloudflare.cf_purge_cache cloudflare.cf_list_dns cloudflare.cf_create_dns clickup.clickup_list_tasks github.list_milestones gitlab.list_milestones gitlab.list_pipelines github.list_packages linear.list_cycles +q139,Create a Stripe invoice,verb_product,stripe.stripe_create_invoice,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,353,1.0,1,732,1.0,stripe.stripe_create_invoice stripe.stripe_create_customer stripe.stripe_create_product stripe.stripe_create_price stripe.stripe_create_refund stripe.stripe_list_invoices stripe.stripe_create_payment_intent stripe.stripe_get_customer gitlab.create_pipeline github.create_package +q140,List GitHub Actions workflows,verb_product,github.actions_list,1.0,1.0,1.0,1.0000,1.0000,1,1,293,1.0,1,459,1.0,1,783,1.0,github.actions_list github.actions_get github.actions_run_trigger github.actions_get_job_logs github.list_discussions github.projects_list github.list_tags github.list_gists github.list_releases github.list_notifications +q141,Get GitHub Actions job logs,verb_product,github.actions_get_job_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,440,1.0,1,802,1.0,github.actions_get_job_logs github.actions_get github.actions_list github.actions_run_trigger github.get_gist gitlab.get_job github.projects_get github.get_label github.get_discussion github.get_latest_release +q142,Create a GitHub repository,verb_product,github.create_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,235,1.0,1,388,1.0,1,874,1.0,github.create_repository github.delete_repository github.star_repository github.fork_repository github.unstar_repository github.create_branch github.list_repository_collaborators github.create_issue github.get_repository_tree github.list_tags +q143,Checkout a git branch,verb_product,git.git_checkout,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,376,1.0,1,864,1.0,git.git_checkout git.git_branch git.git_create_branch github.create_branch git.git_diff github.update_pull_request_branch github.delete_file github.list_commits git.git_show github.push_files +q144,Show git log,identifier,git.git_log,1.0,1.0,1.0,1.0000,1.0000,1,1,206,1.0,1,309,1.0,1,603,1.0,git.git_log git.git_show git.git_diff git.git_status git.git_diff_staged git.git_diff_unstaged git.git_branch git.git_create_branch git.git_checkout git.git_reset +q145,List postgres schemas,verb_product,postgres.list_schemas,1.0,1.0,1.0,1.0000,1.0000,1,1,158,1.0,1,277,1.0,1,596,1.0,postgres.list_schemas postgres.list_tables postgres.query postgres.describe_table postgres.explain_query kubernetes.k8s_list_services aws_s3.s3_list_buckets github.list_packages gitlab.list_pipelines linear.list_cycles +q146,Open nodes in the knowledge graph,verb_product,memory.open_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,163,1.0,1,262,1.0,1,492,1.0,memory.open_nodes memory.search_nodes memory.read_graph memory.create_entities memory.create_relations memory.delete_relations memory.add_observations memory.delete_entities sqlite.sqlite_list_tables postgres.list_schemas +q147,Get file metadata from the filesystem,paraphrase,filesystem.read_file,0.0,1.0,1.0,0.3333,0.5000,3,1,220,1.0,1,334,1.0,1,694,1.0,filesystem.read_file filesystem.read_text_file filesystem.get_file_info filesystem.read_media_file google_drive.gdrive_get_file gitlab.gitlab_get_file filesystem.list_allowed_directories filesystem.move_file filesystem.edit_file figma.figma_get_file_nodes +q148,Write a new local file,paraphrase,filesystem.write_file,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,384,1.0,1,705,1.0,filesystem.write_file filesystem.read_file filesystem.read_text_file filesystem.edit_file filesystem.read_media_file playwright.browser_tab_new slack.delete_file slack.update_file figma.figma_get_file slack.create_file +q149,Search GitHub issues,verb_product,github.search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,382,1.0,1,683,1.0,github.search_issues github.list_issues github.search_users jira.jira_search_issues github.search_orgs linear.linear_search_issues github.search_pull_requests github.search_commits github.search_code gitlab.gitlab_list_issues +q150,Get the authenticated GitHub user,paraphrase,github.get_teams,0.0,1.0,1.0,0.5000,0.6309,2,1,179,1.0,1,356,1.0,1,694,1.0,github.get_teams github.get_me github.list_notifications github.get_latest_release github.get_repository_tree github.projects_get github.get_gist github.get_label github.get_tag github.list_gists diff --git a/experiments/mcp-tool-search/results/hybrid_rrf_weighted_trigram.json b/experiments/mcp-tool-search/results/hybrid_rrf_weighted_trigram.json new file mode 100644 index 00000000..2db180b5 --- /dev/null +++ b/experiments/mcp-tool-search/results/hybrid_rrf_weighted_trigram.json @@ -0,0 +1,7013 @@ +{ + "summary": { + "n": 150, + "index_s": 0.008005067000112831, + "rss_mb": 0.01953125, + "p50_ms": 0.8428800000501724, + "p95_ms": 1.2099685999658047, + "recall@1": 0.86, + "recall@1_lo": 0.8066666666666666, + "recall@1_hi": 0.9133333333333333, + "recall@3": 0.9866666666666667, + "recall@3_lo": 0.9666666666666667, + "recall@3_hi": 1.0, + "recall@5": 0.9933333333333333, + "recall@5_lo": 0.98, + "recall@5_hi": 1.0, + "mrr@10": 0.922074074074074, + "mrr@10_lo": 0.8888148148148148, + "mrr@10_hi": 0.9531851851851852, + "ndcg@10": 0.9362730460311218, + "ndcg@10_lo": 0.9105343483940073, + "ndcg@10_hi": 0.9595090861652102 + }, + "meta": { + "name": "hybrid_rrf_weighted_trigram", + "needs_model": false, + "needs_api": false, + "notes": "RRF of weighted BM25 + char trigram" + }, + "rows": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "github.issue_write", + "github.issue_read", + "github.update_issue_comment", + "gitlab.gitlab_create_issue", + "github.add_issue_comment", + "github.create_gist", + "github.create_branch", + "github.create_repository", + "bitbucket.bitbucket_create_issue" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 388, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 584, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1038, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "kind": "verb_product", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_create_issue", + "jira.jira_create_sprint", + "zendesk.zendesk_create_ticket", + "jira.create_board", + "jira.create_sprint", + "jira.create_filter", + "jira.create_version", + "jira.create_webhook", + "jira.create_attachment", + "zendesk.zendesk_update_ticket" + ], + "hit1": "jira.jira_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 528, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1003, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "kind": "verb_product", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "ranking": [ + "slack.slack_post_message", + "slack.slack_add_reaction", + "slack.slack_search_messages", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_get_channel_history", + "discord.discord_send_message", + "slack.slack_create_channel", + "slack.slack_set_channel_topic", + "slack.slack_get_user_profile" + ], + "hit1": "slack.slack_post_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 693, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "kind": "resource", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "ranking": [ + "github.list_pull_requests", + "github.search_pull_requests", + "gitlab.gitlab_list_merge_requests", + "github.create_pull_request", + "github.pull_request_read", + "bitbucket.bitbucket_list_prs", + "github.merge_pull_request", + "github.update_pull_request", + "github.update_pull_request_branch", + "github.request_copilot_review" + ], + "hit1": "github.list_pull_requests", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 474, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 993, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "ranking": [ + "github.create_pull_request", + "github.pull_request_read", + "github.merge_pull_request", + "github.request_copilot_review", + "github.search_pull_requests", + "github.update_pull_request", + "github.update_pull_request_branch", + "github.list_pull_requests", + "github.pull_request_review_write", + "github.add_comment_to_pending_review" + ], + "hit1": "github.create_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 368, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 522, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1144, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "kind": "verb_product", + "gold": [ + "postgres.query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "postgres.explain_query", + "mysql.mysql_query", + "postgres.list_schemas", + "sqlite.sqlite_query", + "postgres.list_tables", + "salesforce.salesforce_query", + "postgres.describe_table", + "prometheus.prom_query", + "docker.docker_run" + ], + "hit1": "postgres.query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 272, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 590, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "kind": "paraphrase", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "ranking": [ + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "gitlab.gitlab_get_file", + "github.delete_file", + "figma.figma_get_file_nodes", + "figma.figma_get_file", + "filesystem.get_file_info", + "filesystem.move_file", + "github.get_file_contents" + ], + "hit1": "filesystem.read_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 781, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "kind": "paraphrase", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "ranking": [ + "google_drive.gdrive_list_folder", + "slack.list_files", + "github.push_files", + "filesystem.list_allowed_directories", + "google_drive.gdrive_search", + "cloudflare.cf_purge_cache", + "filesystem.search_files", + "git.git_add", + "filesystem.list_directory", + "filesystem.move_file" + ], + "hit1": "google_drive.gdrive_list_folder", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.1111111111111111, + "ndcg@10": 0.1922943146971571, + "hit_rank": 9, + "k3_calls": 3, + "k3_tokens": 664, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 703, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 681, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "ranking": [ + "memory.read_graph", + "memory.create_entities", + "memory.delete_relations", + "memory.open_nodes", + "memory.create_relations", + "memory.add_observations", + "memory.search_nodes", + "memory.delete_entities", + "time.get_current_time", + "sqlite.sqlite_list_tables" + ], + "hit1": "memory.read_graph", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.5105967295264829, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 154, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 255, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 506, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "kind": "verb_product", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "ranking": [ + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "jira.jira_search_issues", + "github.update_pull_request_branch", + "memory.search_nodes", + "github.search_users", + "github.search_orgs", + "github.search_commits", + "github.search_issues", + "confluence.confluence_search" + ], + "hit1": "brave_search.brave_web_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 410, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "kind": "verb_product", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "ranking": [ + "fetch.fetch", + "filesystem.read_file", + "git.git_add", + "playwright.browser_navigate", + "github.get_file_contents", + "github.get_me", + "git.git_status", + "slack.slack_set_channel_topic", + "github.get_repository_tree", + "puppeteer.puppeteer_navigate" + ], + "hit1": "fetch.fetch", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 222, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 722, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "kind": "identifier", + "gold": [ + "git.git_status" + ], + "alts": [], + "ranking": [ + "git.git_status", + "git.git_diff_staged", + "git.git_show", + "git.git_diff_unstaged", + "git.git_diff", + "git.git_log", + "git.git_add", + "sentry.sentry_update_issue", + "shopify.shopify_list_orders", + "gitlab.list_snippets" + ], + "hit1": "git.git_status", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "kind": "paraphrase", + "gold": [ + "git.git_commit" + ], + "alts": [], + "ranking": [ + "git.git_commit", + "git.git_reset", + "git.git_diff_staged", + "git.git_show", + "git.git_log", + "github.get_commit", + "github.push_files", + "linear.list_comments", + "notion.list_comments", + "github.list_packages" + ], + "hit1": "git.git_commit", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 305, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_issue", + "linear.linear_update_issue", + "linear.linear_assign_issue", + "linear.linear_create_comment", + "linear.linear_get_issue", + "linear.linear_create_project", + "linear.create_user", + "linear.create_label", + "linear.create_cycle", + "linear.create_comment" + ], + "hit1": "linear.linear_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 340, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 925, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "kind": "paraphrase", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_project", + "linear.linear_list_projects", + "linear.linear_create_issue", + "linear.delete_cycle", + "linear.get_cycle", + "linear.delete_user", + "linear.get_user", + "linear.linear_list_teams", + "linear.delete_label", + "linear.update_user" + ], + "hit1": "linear.linear_create_project", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 284, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 638, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "ranking": [ + "github.update_issue_comment", + "github.add_issue_comment", + "github.issue_read", + "github.assign_copilot_to_issue", + "figma.figma_post_comment", + "github.create_issue", + "jira.jira_add_comment", + "github.get_discussion_comments", + "linear.linear_create_comment", + "github.issue_write" + ], + "hit1": "github.update_issue_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 510, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1006, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "kind": "ambiguous", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "ranking": [ + "github.merge_pull_request", + "gitlab.gitlab_merge_merge_request", + "github.update_pull_request_branch", + "gitlab.gitlab_get_merge_request", + "github.pull_request_read", + "github.search_pull_requests", + "github.update_pull_request", + "gitlab.gitlab_list_merge_requests", + "github.create_pull_request", + "github.request_copilot_review" + ], + "hit1": "github.merge_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 299, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 468, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 987, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "kind": "verb_product", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "ranking": [ + "github.actions_run_trigger", + "github.actions_get", + "github.actions_list", + "github.actions_get_job_logs", + "github.fork_repository", + "github.star_repository", + "github.unstar_repository", + "github.get_discussion", + "github.delete_repository", + "github.update_gist" + ], + "hit1": "github.actions_run_trigger", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 842, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "kind": "resource", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "ranking": [ + "github.list_dependabot_alerts", + "github.get_dependabot_alert", + "github.list_secret_scanning_alerts", + "github.list_code_scanning_alerts", + "github.get_hook", + "gitlab.get_job", + "gitlab.get_wiki", + "redis.redis_get", + "github.get_package", + "github.get_runner" + ], + "hit1": "github.list_dependabot_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "kind": "verb_product", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_issues", + "sentry.sentry_list_projects", + "sentry.sentry_list_events", + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "gitlab.gitlab_list_issues", + "linear.linear_search_issues", + "github.list_issues", + "github.search_issues", + "linear.linear_list_issues" + ], + "hit1": "sentry.sentry_list_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_list_invoices", + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_price", + "stripe.stripe_create_product", + "stripe.stripe_list_subscriptions", + "stripe.stripe_list_customers", + "stripe.stripe_create_refund" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "kind": "paraphrase", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_refund", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent", + "stripe.stripe_get_customer", + "stripe.stripe_create_customer", + "stripe.stripe_create_price", + "stripe.stripe_create_product", + "stripe.stripe_create_invoice", + "stripe.stripe_list_customers", + "stripe.stripe_list_invoices" + ], + "hit1": "stripe.stripe_create_refund", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "kind": "verb_product", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "ranking": [ + "gmail.gmail_send_email", + "gmail.gmail_get_email", + "gmail.gmail_search_emails", + "gmail.gmail_trash_email", + "puppeteer.puppeteer_click", + "twilio.twilio_send_sms", + "gmail.gmail_list_labels", + "gmail.gmail_modify_email", + "puppeteer.puppeteer_fill", + "gmail.gmail_create_draft" + ], + "hit1": "gmail.gmail_send_email", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 713, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "kind": "verb_product", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "ranking": [ + "gmail.gmail_search_emails", + "gmail.gmail_list_labels", + "elasticsearch.es_search", + "github.search_users", + "confluence.confluence_search", + "gmail.gmail_send_email", + "google_drive.gdrive_search", + "slack.slack_search_messages", + "gmail.gmail_get_email", + "github.search_orgs" + ], + "hit1": "gmail.gmail_search_emails", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 617, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "kind": "paraphrase", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_update_file", + "google_drive.gdrive_share_file", + "google_drive.gdrive_search", + "google_drive.gdrive_get_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_list_folder", + "slack.slack_upload_file", + "figma.figma_get_file", + "google_calendar.gcal_list_calendars", + "slack.delete_file" + ], + "hit1": "google_drive.gdrive_update_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.2, + "ndcg@10": 0.38685280723454163, + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 496, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "linear.create_comment", + "jira.create_attachment", + "github.create_environment", + "jira.create_sprint", + "notion.create_comment" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 909, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_list_pods", + "kubernetes.k8s_list_services", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_get_events", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_apply", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_delete", + "kubernetes.k8s_scale_deployment", + "linear.list_cycles" + ], + "hit1": "kubernetes.k8s_list_pods", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_get_pod", + "docker.docker_logs", + "github.actions_get_job_logs", + "github.get_tag", + "gitlab.gitlab_get_file", + "datadog.datadog_list_logs", + "elasticsearch.es_get_doc", + "figma.figma_get_file_nodes", + "aws_s3.s3_get_object" + ], + "hit1": "kubernetes.k8s_pod_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 441, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 848, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "ranking": [ + "docker.docker_run", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_exec", + "docker.docker_logs", + "docker.docker_ps", + "docker.docker_build", + "docker.docker_images", + "kubernetes.k8s_pod_logs", + "mysql.mysql_query" + ], + "hit1": "docker.docker_run", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 640, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "kind": "verb_product", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "ranking": [ + "docker.docker_logs", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_run", + "docker.docker_exec", + "docker.docker_ps", + "docker.docker_images", + "kubernetes.k8s_pod_logs", + "docker.docker_build", + "git.git_log" + ], + "hit1": "docker.docker_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 649, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "kind": "ambiguous", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "ranking": [ + "playwright.browser_navigate", + "playwright.browser_tab_new", + "puppeteer.puppeteer_navigate", + "playwright.browser_tab_close", + "playwright.browser_evaluate", + "playwright.browser_press_key", + "playwright.browser_tab_list", + "playwright.browser_click", + "playwright.browser_wait_for", + "playwright.browser_type" + ], + "hit1": "playwright.browser_navigate", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 574, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "kind": "ambiguous", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "ranking": [ + "playwright.browser_take_screenshot", + "puppeteer.puppeteer_screenshot", + "playwright.browser_snapshot", + "confluence.confluence_create_page", + "notion.notion_update_page", + "confluence.confluence_get_page", + "playwright.browser_press_key", + "notion.notion_get_page", + "playwright.browser_evaluate", + "confluence.confluence_update_page" + ], + "hit1": "playwright.browser_take_screenshot", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 372, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 675, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_create_merge_request", + "gitlab.gitlab_get_merge_request", + "github.merge_pull_request", + "gitlab.gitlab_list_merge_requests", + "gitlab.gitlab_create_pipeline", + "gitlab.gitlab_create_issue", + "github.pull_request_review_write", + "github.create_pull_request", + "bitbucket.bitbucket_create_pr" + ], + "hit1": "gitlab.gitlab_merge_merge_request", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1054, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "kind": "verb_product", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "ranking": [ + "notion.notion_search", + "notion.notion_get_page", + "notion.notion_update_page", + "notion.notion_get_comments", + "notion.notion_create_page", + "notion.notion_get_block", + "notion.notion_create_database", + "notion.get_comment", + "notion.notion_list_users", + "github.search_orgs" + ], + "hit1": "notion.notion_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 635, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "kind": "verb_product", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "ranking": [ + "notion.notion_get_page", + "notion.notion_create_page", + "notion.notion_update_page", + "notion.notion_create_database", + "notion.notion_get_comments", + "notion.create_comment", + "notion.create_users_token", + "notion.create_data_source", + "confluence.confluence_create_page", + "notion.update_comment" + ], + "hit1": "notion.notion_get_page", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 832, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "kind": "verb_product", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "ranking": [ + "notion.notion_query_database", + "notion.notion_create_database", + "notion.notion_search", + "notion.notion_create_page", + "notion.notion_get_block", + "notion.notion_get_page", + "notion.notion_update_page", + "notion.delete_data_source", + "notion.get_data_source", + "notion.delete_comment" + ], + "hit1": "notion.notion_query_database", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 688, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_search", + "zendesk.zendesk_list_tickets", + "jira.jira_create_issue", + "gitlab.create_snippet", + "gitlab.create_job", + "github.create_hook", + "gitlab.create_wiki" + ], + "hit1": "zendesk.zendesk_create_ticket", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 882, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_create_deal", + "hubspot.hubspot_search_contacts", + "github.create_gist", + "github.create_artifact", + "github.create_branch", + "github.create_or_update_file", + "github.create_repository" + ], + "hit1": "hubspot.hubspot_create_contact", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 867, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "kind": "verb_product", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_convert_lead", + "salesforce.salesforce_create_lead", + "salesforce.salesforce_create_record", + "salesforce.salesforce_update_record", + "salesforce.salesforce_query", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_get_record", + "salesforce.salesforce_list_opportunities", + "github.create_gist", + "github.create_milestone" + ], + "hit1": "salesforce.salesforce_convert_lead", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 787, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "kind": "identifier", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_query", + "postgres.query", + "mysql.mysql_query", + "sqlite.sqlite_query", + "postgres.explain_query", + "prometheus.prom_query", + "notion.notion_query_database", + "memory.search_nodes", + "github.search_users", + "docker.docker_run" + ], + "hit1": "salesforce.salesforce_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 187, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 631, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "kind": "verb_product", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "ranking": [ + "shopify.shopify_list_orders", + "shopify.shopify_list_customers", + "shopify.shopify_list_products", + "shopify.shopify_get_order", + "shopify.shopify_get_product", + "shopify.shopify_create_order", + "shopify.shopify_update_product", + "shopify.shopify_create_product", + "github.list_runners", + "linear.list_users" + ], + "hit1": "shopify.shopify_list_orders", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "github.create_package", + "github.create_environment", + "pagerduty.pagerduty_list_oncalls", + "gitlab.create_pipeline", + "jira.create_attachment" + ], + "hit1": "pagerduty.pagerduty_create_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 848, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "kind": "paraphrase", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_list_oncalls", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_create_incident", + "notion.notion_get_comments", + "playwright.browser_press_key", + "twilio.twilio_make_call", + "github.list_commits" + ], + "hit1": "pagerduty.pagerduty_list_oncalls", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 363, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "kind": "verb_product", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "ranking": [ + "datadog.datadog_query_metrics", + "datadog.datadog_list_monitors", + "datadog.datadog_create_monitor", + "datadog.datadog_list_logs", + "datadog.datadog_get_monitor", + "prometheus.prom_list_metrics", + "prometheus.prom_query", + "notion.notion_query_database", + "prometheus.prom_query_range", + "postgres.explain_query" + ], + "hit1": "datadog.datadog_query_metrics", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 799, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "kind": "verb_product", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "ranking": [ + "datadog.datadog_list_logs", + "datadog.datadog_list_monitors", + "datadog.datadog_get_monitor", + "datadog.datadog_query_metrics", + "notion.notion_search", + "github.search_orgs", + "datadog.datadog_create_monitor", + "confluence.confluence_search", + "github.search_users", + "elasticsearch.es_search" + ], + "hit1": "datadog.datadog_list_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 749, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_list_buckets", + "aws_s3.s3_list_objects", + "gitlab.list_snippets", + "clickup.clickup_list_tasks", + "kubernetes.k8s_list_deployments", + "jira.list_sprints", + "linear.list_comments", + "github.list_artifacts", + "github.list_environments", + "linear.list_attachments" + ], + "hit1": "aws_s3.s3_list_buckets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "kind": "paraphrase", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_put_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_get_object", + "slack.slack_upload_file", + "gmail.gmail_send_email", + "aws_s3.s3_list_objects", + "twilio.twilio_send_sms", + "github.assign_copilot_to_issue", + "slack.slack_add_reaction", + "asana.asana_create_project" + ], + "hit1": "aws_s3.s3_put_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 426, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 850, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "playwright.browser_tab_close", + "playwright.browser_snapshot", + "postgres.list_tables", + "memory.read_graph", + "postgres.list_schemas", + "puppeteer.puppeteer_click" + ], + "hit1": "cloudflare.cf_purge_cache", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 548, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "cloudflare.cf_list_zones", + "salesforce.salesforce_create_record", + "cloudflare.cf_purge_cache", + "salesforce.salesforce_update_record", + "salesforce.salesforce_delete_record", + "jira.create_board", + "jira.create_sprint", + "jira.create_filter" + ], + "hit1": "cloudflare.cf_create_dns", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 810, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_deployments", + "vercel.vercel_list_projects", + "vercel.vercel_get_deployment", + "kubernetes.k8s_list_deployments", + "vercel.vercel_create_project", + "github.list_environments", + "aws_s3.s3_list_buckets", + "linear.list_comments", + "jira.list_attachments", + "linear.list_attachments" + ], + "hit1": "vercel.vercel_list_deployments", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 636, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "ranking": [ + "supabase.supabase_insert", + "supabase.supabase_select", + "supabase.supabase_update", + "mongodb.mongo_insert", + "supabase.supabase_invoke_fn", + "supabase.supabase_list_tables", + "playwright.browser_type", + "gitlab.get_release", + "slack.get_canvase", + "gitlab.delete_release" + ], + "hit1": "supabase.supabase_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 672, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "kind": "paraphrase", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "ranking": [ + "figma.figma_get_images", + "figma.figma_get_file_nodes", + "figma.figma_get_file", + "docker.docker_images", + "figma.figma_get_comments", + "figma.figma_post_comment", + "supabase.supabase_select", + "slack.slack_get_channel_history", + "docker.docker_logs", + "kubernetes.k8s_pod_logs" + ], + "hit1": "figma.figma_get_images", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 701, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "kind": "verb_product", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "ranking": [ + "twilio.twilio_send_sms", + "twilio.twilio_list_messages", + "puppeteer.puppeteer_click", + "gmail.gmail_send_email", + "twilio.twilio_make_call", + "filesystem.list_directory_with_sizes", + "puppeteer.puppeteer_fill", + "jira.jira_search_issues", + "discord.discord_send_message", + "puppeteer.puppeteer_screenshot" + ], + "hit1": "twilio.twilio_send_sms", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "kind": "verb_product", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "ranking": [ + "okta.okta_get_user", + "okta.okta_create_user", + "okta.okta_deactivate_user", + "okta.okta_list_users", + "linear.create_user", + "linear.update_user", + "linear.delete_user", + "asana.asana_create_project", + "asana.asana_create_task", + "github.create_issue" + ], + "hit1": "okta.okta_get_user", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query", + "prometheus.prom_query_range", + "prometheus.prom_list_metrics", + "postgres.query", + "mysql.mysql_query", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "postgres.explain_query", + "datadog.datadog_query_metrics", + "shopify.shopify_list_products" + ], + "hit1": "prometheus.prom_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 666, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "kind": "verb_product", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "ranking": [ + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "github.search_users", + "github.search_orgs", + "confluence.confluence_search", + "github.search_issues", + "grafana.grafana_create_annotation", + "slack.slack_search_messages", + "elasticsearch.es_search", + "notion.notion_search" + ], + "hit1": "grafana.grafana_search_dashboards", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 288, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "kind": "verb_product", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "ranking": [ + "circleci.circleci_trigger_pipeline", + "circleci.circleci_get_pipeline", + "circleci.circleci_list_pipelines", + "gitlab.gitlab_create_pipeline", + "gitlab.get_pipeline", + "github.actions_run_trigger", + "gitlab.create_pipeline", + "gitlab.delete_pipeline", + "gitlab.update_pipeline", + "hubspot.hubspot_create_deal" + ], + "hit1": "circleci.circleci_trigger_pipeline", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 325, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "kind": "identifier", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "ranking": [ + "terraform.tf_plan", + "terraform.tf_apply", + "terraform.tf_state_list", + "postgres.explain_query", + "salesforce.salesforce_query", + "github.get_check_run", + "docker.docker_run", + "github.delete_check_run", + "mysql.mysql_query", + "sqlite.sqlite_query" + ], + "hit1": "terraform.tf_plan", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 287, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 593, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "kind": "verb_product", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "ranking": [ + "asana.asana_get_task", + "asana.asana_create_task", + "asana.asana_update_task", + "asana.asana_create_project", + "clickup.clickup_create_task", + "asana.asana_list_tasks", + "okta.okta_create_user", + "clickup.clickup_update_task", + "github.create_issue", + "filesystem.write_file" + ], + "hit1": "asana.asana_get_task", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 841, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "ranking": [ + "trello.trello_create_card", + "trello.trello_add_comment", + "trello.trello_create_board", + "trello.trello_list_cards", + "jira.create_board", + "github.create_gist", + "gitlab.create_release", + "gitlab.create_job", + "github.create_hook", + "slack.create_canvase" + ], + "hit1": "trello.trello_create_card", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 411, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 888, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_get_task", + "clickup.clickup_create_task", + "clickup.clickup_update_task", + "clickup.clickup_list_tasks", + "asana.asana_create_task", + "github.create_gist", + "jira.create_board", + "jira.create_sprint", + "jira.create_filter", + "jira.create_version" + ], + "hit1": "clickup.clickup_get_task", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 415, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 878, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "kind": "verb_product", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "ranking": [ + "confluence.confluence_create_page", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "confluence.confluence_search", + "notion.notion_create_page", + "notion.notion_update_page", + "github.create_package", + "github.create_gist", + "jira.create_board", + "linear.create_comment" + ], + "hit1": "confluence.confluence_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 833, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "kind": "verb_product", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "discord.discord_add_reaction", + "discord.discord_get_channel", + "discord.discord_list_members", + "slack.slack_post_message", + "gmail.gmail_get_email", + "discord.discord_list_channels", + "gmail.gmail_trash_email", + "git.git_commit", + "slack.slack_add_reaction" + ], + "hit1": "discord.discord_send_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 656, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_search", + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "github.search_users", + "confluence.confluence_search", + "zendesk.zendesk_search", + "github.search_orgs", + "github.search_commits", + "slack.slack_search_messages", + "github.search_issues" + ], + "hit1": "elasticsearch.es_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 343, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 660, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "redis.redis_set", + "redis.redis_delete", + "redis.redis_keys", + "jira.jira_get_issue", + "figma.figma_get_file", + "github.get_discussion", + "aws_s3.s3_get_object", + "github.get_label", + "github.get_runner" + ], + "hit1": "redis.redis_get", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 627, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_find", + "mongodb.mongo_update", + "mongodb.mongo_insert", + "mongodb.mongo_delete", + "mongodb.mongo_list_collections", + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "figma.figma_get_comments", + "github.get_discussion_comments", + "filesystem.read_file" + ], + "hit1": "mongodb.mongo_find", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "kind": "verb_product", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "ranking": [ + "sqlite.sqlite_list_tables", + "mysql.mysql_list_tables", + "postgres.list_tables", + "sqlite.sqlite_describe_table", + "supabase.supabase_list_tables", + "sqlite.sqlite_query", + "postgres.list_schemas", + "clickup.clickup_list_tasks", + "github.list_milestones", + "linear.list_cycles" + ], + "hit1": "sqlite.sqlite_list_tables", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 240, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 536, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "kind": "verb_product", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "ranking": [ + "postgres.describe_table", + "sqlite.sqlite_describe_table", + "postgres.list_tables", + "postgres.query", + "postgres.explain_query", + "postgres.list_schemas", + "supabase.supabase_insert", + "supabase.supabase_update", + "supabase.supabase_select", + "slack.create_pin" + ], + "hit1": "postgres.describe_table", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 290, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 640, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "github.update_issue_comment", + "github.issue_read", + "gitlab.gitlab_create_issue", + "github.issue_write", + "bitbucket.bitbucket_create_issue", + "github.add_issue_comment", + "github.assign_copilot_to_issue", + "github.create_hook", + "gitlab.gitlab_get_issue" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9072278740982787, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 355, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 582, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1050, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "ranking": [ + "bitbucket.bitbucket_create_issue", + "github.create_issue", + "gitlab.gitlab_create_issue", + "github.issue_write", + "linear.linear_create_issue", + "jira.jira_create_issue", + "linear.linear_create_comment", + "github.issue_read", + "jira.jira_update_issue", + "jira.jira_get_issue" + ], + "hit1": "bitbucket.bitbucket_create_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6403591454578191, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 332, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 600, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 999, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "kind": "identifier", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "ranking": [ + "filesystem.list_directory", + "filesystem.create_directory", + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree", + "filesystem.list_allowed_directories", + "filesystem.get_file_info", + "github.get_file_contents", + "filesystem.move_file", + "github.projects_list", + "linear.list_cycles" + ], + "hit1": "filesystem.list_directory", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 643, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "kind": "paraphrase", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "ranking": [ + "sequential_thinking.sequentialthinking", + "filesystem.edit_file", + "confluence.confluence_search", + "playwright.browser_wait_for", + "brave_search.brave_web_search", + "playwright.browser_type", + "brave_search.brave_local_search", + "github.add_comment_to_pending_review", + "github.list_notifications", + "memory.add_observations" + ], + "hit1": "sequential_thinking.sequentialthinking", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 278, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 434, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 862, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "kind": "paraphrase", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "ranking": [ + "time.get_current_time", + "time.convert_time", + "playwright.browser_wait_for", + "grafana.grafana_create_annotation", + "jira.jira_add_worklog", + "google_calendar.gcal_list_events", + "github.create_milestone", + "github.create_package", + "github.create_environment", + "github.create_artifact" + ], + "hit1": "time.get_current_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 392, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 876, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "kind": "verb_product", + "gold": [ + "time.convert_time" + ], + "alts": [], + "ranking": [ + "time.convert_time", + "time.get_current_time", + "salesforce.salesforce_convert_lead", + "git.git_diff", + "memory.create_relations", + "playwright.browser_wait_for", + "grafana.grafana_create_annotation", + "google_calendar.gcal_list_events", + "jira.jira_add_worklog", + "cloudflare.cf_list_zones" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "kind": "verb_product", + "gold": [ + "github.star_repository" + ], + "alts": [], + "ranking": [ + "github.star_repository", + "github.unstar_repository", + "github.delete_repository", + "github.list_tags", + "github.get_label", + "github.fork_repository", + "github.get_tag", + "github.list_issues", + "github.list_labels", + "github.list_branches" + ], + "hit1": "github.star_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 789, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "kind": "resource", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "ranking": [ + "github.fork_repository", + "github.delete_repository", + "github.star_repository", + "github.unstar_repository", + "github.list_releases", + "github.list_dependabot_alerts", + "github.create_repository", + "github.list_discussions", + "github.list_branches", + "github.list_repository_collaborators" + ], + "hit1": "github.fork_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 758, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "kind": "verb_product", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "ranking": [ + "github.get_latest_release", + "github.get_release_by_tag", + "gitlab.get_release", + "github.get_teams", + "github.get_repository_tree", + "github.get_gist", + "github.get_label", + "github.get_tag", + "gitlab.update_release", + "gitlab.delete_release" + ], + "hit1": "github.get_latest_release", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 383, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "kind": "resource", + "gold": [ + "github.create_gist" + ], + "alts": [], + "ranking": [ + "github.create_gist", + "github.update_gist", + "github.create_hook", + "gitlab.create_job", + "gitlab.create_wiki", + "github.create_runner", + "github.create_package", + "github.create_milestone", + "gitlab.create_label", + "github.create_artifact" + ], + "hit1": "github.create_gist", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 272, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 473, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 966, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "kind": "verb_product", + "gold": [ + "github.search_code" + ], + "alts": [], + "ranking": [ + "github.search_code", + "github.get_code_scanning_alert", + "github.list_code_scanning_alerts", + "github.search_users", + "github.search_commits", + "github.search_orgs", + "github.search_issues", + "github.search_pull_requests", + "github.search_repositories", + "github.create_gist" + ], + "hit1": "github.search_code", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "kind": "verb_product", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "ranking": [ + "github.list_notifications", + "github.mark_all_notifications_read", + "github.actions_list", + "github.projects_list", + "github.list_tags", + "github.list_discussions", + "github.list_gists", + "github.list_releases", + "git.git_branch", + "github.list_branches" + ], + "hit1": "github.list_notifications", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 326, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 639, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "kind": "verb_product", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "ranking": [ + "github.assign_copilot_to_issue", + "jira.jira_assign_issue", + "linear.linear_assign_issue", + "github.create_issue", + "jira.jira_transition_issue", + "github.add_issue_comment", + "jira.jira_update_issue", + "github.request_copilot_review", + "sentry.sentry_get_issue", + "jira.jira_add_comment" + ], + "hit1": "github.assign_copilot_to_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 468, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 878, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "kind": "resource", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "ranking": [ + "github.list_secret_scanning_alerts", + "github.get_secret_scanning_alert", + "github.list_code_scanning_alerts", + "github.get_code_scanning_alert", + "github.list_dependabot_alerts", + "github.get_hook", + "gitlab.get_job", + "gitlab.get_wiki", + "github.get_runner", + "github.get_package" + ], + "hit1": "github.list_secret_scanning_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "ranking": [ + "github.create_or_update_file", + "github.projects_write", + "github.update_gist", + "github.create_gist", + "github.label_write", + "github.update_issue_comment", + "slack.update_file", + "github.issue_write", + "github.update_pull_request", + "github.create_pull_request" + ], + "hit1": "github.create_or_update_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 527, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1127, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "kind": "resource", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "ranking": [ + "github.list_repository_collaborators", + "github.list_commits", + "trello.trello_list_cards", + "github.get_discussion_comments", + "github.request_copilot_review", + "github.list_labels", + "github.list_discussions", + "github.list_branches", + "github.update_issue_comment", + "github.list_tags" + ], + "hit1": "github.list_repository_collaborators", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 278, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 472, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 860, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "kind": "verb_product", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "ranking": [ + "slack.slack_reply_to_thread", + "slack.slack_get_thread_replies", + "slack.slack_join_channel", + "slack.get_reminder", + "slack.delete_reminder", + "slack.slack_create_channel", + "slack.get_canvase", + "slack.slack_leave_channel", + "slack.update_reminder", + "slack.slack_list_channels" + ], + "hit1": "slack.slack_reply_to_thread", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 668, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "kind": "verb_product", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "ranking": [ + "slack.slack_add_reaction", + "discord.discord_add_reaction", + "slack.create_reminder", + "slack.get_reminder", + "slack.create_canvase", + "slack.create_bookmark", + "slack.delete_reminder", + "slack.slack_join_channel", + "slack.slack_create_channel", + "slack.update_reminder" + ], + "hit1": "slack.slack_add_reaction", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 302, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 465, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 816, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "kind": "verb_product", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "ranking": [ + "slack.slack_search_messages", + "slack.slack_get_channel_history", + "slack.slack_post_message", + "slack.slack_add_reaction", + "twilio.twilio_list_messages", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "gmail.gmail_search_emails", + "slack.get_canvase", + "slack.get_reminder" + ], + "hit1": "slack.slack_search_messages", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 661, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "kind": "verb_product", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "ranking": [ + "slack.slack_join_channel", + "slack.slack_create_channel", + "slack.slack_leave_channel", + "slack.slack_set_channel_topic", + "slack.slack_post_message", + "slack.slack_get_channel_history", + "slack.slack_get_thread_replies", + "slack.slack_reply_to_thread", + "slack.slack_add_reaction", + "slack.slack_upload_file" + ], + "hit1": "slack.slack_join_channel", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "kind": "verb_product", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_transition_issue", + "jira.jira_assign_issue", + "jira.jira_get_issue", + "jira.jira_create_issue", + "jira.jira_update_issue", + "jira.jira_add_comment", + "jira.jira_add_worklog", + "jira.jira_search_issues", + "jira.create_board", + "jira.jira_list_projects" + ], + "hit1": "jira.jira_transition_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 748, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "kind": "identifier", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "ranking": [ + "jira.jira_search_issues", + "jira.jira_list_projects", + "jira.get_board", + "jira.jira_get_issue", + "jira.get_webhook", + "jira.jira_add_worklog", + "jira.get_sprint", + "github.search_issues", + "confluence.confluence_search", + "jira.delete_board" + ], + "hit1": "jira.jira_search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 283, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 617, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "kind": "verb_product", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "ranking": [ + "jira.jira_add_worklog", + "jira.jira_add_comment", + "jira.jira_transition_issue", + "jira.get_board", + "jira.jira_get_issue", + "jira.get_sprint", + "jira.get_webhook", + "jira.jira_assign_issue", + "jira.delete_board", + "jira.jira_list_projects" + ], + "hit1": "jira.jira_add_worklog", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 610, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "kind": "verb_product", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "ranking": [ + "linear.linear_list_teams", + "linear.linear_list_projects", + "linear.linear_list_issues", + "linear.linear_create_project", + "linear.list_labels", + "linear.list_cycles", + "linear.list_users", + "linear.linear_search_issues", + "linear.list_comments", + "linear.list_attachments" + ], + "hit1": "linear.linear_list_teams", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 690, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "ranking": [ + "linear.linear_create_comment", + "linear.get_comment", + "linear.linear_assign_issue", + "linear.linear_get_issue", + "linear.linear_update_issue", + "linear.delete_comment", + "linear.linear_create_issue", + "linear.create_comment", + "linear.update_comment", + "github.update_issue_comment" + ], + "hit1": "linear.linear_create_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 383, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 831, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "kind": "paraphrase", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "ranking": [ + "filesystem.read_text_file", + "filesystem.read_file", + "filesystem.move_file", + "filesystem.read_media_file", + "slack.delete_file", + "figma.figma_get_file", + "slack.update_file", + "google_drive.gdrive_share_file", + "slack.get_file", + "google_drive.gdrive_update_file" + ], + "hit1": "filesystem.read_text_file", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "kind": "paraphrase", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "ranking": [ + "filesystem.search_files", + "filesystem.directory_tree", + "redis.redis_keys", + "slack.list_files", + "google_drive.gdrive_search", + "linear.linear_search_issues", + "filesystem.list_directory", + "github.search_repositories", + "filesystem.list_directory_with_sizes", + "github.push_files" + ], + "hit1": "filesystem.search_files", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 696, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "kind": "paraphrase", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "ranking": [ + "filesystem.edit_file", + "filesystem.list_directory_with_sizes", + "slack.get_file", + "slack.delete_file", + "filesystem.write_file", + "slack.update_file", + "slack.create_file", + "figma.figma_get_file", + "github.get_file_blame", + "google_drive.gdrive_create_file" + ], + "hit1": "filesystem.edit_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 776, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "ranking": [ + "memory.read_graph", + "memory.delete_relations", + "memory.create_entities", + "memory.create_relations", + "memory.open_nodes", + "memory.add_observations", + "memory.delete_entities", + "memory.search_nodes", + "filesystem.read_file", + "filesystem.read_text_file" + ], + "hit1": "memory.read_graph", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 253, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 561, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "kind": "verb_product", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "memory.open_nodes", + "github.search_users", + "github.search_issues", + "notion.notion_search", + "confluence.confluence_search", + "slack.slack_search_messages", + "memory.create_entities", + "zendesk.zendesk_search", + "elasticsearch.es_search" + ], + "hit1": "memory.search_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 176, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 331, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 625, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "kind": "verb_product", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "ranking": [ + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_diff", + "git.git_show", + "git.git_status", + "git.git_reset", + "git.git_log", + "git.git_commit", + "git.git_add", + "github.unstar_repository" + ], + "hit1": "git.git_diff_unstaged", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 613, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "kind": "identifier", + "gold": [ + "git.git_add" + ], + "alts": [], + "ranking": [ + "git.git_add", + "filesystem.list_directory_with_sizes", + "git.git_reset", + "git.git_commit", + "slack.list_files", + "github.push_files", + "git.git_diff_staged", + "git.git_checkout", + "git.git_diff_unstaged", + "github.update_gist" + ], + "hit1": "git.git_add", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 332, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "kind": "verb_product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "ranking": [ + "shopify.shopify_create_product", + "shopify.shopify_get_product", + "shopify.shopify_update_product", + "shopify.shopify_create_order", + "shopify.shopify_list_products", + "stripe.stripe_create_product", + "shopify.shopify_get_order", + "stripe.stripe_create_price", + "shopify.shopify_list_customers", + "shopify.shopify_list_orders" + ], + "hit1": "shopify.shopify_create_product", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "kind": "verb_product", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "ranking": [ + "stripe.stripe_cancel_subscription", + "stripe.stripe_get_customer", + "stripe.stripe_create_customer", + "stripe.stripe_create_product", + "stripe.stripe_list_subscriptions", + "stripe.stripe_create_invoice", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_refund", + "stripe.stripe_create_price", + "stripe.stripe_create_payment_intent" + ], + "hit1": "stripe.stripe_cancel_subscription", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "kind": "resource", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_refund", + "stripe.stripe_get_payment_intent", + "github.create_environment", + "github.create_package", + "jira.create_attachment", + "jira.create_sprint", + "linear.create_comment", + "notion.create_comment", + "linear.create_attachment" + ], + "hit1": "stripe.stripe_create_payment_intent", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 424, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 906, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "memory.read_graph", + "github.delete_package", + "memory.create_entities", + "github.update_package" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "kind": "verb_product", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "ranking": [ + "datadog.datadog_get_monitor", + "datadog.datadog_create_monitor", + "datadog.datadog_list_monitors", + "github.create_gist", + "filesystem.create_directory", + "notion.notion_create_database", + "github.create_branch", + "github.create_repository", + "github.create_hook", + "gitlab.create_job" + ], + "hit1": "datadog.datadog_get_monitor", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 392, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 851, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "kind": "paraphrase", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "ranking": [ + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "sentry.sentry_list_events", + "pagerduty.pagerduty_resolve_incident", + "sentry.sentry_list_issues", + "jira.jira_assign_issue", + "sentry.sentry_list_projects", + "github.issue_write", + "github.issue_read", + "jira.jira_get_issue" + ], + "hit1": "sentry.sentry_get_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 817, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_delete", + "kubernetes.k8s_apply", + "vercel.vercel_get_deployment", + "kubernetes.k8s_get_events", + "kubernetes.k8s_list_services", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_pod_logs" + ], + "hit1": "kubernetes.k8s_scale_deployment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_apply", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_delete", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_get_events", + "kubernetes.k8s_pod_logs", + "terraform.tf_apply" + ], + "hit1": "kubernetes.k8s_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 253, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "ranking": [ + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_run", + "docker.docker_exec", + "docker.docker_ps", + "docker.docker_build", + "docker.docker_images", + "kubernetes.k8s_pod_logs", + "stripe.stripe_create_customer" + ], + "hit1": "docker.docker_stop", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 655, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "kind": "verb_product", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "ranking": [ + "docker.docker_build", + "docker.docker_run", + "docker.docker_images", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_ps", + "docker.docker_exec", + "filesystem.read_media_file", + "jira.delete_filter" + ], + "hit1": "docker.docker_build", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 598, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "kind": "verb_product", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "ranking": [ + "playwright.browser_click", + "puppeteer.puppeteer_click", + "playwright.browser_type", + "playwright.browser_take_screenshot", + "playwright.browser_snapshot", + "playwright.browser_navigate", + "playwright.browser_tab_close", + "playwright.browser_tab_list", + "playwright.browser_fill_form", + "playwright.browser_tab_new" + ], + "hit1": "playwright.browser_click", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 564, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "kind": "paraphrase", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "ranking": [ + "playwright.browser_type", + "playwright.browser_navigate", + "playwright.browser_wait_for", + "playwright.browser_tab_close", + "playwright.browser_tab_new", + "playwright.browser_tab_list", + "supabase.supabase_insert", + "playwright.browser_press_key", + "playwright.browser_fill_form", + "slack.slack_post_message" + ], + "hit1": "playwright.browser_type", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 316, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 589, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_list_projects", + "github.projects_list", + "linear.linear_list_projects", + "gitlab.gitlab_list_issues", + "jira.jira_list_projects", + "gitlab.gitlab_list_merge_requests", + "vercel.vercel_list_projects", + "sentry.sentry_list_projects", + "git.git_branch", + "github.projects_get" + ], + "hit1": "gitlab.gitlab_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 289, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 564, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "kind": "verb_product", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "ranking": [ + "bitbucket.bitbucket_create_pr", + "bitbucket.bitbucket_list_prs", + "bitbucket.bitbucket_create_issue", + "github.pull_request_read", + "github.create_pull_request", + "github.merge_pull_request", + "github.pull_request_review_write", + "bitbucket.bitbucket_list_repos", + "github.update_pull_request_branch", + "github.list_pull_requests" + ], + "hit1": "bitbucket.bitbucket_create_pr", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 553, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1037, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "kind": "verb_product", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "ranking": [ + "gmail.gmail_create_draft", + "gmail.gmail_trash_email", + "github.create_gist", + "github.create_pull_request", + "gmail.gmail_get_email", + "stripe.stripe_create_customer", + "gitlab.create_job", + "github.create_hook", + "gitlab.create_wiki", + "github.create_branch" + ], + "hit1": "gmail.gmail_create_draft", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 439, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 903, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "kind": "verb_product", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_share_file", + "google_drive.gdrive_update_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_list_folder", + "google_drive.gdrive_create_file", + "google_drive.gdrive_search", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "gitlab.gitlab_get_file", + "google_calendar.gcal_list_calendars" + ], + "hit1": "google_drive.gdrive_share_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 762, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "kind": "verb_product", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_list_events", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_list_calendars", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "kubernetes.k8s_get_events", + "sentry.sentry_list_events", + "google_drive.gdrive_list_folder", + "google_drive.gdrive_get_file", + "google_drive.gdrive_update_file" + ], + "hit1": "google_calendar.gcal_list_events", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 754, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_list_events", + "github.delete_environment", + "linear.delete_comment", + "github.delete_hook", + "linear.delete_cycle", + "gitlab.delete_job", + "google_calendar.gcal_list_calendars" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 440, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 685, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_projects", + "vercel.vercel_list_deployments", + "linear.linear_list_projects", + "vercel.vercel_create_project", + "jira.jira_list_projects", + "github.projects_list", + "gitlab.gitlab_list_projects", + "sentry.sentry_list_projects", + "vercel.vercel_get_deployment", + "github.projects_get" + ], + "hit1": "vercel.vercel_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 255, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "ranking": [ + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update", + "supabase.supabase_invoke_fn", + "supabase.supabase_list_tables", + "aws_s3.s3_get_object", + "mongodb.mongo_delete", + "memory.delete_observations", + "docker.docker_logs", + "slack.slack_get_channel_history" + ], + "hit1": "supabase.supabase_select", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 688, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "kind": "verb_product", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "figma.figma_get_comments", + "linear.get_comment", + "figma.figma_get_file", + "linear.update_comment", + "notion.get_comment", + "linear.delete_comment", + "notion.update_comment", + "linear.create_comment", + "notion.delete_comment" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 201, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 639, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "kind": "verb_product", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "ranking": [ + "okta.okta_list_users", + "okta.okta_get_user", + "linear.list_users", + "okta.okta_create_user", + "notion.notion_list_users", + "slack.slack_get_users", + "okta.okta_deactivate_user", + "slack.slack_list_users", + "notion.list_users_tokens", + "pagerduty.pagerduty_list_oncalls" + ], + "hit1": "okta.okta_list_users", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 198, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 288, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 618, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query_range", + "prometheus.prom_query", + "prometheus.prom_list_metrics", + "postgres.explain_query", + "datadog.datadog_query_metrics", + "mysql.mysql_query", + "sqlite.sqlite_query", + "github.search_users", + "memory.search_nodes", + "notion.notion_query_database" + ], + "hit1": "prometheus.prom_query_range", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 674, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "kind": "verb_product", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "ranking": [ + "grafana.grafana_create_annotation", + "github.create_gist", + "grafana.grafana_get_dashboard", + "github.create_artifact", + "gitlab.create_job", + "github.create_hook", + "jira.create_attachment", + "gitlab.create_wiki", + "notion.create_comment", + "gitlab.create_label" + ], + "hit1": "grafana.grafana_create_annotation", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 435, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 923, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "kind": "identifier", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "ranking": [ + "terraform.tf_apply", + "terraform.tf_plan", + "terraform.tf_state_list", + "kubernetes.k8s_apply", + "slack.slack_reply_to_thread", + "playwright.browser_fill_form", + "filesystem.search_files", + "linear.linear_list_issues", + "docker.docker_rm", + "linear.linear_search_issues" + ], + "hit1": "terraform.tf_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 644, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "kind": "verb_product", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "ranking": [ + "asana.asana_list_tasks", + "asana.asana_get_task", + "asana.asana_create_task", + "asana.asana_update_task", + "clickup.clickup_list_tasks", + "aws_s3.s3_list_buckets", + "asana.asana_create_project", + "trello.trello_list_cards", + "github.list_hooks", + "linear.list_attachments" + ], + "hit1": "asana.asana_list_tasks", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "ranking": [ + "trello.trello_add_comment", + "trello.trello_create_card", + "trello.trello_list_cards", + "trello.trello_create_board", + "figma.figma_post_comment", + "github.update_issue_comment", + "linear.get_comment", + "notion.get_comment", + "notion.delete_comment", + "linear.delete_comment" + ], + "hit1": "trello.trello_add_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_update_task", + "clickup.clickup_get_task", + "clickup.clickup_create_task", + "clickup.clickup_list_tasks", + "asana.asana_update_task", + "github.update_hook", + "linear.update_cycle", + "gitlab.update_job", + "linear.update_comment", + "slack.update_canvase" + ], + "hit1": "clickup.clickup_update_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 736, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "kind": "verb_product", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "ranking": [ + "confluence.confluence_search", + "confluence.confluence_get_page", + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "github.search_users", + "github.search_commits", + "brave_search.brave_web_search", + "github.search_orgs", + "elasticsearch.es_search", + "github.search_issues" + ], + "hit1": "confluence.confluence_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 708, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_deal", + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "github.create_gist", + "github.create_repository", + "github.create_hook", + "hubspot.hubspot_list_contacts", + "github.create_branch", + "github.create_runner", + "github.create_milestone" + ], + "hit1": "hubspot.hubspot_create_deal", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 413, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 860, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_search", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_create_ticket", + "aws_s3.s3_list_buckets", + "clickup.clickup_list_tasks", + "gitlab.list_snippets", + "kubernetes.k8s_list_deployments", + "aws_s3.s3_list_objects" + ], + "hit1": "zendesk.zendesk_list_tickets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "kind": "verb_product", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "ranking": [ + "discord.discord_list_channels", + "discord.discord_get_channel", + "discord.discord_list_members", + "discord.discord_send_message", + "discord.discord_add_reaction", + "slack.slack_list_channels", + "linear.list_labels", + "trello.trello_list_cards", + "gitlab.list_labels", + "clickup.clickup_list_tasks" + ], + "hit1": "discord.discord_list_channels", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 166, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 666, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "ranking": [ + "redis.redis_set", + "redis.redis_get", + "redis.redis_delete", + "redis.redis_keys", + "jira.jira_get_issue", + "figma.figma_get_file", + "aws_s3.s3_get_object", + "slack.slack_set_channel_topic", + "playwright.browser_press_key", + "figma.figma_get_comments" + ], + "hit1": "redis.redis_set", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 586, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert", + "mongodb.mongo_update", + "mongodb.mongo_find", + "mongodb.mongo_delete", + "mongodb.mongo_list_collections", + "supabase.supabase_insert", + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "linear.list_comments", + "notion.list_comments" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "kind": "verb_product", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "mysql.mysql_query", + "postgres.explain_query", + "sqlite.sqlite_query", + "salesforce.salesforce_query", + "mysql.mysql_list_tables", + "prometheus.prom_query", + "notion.notion_query_database", + "memory.search_nodes", + "github.search_users" + ], + "hit1": "postgres.query", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 585, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_index_doc", + "elasticsearch.es_search", + "elasticsearch.es_get_doc", + "playwright.browser_tab_close", + "github.get_environment", + "github.delete_environment", + "github.update_environment", + "github.create_environment", + "mongodb.mongo_find", + "github.create_milestone" + ], + "hit1": "elasticsearch.es_index_doc", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 734, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_put_object", + "aws_s3.s3_list_objects", + "okta.okta_get_user", + "asana.asana_get_task", + "gmail.gmail_send_email", + "github.projects_get", + "github.get_artifact", + "github.get_hook" + ], + "hit1": "aws_s3.s3_get_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 691, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones", + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_dns", + "cloudflare.cf_create_dns", + "clickup.clickup_list_tasks", + "github.list_milestones", + "gitlab.list_milestones", + "gitlab.list_pipelines", + "github.list_packages", + "linear.list_cycles" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_create_customer", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "stripe.stripe_create_refund", + "stripe.stripe_list_invoices", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_customer", + "gitlab.create_pipeline", + "github.create_package" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 732, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "kind": "verb_product", + "gold": [ + "github.actions_list" + ], + "alts": [], + "ranking": [ + "github.actions_list", + "github.actions_get", + "github.actions_run_trigger", + "github.actions_get_job_logs", + "github.list_discussions", + "github.projects_list", + "github.list_tags", + "github.list_gists", + "github.list_releases", + "github.list_notifications" + ], + "hit1": "github.actions_list", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 783, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "kind": "verb_product", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "ranking": [ + "github.actions_get_job_logs", + "github.actions_get", + "github.actions_list", + "github.actions_run_trigger", + "github.get_gist", + "gitlab.get_job", + "github.projects_get", + "github.get_label", + "github.get_discussion", + "github.get_latest_release" + ], + "hit1": "github.actions_get_job_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 440, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 802, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "kind": "verb_product", + "gold": [ + "github.create_repository" + ], + "alts": [], + "ranking": [ + "github.create_repository", + "github.delete_repository", + "github.star_repository", + "github.fork_repository", + "github.unstar_repository", + "github.create_branch", + "github.list_repository_collaborators", + "github.create_issue", + "github.get_repository_tree", + "github.list_tags" + ], + "hit1": "github.create_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 874, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "kind": "verb_product", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "ranking": [ + "git.git_checkout", + "git.git_branch", + "git.git_create_branch", + "github.create_branch", + "git.git_diff", + "github.update_pull_request_branch", + "github.delete_file", + "github.list_commits", + "git.git_show", + "github.push_files" + ], + "hit1": "git.git_checkout", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 376, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 864, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "kind": "identifier", + "gold": [ + "git.git_log" + ], + "alts": [], + "ranking": [ + "git.git_log", + "git.git_show", + "git.git_diff", + "git.git_status", + "git.git_diff_staged", + "git.git_diff_unstaged", + "git.git_branch", + "git.git_create_branch", + "git.git_checkout", + "git.git_reset" + ], + "hit1": "git.git_log", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 309, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 603, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "kind": "verb_product", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "ranking": [ + "postgres.list_schemas", + "postgres.list_tables", + "postgres.query", + "postgres.describe_table", + "postgres.explain_query", + "kubernetes.k8s_list_services", + "aws_s3.s3_list_buckets", + "github.list_packages", + "gitlab.list_pipelines", + "linear.list_cycles" + ], + "hit1": "postgres.list_schemas", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 158, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 596, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "kind": "verb_product", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "ranking": [ + "memory.open_nodes", + "memory.search_nodes", + "memory.read_graph", + "memory.create_entities", + "memory.create_relations", + "memory.delete_relations", + "memory.add_observations", + "memory.delete_entities", + "sqlite.sqlite_list_tables", + "postgres.list_schemas" + ], + "hit1": "memory.open_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 492, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "kind": "paraphrase", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "ranking": [ + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.get_file_info", + "filesystem.read_media_file", + "google_drive.gdrive_get_file", + "gitlab.gitlab_get_file", + "filesystem.list_allowed_directories", + "filesystem.move_file", + "filesystem.edit_file", + "figma.figma_get_file_nodes" + ], + "hit1": "filesystem.read_file", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 220, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "kind": "paraphrase", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "ranking": [ + "filesystem.write_file", + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.edit_file", + "filesystem.read_media_file", + "playwright.browser_tab_new", + "slack.delete_file", + "slack.update_file", + "figma.figma_get_file", + "slack.create_file" + ], + "hit1": "filesystem.write_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 384, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "kind": "verb_product", + "gold": [ + "github.search_issues" + ], + "alts": [], + "ranking": [ + "github.search_issues", + "github.list_issues", + "github.search_users", + "jira.jira_search_issues", + "github.search_orgs", + "linear.linear_search_issues", + "github.search_pull_requests", + "github.search_commits", + "github.search_code", + "gitlab.gitlab_list_issues" + ], + "hit1": "github.search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 683, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "kind": "paraphrase", + "gold": [ + "github.get_me" + ], + "alts": [], + "ranking": [ + "github.get_teams", + "github.get_me", + "github.list_notifications", + "github.get_latest_release", + "github.get_repository_tree", + "github.projects_get", + "github.get_gist", + "github.get_label", + "github.get_tag", + "github.list_gists" + ], + "hit1": "github.get_teams", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "cost": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.02, + "p95_calls": 1.0, + "success_within_1": 0.9866666666666667, + "success_within_2": 0.9933333333333333, + "success_within_3": 1.0, + "mean_tokens": 236.66666666666666, + "p95_tokens": 333.0, + "total_tokens": 35500, + "tokens_per_successful_query": 236.66666666666666, + "total_tokens_successful": 35500 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.0066666666666666, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 376.52, + "p95_tokens": 516.5999999999998, + "total_tokens": 56478, + "tokens_per_successful_query": 376.52, + "total_tokens_successful": 56478 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.0, + "p95_calls": 1.0, + "success_within_1": 1.0, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 740.5866666666667, + "p95_tokens": 1001.1999999999999, + "total_tokens": 111088, + "tokens_per_successful_query": 740.5866666666667, + "total_tokens_successful": 111088 + } + } +} diff --git a/experiments/mcp-tool-search/results/lexical_rerank_weighted_top20.csv b/experiments/mcp-tool-search/results/lexical_rerank_weighted_top20.csv new file mode 100644 index 00000000..6fd6da05 --- /dev/null +++ b/experiments/mcp-tool-search/results/lexical_rerank_weighted_top20.csv @@ -0,0 +1,151 @@ +id,query,kind,hit1,recall@1,recall@3,recall@5,mrr@10,ndcg@10,hit_rank,k3_calls,k3_tokens,k3_success1,k5_calls,k5_tokens,k5_success1,k10_calls,k10_tokens,k10_success1,ranking +q001,Create a GitHub issue,verb_product,github.create_issue,1.0,1.0,1.0,1.0000,0.9502,1,1,388,1.0,1,598,1.0,1,955,1.0,github.create_issue github.issue_read github.issue_write github.add_issue_comment github.update_issue_comment github.assign_copilot_to_issue git.git_checkout git.git_show git.git_branch git.git_create_branch +q002,Create a Jira ticket,verb_product,jira.jira_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,312,1.0,1,412,1.0,1,763,1.0,jira.jira_create_issue jira.jira_create_sprint jira.jira_assign_issue jira.jira_list_projects jira.jira_update_issue jira.jira_get_issue jira.jira_search_issues jira.jira_add_comment jira.jira_add_worklog jira.jira_transition_issue +q003,Send a Slack message,verb_product,slack.slack_post_message,1.0,1.0,1.0,1.0000,1.0000,1,1,232,1.0,1,356,1.0,1,707,1.0,slack.slack_post_message slack.slack_add_reaction slack.slack_join_channel slack.slack_leave_channel slack.slack_create_channel slack.slack_search_messages slack.slack_upload_file slack.slack_list_users slack.slack_get_users slack.slack_list_channels +q004,List open pull requests,resource,github.list_pull_requests,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,429,1.0,1,1035,1.0,github.list_pull_requests github.search_pull_requests gitlab.gitlab_list_merge_requests github.merge_pull_request github.pull_request_read github.create_pull_request github.update_pull_request github.update_pull_request_branch github.pull_request_review_write bitbucket.bitbucket_list_prs +q005,Open a pull request on GitHub,verb_product,github.create_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,368,1.0,1,653,1.0,1,1052,1.0,github.create_pull_request github.merge_pull_request github.pull_request_read github.update_pull_request github.pull_request_review_write github.request_copilot_review github.update_pull_request_branch github.search_pull_requests github.list_pull_requests git.git_checkout +q006,Run a SQL query against postgres,verb_product,postgres.query,1.0,1.0,1.0,1.0000,1.0000,1,1,167,1.0,1,280,1.0,1,544,1.0,postgres.query postgres.explain_query postgres.list_schemas postgres.list_tables postgres.describe_table mysql.mysql_query sqlite.sqlite_query salesforce.salesforce_query mysql.mysql_list_tables prometheus.prom_query +q007,Read a local file from disk,paraphrase,filesystem.read_file,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,385,1.0,1,782,1.0,filesystem.read_file filesystem.read_text_file filesystem.read_media_file filesystem.write_file filesystem.edit_file filesystem.get_file_info filesystem.move_file brave_search.brave_local_search github.delete_file gitlab.gitlab_get_file +q008,What files are in this folder,paraphrase,filesystem.search_files,0.0,0.0,0.0,0.1667,0.4348,6,2,473,0.0,2,730,0.0,1,708,1.0,filesystem.search_files google_drive.gdrive_list_folder slack.list_files github.push_files filesystem.move_file filesystem.list_directory filesystem.list_directory_with_sizes filesystem.directory_tree filesystem.read_file filesystem.read_text_file +q009,Remember this fact in the knowledge graph,paraphrase,memory.read_graph,0.0,1.0,1.0,0.5000,0.5672,2,1,156,1.0,1,262,1.0,1,524,1.0,memory.read_graph memory.create_entities memory.create_relations memory.add_observations memory.open_nodes memory.delete_relations memory.search_nodes memory.delete_entities memory.delete_observations time.get_current_time +q010,Search the web with Brave,verb_product,brave_search.brave_web_search,1.0,1.0,1.0,1.0000,1.0000,1,1,245,1.0,1,412,1.0,1,726,1.0,brave_search.brave_web_search brave_search.brave_local_search filesystem.list_directory_with_sizes jira.jira_search_issues github.update_pull_request_branch memory.search_nodes puppeteer.puppeteer_screenshot puppeteer.puppeteer_navigate puppeteer.puppeteer_click puppeteer.puppeteer_fill +q011,Fetch the contents of a URL,verb_product,fetch.fetch,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,372,1.0,1,707,1.0,fetch.fetch github.get_file_contents playwright.browser_navigate filesystem.read_file puppeteer.puppeteer_navigate git.git_add slack.slack_set_channel_topic github.get_repository_tree git.git_status playwright.browser_snapshot +q012,Show git status,identifier,git.git_status,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,304,1.0,1,600,1.0,git.git_status git.git_show git.git_diff git.git_checkout git.git_branch git.git_log git.git_diff_unstaged git.git_diff_staged git.git_commit git.git_add +q013,Commit staged changes,paraphrase,git.git_commit,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,328,1.0,1,663,1.0,git.git_commit github.get_commit git.git_diff_staged git.git_reset git.git_log git.git_show github.push_files +q014,Create a Linear issue,verb_product,linear.linear_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,340,1.0,1,463,1.0,1,765,1.0,linear.linear_create_issue linear.linear_update_issue linear.linear_assign_issue linear.linear_get_issue linear.linear_create_comment linear.linear_create_project linear.linear_list_teams linear.linear_list_projects linear.linear_list_issues linear.linear_search_issues +q015,Make a task in my project tracker Linear,paraphrase,linear.linear_create_project,0.0,1.0,1.0,0.5000,0.6309,2,1,352,1.0,1,432,1.0,1,769,1.0,linear.linear_create_project linear.linear_create_issue linear.linear_update_issue linear.linear_list_teams linear.linear_list_projects linear.linear_list_issues linear.linear_search_issues linear.linear_create_comment linear.linear_assign_issue linear.linear_get_issue +q016,Post a comment on a GitHub issue,verb_product,github.update_issue_comment,0.0,1.0,1.0,0.5000,0.6309,2,1,333,1.0,1,573,1.0,1,1028,1.0,github.update_issue_comment github.add_issue_comment github.issue_read github.assign_copilot_to_issue github.issue_write github.create_issue github.add_comment_to_pending_review git.git_checkout git.git_show git.git_branch +q017,Merge the pull request,ambiguous,github.merge_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,283,1.0,1,468,1.0,1,1121,1.0,github.merge_pull_request gitlab.gitlab_merge_merge_request gitlab.gitlab_get_merge_request github.update_pull_request_branch github.pull_request_read github.create_pull_request github.update_pull_request gitlab.gitlab_create_merge_request github.pull_request_review_write github.request_copilot_review +q018,Trigger a GitHub Actions workflow,verb_product,github.actions_run_trigger,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,450,1.0,1,737,1.0,github.actions_run_trigger github.actions_get github.actions_list github.actions_get_job_logs git.git_checkout git.git_show git.git_branch git.git_diff git.git_commit git.git_status +q019,Get Dependabot alerts,resource,github.list_dependabot_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,287,1.0,1,445,1.0,1,798,1.0,github.list_dependabot_alerts github.get_dependabot_alert github.list_code_scanning_alerts github.list_secret_scanning_alerts kubernetes.k8s_get_pod redis.redis_get github.get_discussion github.projects_get github.get_label aws_s3.s3_get_object +q020,List Sentry issues,verb_product,sentry.sentry_list_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,396,1.0,1,796,1.0,sentry.sentry_list_issues sentry.sentry_list_projects sentry.sentry_list_events sentry.sentry_get_issue sentry.sentry_update_issue github.list_issues github.search_issues gitlab.gitlab_list_issues linear.linear_list_issues linear.linear_search_issues +q021,Create a Stripe customer,verb_product,stripe.stripe_create_customer,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,300,1.0,1,669,1.0,stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_list_subscriptions stripe.stripe_create_payment_intent stripe.stripe_create_product stripe.stripe_create_price stripe.stripe_create_refund stripe.stripe_list_customers +q022,Refund a Stripe payment,paraphrase,stripe.stripe_create_refund,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,352,1.0,1,662,1.0,stripe.stripe_create_refund stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_list_customers stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_create_product stripe.stripe_create_price +q023,Send an email with Gmail,verb_product,gmail.gmail_send_email,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,389,1.0,1,685,1.0,gmail.gmail_send_email gmail.gmail_get_email gmail.gmail_trash_email gmail.gmail_modify_email gmail.gmail_search_emails gmail.gmail_create_draft gmail.gmail_list_labels filesystem.list_directory_with_sizes puppeteer.puppeteer_click puppeteer.puppeteer_fill +q024,Search my Gmail inbox,verb_product,gmail.gmail_search_emails,1.0,1.0,1.0,1.0000,1.0000,1,1,205,1.0,1,352,1.0,1,714,1.0,gmail.gmail_search_emails gmail.gmail_create_draft gmail.gmail_list_labels gmail.gmail_send_email gmail.gmail_get_email gmail.gmail_trash_email gmail.gmail_modify_email mysql.mysql_list_tables brave_search.brave_web_search brave_search.brave_local_search +q025,Upload a file to Google Drive,paraphrase,google_drive.gdrive_update_file,0.0,0.0,1.0,0.2500,0.4307,4,2,496,0.0,1,418,1.0,1,829,1.0,google_drive.gdrive_update_file google_drive.gdrive_share_file google_drive.gdrive_get_file google_drive.gdrive_create_file google_drive.gdrive_search google_drive.gdrive_list_folder slack.slack_upload_file gitlab.gitlab_get_file filesystem.edit_file slack.slack_reply_to_thread +q026,Create a calendar event,resource,google_calendar.gcal_create_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,427,1.0,1,895,1.0,google_calendar.gcal_create_event google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars github.create_gist github.create_milestone github.create_hook github.create_environment github.create_package +q027,List Kubernetes pods,verb_product,kubernetes.k8s_list_pods,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,314,1.0,1,718,1.0,kubernetes.k8s_list_pods kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_get_pod kubernetes.k8s_apply kubernetes.k8s_delete kubernetes.k8s_scale_deployment kubernetes.k8s_pod_logs clickup.clickup_list_tasks +q028,Get logs from a k8s pod,verb_product,kubernetes.k8s_pod_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,262,1.0,1,455,1.0,1,834,1.0,kubernetes.k8s_pod_logs kubernetes.k8s_get_pod docker.docker_logs datadog.datadog_list_logs github.actions_get_job_logs aws_s3.s3_get_object elasticsearch.es_get_doc git.git_log gitlab.gitlab_get_file github.get_tag +q029,Run a docker container,verb_product,docker.docker_run,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,345,1.0,1,644,1.0,docker.docker_run docker.docker_stop docker.docker_rm docker.docker_exec docker.docker_logs docker.docker_images docker.docker_build docker.docker_ps github.create_check_run github.get_check_run +q030,Show docker container logs,verb_product,docker.docker_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,345,1.0,1,686,1.0,docker.docker_logs docker.docker_stop docker.docker_rm docker.docker_run docker.docker_exec docker.docker_images docker.docker_build docker.docker_ps kubernetes.k8s_pod_logs datadog.datadog_list_logs +q031,Navigate the browser to a URL,ambiguous,playwright.browser_navigate,1.0,1.0,1.0,1.0000,0.9502,1,1,180,1.0,1,294,1.0,1,536,1.0,playwright.browser_navigate playwright.browser_tab_new puppeteer.puppeteer_navigate playwright.browser_click playwright.browser_evaluate playwright.browser_tab_close playwright.browser_snapshot playwright.browser_tab_list playwright.browser_press_key playwright.browser_wait_for +q032,Take a screenshot of the page,ambiguous,playwright.browser_take_screenshot,1.0,1.0,1.0,1.0000,1.0000,1,1,226,1.0,1,352,1.0,1,711,1.0,playwright.browser_take_screenshot puppeteer.puppeteer_screenshot notion.notion_get_page notion.notion_update_page confluence.confluence_get_page confluence.confluence_create_page confluence.confluence_update_page notion.notion_create_page playwright.browser_snapshot playwright.browser_press_key +q033,Create a GitLab merge request,verb_product,gitlab.gitlab_merge_merge_request,0.0,1.0,1.0,0.3333,0.5000,3,1,297,1.0,1,475,1.0,1,792,1.0,gitlab.gitlab_merge_merge_request gitlab.gitlab_get_merge_request gitlab.gitlab_create_merge_request github.merge_pull_request gitlab.gitlab_list_merge_requests git.git_checkout git.git_show git.git_branch git.git_create_branch git.git_diff +q034,Search Notion pages,verb_product,notion.notion_search,1.0,1.0,1.0,1.0000,1.0000,1,1,195,1.0,1,366,1.0,1,686,1.0,notion.notion_search notion.notion_get_page notion.notion_get_block notion.notion_query_database notion.notion_create_database notion.notion_update_page notion.notion_list_users notion.notion_get_comments notion.notion_append_block notion.notion_create_page +q035,Create a Notion page,verb_product,notion.notion_get_page,0.0,1.0,1.0,0.3333,0.5000,3,1,230,1.0,1,371,1.0,1,687,1.0,notion.notion_get_page notion.notion_update_page notion.notion_create_page notion.notion_get_comments notion.notion_create_database notion.notion_search notion.notion_get_block notion.notion_query_database notion.notion_list_users notion.notion_append_block +q036,Query a Notion database,verb_product,notion.notion_query_database,1.0,1.0,1.0,1.0000,1.0000,1,1,279,1.0,1,401,1.0,1,687,1.0,notion.notion_query_database notion.notion_create_database notion.notion_create_page notion.notion_search notion.notion_get_page notion.notion_get_block notion.notion_update_page notion.notion_list_users notion.notion_get_comments notion.notion_append_block +q037,Create a Zendesk ticket,verb_product,zendesk.zendesk_create_ticket,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,368,1.0,1,856,1.0,zendesk.zendesk_create_ticket zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_search zendesk.zendesk_list_tickets jira.jira_create_issue github.create_gist github.create_milestone github.create_hook github.create_environment +q038,Create a HubSpot contact,verb_product,hubspot.hubspot_create_contact,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,904,1.0,hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_create_deal hubspot.hubspot_list_contacts hubspot.hubspot_search_contacts github.create_gist github.create_repository github.create_branch github.create_issue github.create_pull_request +q039,Create a Salesforce lead,verb_product,salesforce.salesforce_convert_lead,0.0,1.0,1.0,0.5000,0.6309,2,1,252,1.0,1,402,1.0,1,787,1.0,salesforce.salesforce_convert_lead salesforce.salesforce_create_lead salesforce.salesforce_create_record salesforce.salesforce_query salesforce.salesforce_update_record salesforce.salesforce_delete_record salesforce.salesforce_list_opportunities salesforce.salesforce_get_record github.create_gist github.create_milestone +q040,Run a SOQL query,identifier,salesforce.salesforce_query,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,291,1.0,1,724,1.0,salesforce.salesforce_query postgres.query sqlite.sqlite_query mysql.mysql_query prometheus.prom_query docker.docker_run postgres.explain_query prometheus.prom_query_range datadog.datadog_query_metrics notion.notion_query_database +q041,List Shopify orders,verb_product,shopify.shopify_list_orders,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,354,1.0,1,682,1.0,shopify.shopify_list_orders shopify.shopify_list_products shopify.shopify_list_customers shopify.shopify_get_product shopify.shopify_create_product shopify.shopify_update_product shopify.shopify_get_order shopify.shopify_create_order clickup.clickup_list_tasks trello.trello_list_cards +q042,Create a PagerDuty incident,verb_product,pagerduty.pagerduty_create_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,403,1.0,1,838,1.0,pagerduty.pagerduty_create_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls github.create_gist github.create_milestone github.create_hook github.create_environment +q043,Who is on call in PagerDuty,paraphrase,pagerduty.pagerduty_list_oncalls,1.0,1.0,1.0,1.0000,1.0000,1,1,227,1.0,1,401,1.0,1,723,1.0,pagerduty.pagerduty_list_oncalls pagerduty.pagerduty_list_incidents pagerduty.pagerduty_get_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident twilio.twilio_make_call memory.search_nodes filesystem.list_allowed_directories trello.trello_add_comment +q044,Query Datadog metrics,verb_product,datadog.datadog_query_metrics,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,458,1.0,1,728,1.0,datadog.datadog_query_metrics datadog.datadog_list_monitors datadog.datadog_create_monitor datadog.datadog_list_logs datadog.datadog_get_monitor prometheus.prom_list_metrics prometheus.prom_query postgres.query postgres.explain_query salesforce.salesforce_query +q045,Search Datadog logs,verb_product,datadog.datadog_list_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,458,1.0,1,863,1.0,datadog.datadog_list_logs datadog.datadog_list_monitors datadog.datadog_get_monitor datadog.datadog_create_monitor datadog.datadog_query_metrics docker.docker_logs kubernetes.k8s_pod_logs github.actions_get_job_logs git.git_log brave_search.brave_web_search +q046,List S3 buckets,verb_product,aws_s3.s3_list_buckets,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,318,1.0,1,667,1.0,aws_s3.s3_list_buckets aws_s3.s3_list_objects clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments kubernetes.k8s_list_services github.projects_list github.list_milestones github.list_hooks github.list_environments +q047,Upload an object to S3,paraphrase,aws_s3.s3_put_object,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,437,1.0,1,950,1.0,aws_s3.s3_put_object aws_s3.s3_delete_object aws_s3.s3_get_object slack.slack_upload_file github.assign_copilot_to_issue slack.slack_reply_to_thread github.add_comment_to_pending_review slack.slack_add_reaction twilio.twilio_send_sms gmail.gmail_send_email +q048,Purge Cloudflare cache,verb_product,cloudflare.cf_purge_cache,1.0,1.0,1.0,1.0000,1.0000,1,1,231,1.0,1,340,1.0,1,746,1.0,cloudflare.cf_purge_cache cloudflare.cf_list_zones cloudflare.cf_create_dns cloudflare.cf_list_dns +q049,Create a Cloudflare DNS record,verb_product,cloudflare.cf_create_dns,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,356,1.0,1,738,1.0,cloudflare.cf_create_dns cloudflare.cf_list_dns cloudflare.cf_list_zones cloudflare.cf_purge_cache salesforce.salesforce_create_record salesforce.salesforce_update_record salesforce.salesforce_delete_record salesforce.salesforce_get_record git.git_commit github.create_gist +q050,List Vercel deployments,verb_product,vercel.vercel_list_deployments,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,291,1.0,1,579,1.0,vercel.vercel_list_deployments vercel.vercel_list_projects vercel.vercel_get_deployment vercel.vercel_create_project kubernetes.k8s_list_deployments clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_services aws_s3.s3_list_buckets github.projects_list +q051,Insert a row into Supabase,verb_product,supabase.supabase_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,355,1.0,1,665,1.0,supabase.supabase_insert supabase.supabase_select supabase.supabase_update supabase.supabase_invoke_fn supabase.supabase_list_tables mongodb.mongo_insert playwright.browser_type +q052,Export images from Figma,paraphrase,figma.figma_get_images,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,368,1.0,1,658,1.0,figma.figma_get_images figma.figma_get_file_nodes figma.figma_get_file figma.figma_get_comments figma.figma_post_comment docker.docker_images elasticsearch.es_get_doc aws_s3.s3_get_object memory.delete_observations docker.docker_logs +q053,Send an SMS with Twilio,verb_product,twilio.twilio_send_sms,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,362,1.0,1,730,1.0,twilio.twilio_send_sms twilio.twilio_list_messages twilio.twilio_make_call filesystem.list_directory_with_sizes puppeteer.puppeteer_click puppeteer.puppeteer_fill gmail.gmail_send_email puppeteer.puppeteer_screenshot jira.jira_search_issues puppeteer.puppeteer_navigate +q054,Create an Okta user,verb_product,okta.okta_get_user,0.0,1.0,1.0,0.3333,0.5000,3,1,183,1.0,1,329,1.0,1,656,1.0,okta.okta_get_user okta.okta_deactivate_user okta.okta_create_user okta.okta_list_users linear.create_user linear.get_user linear.update_user linear.delete_user slack.slack_get_user_profile asana.asana_create_task +q055,Instant query Prometheus,verb_product,prometheus.prom_query,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,329,1.0,1,680,1.0,prometheus.prom_query prometheus.prom_query_range prometheus.prom_list_metrics postgres.query postgres.explain_query salesforce.salesforce_query sqlite.sqlite_query mysql.mysql_query datadog.datadog_query_metrics notion.notion_query_database +q056,Search Grafana dashboards,verb_product,grafana.grafana_search_dashboards,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,382,1.0,1,690,1.0,grafana.grafana_search_dashboards grafana.grafana_get_dashboard grafana.grafana_create_annotation brave_search.brave_web_search brave_search.brave_local_search confluence.confluence_search github.search_issues github.search_users github.search_orgs elasticsearch.es_search +q057,Trigger a CircleCI pipeline,verb_product,circleci.circleci_trigger_pipeline,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,390,1.0,1,755,1.0,circleci.circleci_trigger_pipeline circleci.circleci_get_pipeline circleci.circleci_list_pipelines gitlab.gitlab_create_pipeline github.actions_run_trigger gitlab.create_pipeline gitlab.get_pipeline gitlab.update_pipeline gitlab.delete_pipeline hubspot.hubspot_create_deal +q058,Run terraform plan,identifier,terraform.tf_plan,1.0,1.0,1.0,1.0000,1.0000,1,1,172,1.0,1,320,1.0,1,717,1.0,terraform.tf_plan terraform.tf_apply terraform.tf_state_list postgres.explain_query docker.docker_run github.create_check_run github.get_check_run github.update_check_run github.delete_check_run github.actions_run_trigger +q059,Create an Asana task,verb_product,asana.asana_get_task,0.0,1.0,1.0,0.5000,0.6309,2,1,267,1.0,1,405,1.0,1,825,1.0,asana.asana_get_task asana.asana_create_task asana.asana_update_task asana.asana_create_project asana.asana_list_tasks clickup.clickup_get_task clickup.clickup_create_task clickup.clickup_update_task okta.okta_create_user github.create_issue +q060,Create a Trello card,verb_product,trello.trello_create_card,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,394,1.0,1,884,1.0,trello.trello_create_card trello.trello_add_comment trello.trello_create_board trello.trello_list_cards github.create_gist github.create_milestone github.create_hook github.create_environment github.create_package github.create_artifact +q061,Create a ClickUp task,verb_product,clickup.clickup_get_task,0.0,1.0,1.0,0.5000,0.6309,2,1,257,1.0,1,364,1.0,1,751,1.0,clickup.clickup_get_task clickup.clickup_create_task clickup.clickup_update_task clickup.clickup_list_tasks puppeteer.puppeteer_click playwright.browser_click asana.asana_get_task asana.asana_create_task asana.asana_update_task github.create_gist +q062,Create a Confluence page,verb_product,confluence.confluence_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,401,1.0,1,703,1.0,confluence.confluence_create_page confluence.confluence_get_page confluence.confluence_update_page confluence.confluence_search notion.notion_create_page notion.notion_get_page notion.notion_update_page playwright.browser_take_screenshot notion.notion_get_comments playwright.browser_press_key +q063,Send a Discord message,verb_product,discord.discord_send_message,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,322,1.0,1,686,1.0,discord.discord_send_message discord.discord_add_reaction discord.discord_get_channel discord.discord_list_channels discord.discord_list_members slack.slack_post_message gmail.gmail_get_email gmail.gmail_trash_email slack.slack_add_reaction gmail.gmail_modify_email +q064,Search Elasticsearch,verb_product,elasticsearch.es_search,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,404,1.0,1,699,1.0,elasticsearch.es_search elasticsearch.es_index_doc elasticsearch.es_get_doc brave_search.brave_web_search brave_search.brave_local_search confluence.confluence_search github.search_issues github.search_users github.search_orgs github.search_commits +q065,Get a Redis key,verb_product,redis.redis_get,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,280,1.0,1,589,1.0,redis.redis_get redis.redis_set redis.redis_delete redis.redis_keys playwright.browser_press_key jira.jira_get_issue aws_s3.s3_get_object figma.figma_get_file figma.figma_get_comments figma.figma_get_file_nodes +q066,Find MongoDB documents,verb_product,mongodb.mongo_find,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,381,1.0,1,738,1.0,mongodb.mongo_find mongodb.mongo_insert mongodb.mongo_update mongodb.mongo_delete mongodb.mongo_list_collections +q067,List SQLite tables,verb_product,sqlite.sqlite_list_tables,1.0,1.0,1.0,1.0000,1.0000,1,1,151,1.0,1,240,1.0,1,514,1.0,sqlite.sqlite_list_tables postgres.list_tables mysql.mysql_list_tables supabase.supabase_list_tables sqlite.sqlite_describe_table sqlite.sqlite_query postgres.list_schemas clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments +q068,Describe a postgres table,verb_product,postgres.describe_table,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,272,1.0,1,597,1.0,postgres.describe_table sqlite.sqlite_describe_table postgres.query postgres.list_tables postgres.list_schemas postgres.explain_query supabase.supabase_select supabase.supabase_insert supabase.supabase_update +q069,github_create_issue,identifier,github.create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,386,1.0,1,596,1.0,1,1084,1.0,github.create_issue github.issue_write github.issue_read github.add_issue_comment github.update_issue_comment github.assign_copilot_to_issue bitbucket.bitbucket_create_issue gitlab.gitlab_create_issue linear.linear_create_issue sentry.sentry_get_issue +q070,createIssue,identifier,github.create_issue,1.0,1.0,1.0,1.0000,0.8688,1,1,377,1.0,1,600,1.0,1,1003,1.0,github.create_issue bitbucket.bitbucket_create_issue github.issue_write gitlab.gitlab_create_issue linear.linear_create_issue github.issue_read jira.jira_create_issue sentry.sentry_get_issue jira.jira_assign_issue jira.jira_get_issue +q071,list-directory,identifier,filesystem.create_directory,0.0,1.0,1.0,0.5000,0.6309,2,1,182,1.0,1,296,1.0,1,607,1.0,filesystem.create_directory filesystem.list_directory filesystem.directory_tree filesystem.list_directory_with_sizes filesystem.get_file_info filesystem.list_allowed_directories filesystem.read_file filesystem.read_text_file filesystem.read_media_file filesystem.write_file +q072,sequential thinking next thought,paraphrase,sequential_thinking.sequentialthinking,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,382,1.0,1,797,1.0,sequential_thinking.sequentialthinking +q073,What time is it in Tokyo,paraphrase,time.convert_time,1.0,1.0,1.0,1.0000,0.8597,1,1,217,1.0,1,329,1.0,1,783,1.0,time.convert_time time.get_current_time playwright.browser_wait_for filesystem.list_allowed_directories slack.slack_create_channel google_calendar.gcal_list_events grafana.grafana_create_annotation jira.jira_add_worklog elasticsearch.es_index_doc github.create_milestone +q074,Convert time between timezones,verb_product,time.convert_time,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,335,1.0,1,704,1.0,time.convert_time time.get_current_time salesforce.salesforce_convert_lead playwright.browser_wait_for git.git_diff memory.create_relations google_calendar.gcal_list_events grafana.grafana_create_annotation jira.jira_add_worklog +q075,Star a GitHub repo,verb_product,github.star_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,195,1.0,1,309,1.0,1,604,1.0,github.star_repository git.git_branch git.git_show git.git_checkout git.git_status git.git_diff git.git_commit git.git_add git.git_reset git.git_diff_unstaged +q076,Fork a repository,resource,github.fork_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,235,1.0,1,386,1.0,1,792,1.0,github.fork_repository github.delete_repository github.star_repository github.unstar_repository github.create_repository github.list_repository_collaborators github.get_repository_tree github.list_dependabot_alerts github.list_branches github.list_releases +q077,Get the latest GitHub release,verb_product,github.get_latest_release,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,366,1.0,1,647,1.0,github.get_latest_release github.get_release_by_tag git.git_commit git.git_status git.git_add git.git_checkout git.git_show git.git_branch git.git_diff git.git_reset +q078,Create a gist,resource,github.create_gist,1.0,1.0,1.0,1.0000,1.0000,1,1,229,1.0,1,429,1.0,1,912,1.0,github.create_gist github.update_gist github.get_gist github.create_milestone github.create_hook github.create_environment github.create_package github.create_artifact github.create_runner gitlab.create_pipeline +q079,Search GitHub code,verb_product,github.search_code,1.0,1.0,1.0,1.0000,1.0000,1,1,283,1.0,1,406,1.0,1,697,1.0,github.search_code github.get_code_scanning_alert github.list_code_scanning_alerts git.git_checkout git.git_show git.git_branch git.git_diff git.git_commit git.git_status git.git_add +q080,List GitHub notifications,verb_product,github.list_notifications,1.0,1.0,1.0,1.0000,1.0000,1,1,169,1.0,1,292,1.0,1,585,1.0,github.list_notifications github.mark_all_notifications_read git.git_branch git.git_checkout git.git_show git.git_diff git.git_commit git.git_status git.git_add git.git_reset +q081,Assign Copilot to an issue,verb_product,github.assign_copilot_to_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,434,1.0,1,928,1.0,github.assign_copilot_to_issue linear.linear_assign_issue jira.jira_assign_issue github.request_copilot_review jira.jira_update_issue jira.jira_transition_issue github.create_issue github.add_issue_comment slack.slack_reply_to_thread github.issue_read +q082,Get secret scanning alerts,resource,github.list_secret_scanning_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,295,1.0,1,470,1.0,1,830,1.0,github.list_secret_scanning_alerts github.get_secret_scanning_alert github.list_code_scanning_alerts github.list_dependabot_alerts github.get_code_scanning_alert kubernetes.k8s_get_pod redis.redis_get github.get_discussion github.projects_get github.get_label +q083,Create or update a file on GitHub,verb_product,github.create_or_update_file,1.0,1.0,1.0,1.0000,0.7602,1,1,390,1.0,1,519,1.0,1,869,1.0,github.create_or_update_file github.get_file_contents github.delete_file git.git_diff git.git_add github.get_file_blame git.git_checkout git.git_show git.git_branch git.git_create_branch +q084,List collaborators on a repo,resource,github.list_repository_collaborators,1.0,1.0,1.0,1.0000,1.0000,1,1,295,1.0,1,501,1.0,1,964,1.0,github.list_repository_collaborators github.list_commits github.get_discussion_comments github.update_issue_comment github.request_copilot_review github.create_pull_request trello.trello_list_cards github.assign_copilot_to_issue pagerduty.pagerduty_list_oncalls trello.trello_add_comment +q085,Reply in a Slack thread,verb_product,slack.slack_reply_to_thread,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,358,1.0,1,717,1.0,slack.slack_reply_to_thread slack.slack_get_thread_replies slack.slack_list_channels slack.slack_join_channel slack.slack_leave_channel slack.slack_create_channel slack.slack_search_messages slack.slack_upload_file slack.slack_list_users slack.slack_post_message +q086,Add an emoji reaction on Slack,verb_product,slack.slack_add_reaction,1.0,1.0,1.0,1.0000,0.7602,1,1,216,1.0,1,357,1.0,1,709,1.0,slack.slack_add_reaction slack.slack_join_channel slack.slack_leave_channel slack.slack_create_channel slack.slack_search_messages slack.slack_upload_file slack.slack_list_users slack.slack_post_message slack.slack_get_users slack.slack_list_channels +q087,Search Slack messages,verb_product,slack.slack_search_messages,1.0,1.0,1.0,1.0000,1.0000,1,1,213,1.0,1,337,1.0,1,689,1.0,slack.slack_search_messages slack.slack_get_channel_history slack.slack_join_channel slack.slack_leave_channel slack.slack_create_channel slack.slack_upload_file slack.slack_list_users slack.slack_post_message slack.slack_get_users slack.slack_list_channels +q088,Create a Slack channel,verb_product,slack.slack_create_channel,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,342,1.0,1,751,1.0,slack.slack_create_channel slack.slack_join_channel slack.slack_leave_channel slack.slack_get_channel_history slack.slack_set_channel_topic slack.slack_post_message slack.slack_upload_file slack.slack_add_reaction slack.slack_reply_to_thread slack.slack_get_thread_replies +q089,Transition a Jira issue,verb_product,jira.jira_transition_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,389,1.0,1,763,1.0,jira.jira_transition_issue jira.jira_assign_issue jira.jira_get_issue jira.jira_update_issue jira.jira_create_issue jira.jira_add_comment jira.jira_add_worklog jira.jira_list_projects jira.jira_create_sprint jira.jira_search_issues +q090,Search Jira with JQL,identifier,jira.jira_search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,194,1.0,1,370,1.0,1,764,1.0,jira.jira_search_issues jira.jira_assign_issue jira.jira_list_projects jira.jira_create_sprint jira.jira_update_issue jira.jira_get_issue jira.jira_add_comment jira.jira_add_worklog jira.jira_transition_issue jira.jira_create_issue +q091,Add a worklog to Jira,verb_product,jira.jira_add_worklog,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,350,1.0,1,765,1.0,jira.jira_add_worklog jira.jira_add_comment jira.jira_transition_issue jira.jira_assign_issue jira.jira_list_projects jira.jira_create_sprint jira.jira_update_issue jira.jira_get_issue jira.jira_search_issues jira.jira_create_issue +q092,List Linear teams,verb_product,linear.linear_list_teams,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,380,1.0,1,764,1.0,linear.linear_list_teams linear.linear_list_projects linear.linear_list_issues linear.linear_update_issue linear.linear_create_project linear.linear_create_issue linear.linear_search_issues linear.linear_create_comment linear.linear_assign_issue linear.linear_get_issue +q093,Comment on a Linear issue,verb_product,linear.linear_create_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,334,1.0,1,464,1.0,1,831,1.0,linear.linear_create_comment linear.linear_update_issue linear.linear_create_issue linear.linear_assign_issue linear.linear_get_issue linear.create_comment linear.get_comment linear.update_comment linear.delete_comment github.update_issue_comment +q094,Move a local file,paraphrase,filesystem.move_file,1.0,1.0,1.0,1.0000,1.0000,1,1,214,1.0,1,358,1.0,1,699,1.0,filesystem.move_file brave_search.brave_local_search filesystem.read_file filesystem.read_text_file filesystem.read_media_file figma.figma_get_file filesystem.write_file slack.create_file slack.get_file slack.update_file +q095,Recursively search files by pattern,paraphrase,filesystem.search_files,1.0,1.0,1.0,1.0000,1.0000,1,1,280,1.0,1,433,1.0,1,703,1.0,filesystem.search_files slack.list_files github.push_files github.get_release_by_tag filesystem.move_file filesystem.list_directory filesystem.list_directory_with_sizes filesystem.directory_tree redis.redis_keys google_drive.gdrive_search +q096,Edit a file in place with old and new string,paraphrase,filesystem.edit_file,1.0,1.0,1.0,1.0000,1.0000,1,1,227,1.0,1,394,1.0,1,682,1.0,filesystem.edit_file filesystem.list_directory_with_sizes playwright.browser_tab_new filesystem.write_file slack.create_file slack.get_file slack.update_file slack.delete_file filesystem.move_file figma.figma_get_file +q097,Read the whole knowledge graph,paraphrase,memory.read_graph,1.0,1.0,1.0,1.0000,1.0000,1,1,152,1.0,1,259,1.0,1,529,1.0,memory.read_graph memory.delete_relations memory.create_entities memory.create_relations memory.add_observations memory.delete_entities memory.open_nodes memory.search_nodes memory.delete_observations filesystem.read_file +q098,Search memory nodes,verb_product,memory.search_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,202,1.0,1,301,1.0,1,546,1.0,memory.search_nodes memory.open_nodes figma.figma_get_file_nodes memory.create_entities memory.create_relations memory.add_observations memory.delete_entities memory.delete_observations memory.delete_relations memory.read_graph +q099,Show unstaged git diff,verb_product,git.git_diff_unstaged,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,303,1.0,1,602,1.0,git.git_diff_unstaged git.git_diff git.git_diff_staged git.git_show git.git_status git.git_checkout git.git_branch git.git_log git.git_commit git.git_add +q100,Stage files with git add,identifier,filesystem.list_directory_with_sizes,0.0,1.0,1.0,0.5000,0.6309,2,1,213,1.0,1,336,1.0,1,677,1.0,filesystem.list_directory_with_sizes git.git_add filesystem.search_files git.git_checkout git.git_show git.git_branch github.push_files git.git_diff git.git_commit git.git_status +q101,Create a Shopify product,verb_product,shopify.shopify_create_product,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,380,1.0,1,702,1.0,shopify.shopify_create_product shopify.shopify_get_product shopify.shopify_update_product shopify.shopify_create_order shopify.shopify_list_products shopify.shopify_list_orders shopify.shopify_get_order shopify.shopify_list_customers stripe.stripe_create_product stripe.stripe_create_price +q102,Cancel a Stripe subscription,verb_product,stripe.stripe_cancel_subscription,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,301,1.0,1,632,1.0,stripe.stripe_cancel_subscription stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_list_customers stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_create_product stripe.stripe_create_price stripe.stripe_list_subscriptions stripe.stripe_create_refund +q103,Create a payment intent,resource,stripe.stripe_create_payment_intent,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,417,1.0,1,897,1.0,stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent stripe.stripe_create_refund github.create_gist github.create_milestone github.create_hook github.create_environment github.create_package github.create_artifact github.create_runner +q104,Acknowledge a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,261,1.0,1,405,1.0,1,717,1.0,pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls +q105,Create a Datadog monitor,verb_product,datadog.datadog_get_monitor,0.0,1.0,1.0,0.5000,0.6309,2,1,259,1.0,1,459,1.0,1,927,1.0,datadog.datadog_get_monitor datadog.datadog_create_monitor datadog.datadog_list_monitors datadog.datadog_list_logs datadog.datadog_query_metrics github.create_gist github.create_milestone github.create_hook github.create_environment github.create_package +q106,Resolve a Sentry issue,paraphrase,sentry.sentry_get_issue,0.0,1.0,1.0,0.5000,0.6309,2,1,251,1.0,1,397,1.0,1,817,1.0,sentry.sentry_get_issue sentry.sentry_update_issue sentry.sentry_list_events sentry.sentry_list_projects sentry.sentry_list_issues pagerduty.pagerduty_resolve_incident github.issue_read jira.jira_assign_issue github.issue_write jira.jira_get_issue +q107,Scale a Kubernetes deployment,verb_product,kubernetes.k8s_scale_deployment,1.0,1.0,1.0,1.0000,1.0000,1,1,263,1.0,1,407,1.0,1,726,1.0,kubernetes.k8s_scale_deployment kubernetes.k8s_get_pod kubernetes.k8s_apply kubernetes.k8s_delete kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_list_pods kubernetes.k8s_pod_logs vercel.vercel_get_deployment +q108,Apply a Kubernetes manifest,verb_product,kubernetes.k8s_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,253,1.0,1,407,1.0,1,727,1.0,kubernetes.k8s_apply kubernetes.k8s_get_pod kubernetes.k8s_delete kubernetes.k8s_list_deployments kubernetes.k8s_scale_deployment kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_list_pods kubernetes.k8s_pod_logs terraform.tf_apply +q109,Stop a docker container,verb_product,docker.docker_stop,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,345,1.0,1,641,1.0,docker.docker_stop docker.docker_rm docker.docker_logs docker.docker_run docker.docker_exec docker.docker_images docker.docker_build docker.docker_ps kubernetes.k8s_pod_logs +q110,Build a docker image,verb_product,docker.docker_build,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,317,1.0,1,601,1.0,docker.docker_build docker.docker_run docker.docker_stop docker.docker_rm docker.docker_images docker.docker_ps docker.docker_logs docker.docker_exec filesystem.read_media_file +q111,Click an element in Playwright,verb_product,playwright.browser_click,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,329,1.0,1,566,1.0,playwright.browser_click puppeteer.puppeteer_click playwright.browser_type playwright.browser_take_screenshot playwright.browser_snapshot playwright.browser_navigate playwright.browser_fill_form playwright.browser_evaluate playwright.browser_tab_list playwright.browser_tab_new +q112,Type text into a browser input,paraphrase,playwright.browser_type,1.0,1.0,1.0,1.0000,0.7602,1,1,217,1.0,1,302,1.0,1,554,1.0,playwright.browser_type playwright.browser_wait_for playwright.browser_navigate playwright.browser_tab_close playwright.browser_tab_list playwright.browser_tab_new playwright.browser_snapshot playwright.browser_click playwright.browser_evaluate playwright.browser_fill_form +q113,List GitLab projects,verb_product,gitlab.gitlab_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,183,1.0,1,306,1.0,1,599,1.0,gitlab.gitlab_list_projects git.git_branch github.projects_list git.git_checkout git.git_show git.git_diff git.git_commit git.git_status git.git_add git.git_reset +q114,Create a Bitbucket pull request,verb_product,bitbucket.bitbucket_create_pr,1.0,1.0,1.0,1.0000,1.0000,1,1,313,1.0,1,471,1.0,1,1091,1.0,bitbucket.bitbucket_create_pr bitbucket.bitbucket_list_prs bitbucket.bitbucket_create_issue bitbucket.bitbucket_list_repos github.merge_pull_request github.create_pull_request github.pull_request_read github.update_pull_request github.pull_request_review_write github.update_pull_request_branch +q115,Create a Gmail draft,verb_product,gmail.gmail_create_draft,1.0,1.0,1.0,1.0000,1.0000,1,1,205,1.0,1,352,1.0,1,835,1.0,gmail.gmail_create_draft gmail.gmail_search_emails gmail.gmail_list_labels gmail.gmail_send_email gmail.gmail_get_email gmail.gmail_trash_email gmail.gmail_modify_email github.create_pull_request github.create_gist github.create_milestone +q116,Share a Google Drive file,verb_product,google_drive.gdrive_share_file,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,417,1.0,1,726,1.0,google_drive.gdrive_share_file google_drive.gdrive_update_file google_drive.gdrive_get_file google_drive.gdrive_create_file google_drive.gdrive_search google_drive.gdrive_list_folder gitlab.gitlab_get_file figma.figma_get_file google_calendar.gcal_list_calendars filesystem.write_file +q117,List Google Calendar events,verb_product,google_calendar.gcal_list_events,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,427,1.0,1,784,1.0,google_calendar.gcal_list_events google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event google_calendar.gcal_list_calendars kubernetes.k8s_get_events sentry.sentry_list_events google_drive.gdrive_list_folder google_drive.gdrive_update_file google_drive.gdrive_share_file +q118,Delete a calendar event,resource,google_calendar.gcal_delete_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,427,1.0,1,751,1.0,google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars kubernetes.k8s_delete redis.redis_delete aws_s3.s3_delete_object github.delete_repository github.delete_milestone +q119,List Vercel projects,verb_product,vercel.vercel_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,301,1.0,1,550,1.0,vercel.vercel_list_projects vercel.vercel_list_deployments vercel.vercel_get_deployment vercel.vercel_create_project github.projects_list linear.linear_list_projects jira.jira_list_projects sentry.sentry_list_projects gitlab.gitlab_list_projects github.projects_get +q120,Select rows from Supabase,verb_product,supabase.supabase_select,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,354,1.0,1,709,1.0,supabase.supabase_select supabase.supabase_insert supabase.supabase_update supabase.supabase_invoke_fn supabase.supabase_list_tables elasticsearch.es_get_doc aws_s3.s3_get_object memory.delete_observations docker.docker_logs kubernetes.k8s_pod_logs +q121,Post a Figma comment,verb_product,figma.figma_post_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,368,1.0,1,727,1.0,figma.figma_post_comment figma.figma_get_file figma.figma_get_comments figma.figma_get_images figma.figma_get_file_nodes linear.create_comment linear.get_comment linear.update_comment linear.delete_comment notion.create_comment +q122,List Okta users,verb_product,okta.okta_list_users,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,306,1.0,1,606,1.0,okta.okta_list_users okta.okta_get_user okta.okta_create_user okta.okta_deactivate_user linear.list_users slack.slack_list_users notion.notion_list_users github.search_users notion.list_users_tokens slack.slack_get_users +q123,Range query Prometheus,verb_product,prometheus.prom_query_range,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,329,1.0,1,680,1.0,prometheus.prom_query_range prometheus.prom_query prometheus.prom_list_metrics postgres.query postgres.explain_query salesforce.salesforce_query sqlite.sqlite_query mysql.mysql_query datadog.datadog_query_metrics notion.notion_query_database +q124,Create a Grafana annotation,verb_product,grafana.grafana_create_annotation,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,401,1.0,1,881,1.0,grafana.grafana_create_annotation grafana.grafana_search_dashboards grafana.grafana_get_dashboard github.create_gist github.create_milestone github.create_hook github.create_environment github.create_package github.create_artifact github.create_runner +q125,Apply terraform,identifier,terraform.tf_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,171,1.0,1,292,1.0,1,698,1.0,terraform.tf_apply terraform.tf_plan terraform.tf_state_list kubernetes.k8s_apply +q126,List Asana tasks,verb_product,asana.asana_list_tasks,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,404,1.0,1,682,1.0,asana.asana_list_tasks asana.asana_create_task asana.asana_get_task asana.asana_update_task asana.asana_create_project clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments kubernetes.k8s_list_services aws_s3.s3_list_buckets +q127,Comment on a Trello card,verb_product,trello.trello_add_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,422,1.0,1,757,1.0,trello.trello_add_comment trello.trello_create_card trello.trello_list_cards trello.trello_create_board github.update_issue_comment figma.figma_post_comment linear.create_comment linear.get_comment linear.update_comment linear.delete_comment +q128,Update a ClickUp task,verb_product,clickup.clickup_update_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,364,1.0,1,761,1.0,clickup.clickup_update_task clickup.clickup_get_task clickup.clickup_create_task clickup.clickup_list_tasks puppeteer.puppeteer_click playwright.browser_click asana.asana_update_task asana.asana_get_task asana.asana_create_task mongodb.mongo_update +q129,Search Confluence,verb_product,confluence.confluence_search,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,401,1.0,1,731,1.0,confluence.confluence_search confluence.confluence_get_page confluence.confluence_create_page confluence.confluence_update_page brave_search.brave_web_search brave_search.brave_local_search github.search_issues github.search_users github.search_orgs elasticsearch.es_search +q130,Create a HubSpot deal,verb_product,hubspot.hubspot_create_deal,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,904,1.0,hubspot.hubspot_create_deal hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_list_contacts hubspot.hubspot_search_contacts github.create_gist github.create_repository github.create_branch github.create_issue github.create_pull_request +q131,List Zendesk tickets,verb_product,zendesk.zendesk_list_tickets,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,367,1.0,1,645,1.0,zendesk.zendesk_list_tickets zendesk.zendesk_search zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_create_ticket clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments kubernetes.k8s_list_services aws_s3.s3_list_buckets +q132,List Discord channels,verb_product,discord.discord_list_channels,1.0,1.0,1.0,1.0000,1.0000,1,1,166,1.0,1,321,1.0,1,625,1.0,discord.discord_list_channels discord.discord_list_members discord.discord_get_channel discord.discord_send_message discord.discord_add_reaction slack.slack_list_channels clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments kubernetes.k8s_list_services +q133,Set a Redis key,verb_product,redis.redis_set,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,280,1.0,1,600,1.0,redis.redis_set redis.redis_get redis.redis_delete redis.redis_keys playwright.browser_press_key jira.jira_get_issue slack.slack_set_channel_topic aws_s3.s3_get_object aws_s3.s3_delete_object figma.figma_get_file +q134,Insert MongoDB documents,verb_product,mongodb.mongo_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,256,1.0,1,381,1.0,1,698,1.0,mongodb.mongo_insert mongodb.mongo_update mongodb.mongo_delete mongodb.mongo_find mongodb.mongo_list_collections supabase.supabase_insert +q135,Run a MySQL query,verb_product,postgres.query,0.0,1.0,1.0,0.5000,0.6309,2,1,182,1.0,1,266,1.0,1,543,1.0,postgres.query mysql.mysql_query postgres.explain_query mysql.mysql_list_tables postgres.list_schemas postgres.list_tables postgres.describe_table sqlite.sqlite_query salesforce.salesforce_query prometheus.prom_query +q136,Index a document in Elasticsearch,verb_product,elasticsearch.es_index_doc,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,396,1.0,1,753,1.0,elasticsearch.es_index_doc elasticsearch.es_get_doc elasticsearch.es_search playwright.browser_tab_close github.create_milestone github.get_milestone github.list_milestones github.update_milestone github.delete_milestone github.create_hook +q137,Get an S3 object,verb_product,aws_s3.s3_get_object,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,355,1.0,1,706,1.0,aws_s3.s3_get_object aws_s3.s3_delete_object aws_s3.s3_put_object okta.okta_get_user asana.asana_get_task aws_s3.s3_list_objects elasticsearch.es_search okta.okta_create_user okta.okta_deactivate_user asana.asana_create_task +q138,List Cloudflare zones,verb_product,cloudflare.cf_list_zones,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,338,1.0,1,629,1.0,cloudflare.cf_list_zones cloudflare.cf_list_dns cloudflare.cf_purge_cache cloudflare.cf_create_dns clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments kubernetes.k8s_list_services aws_s3.s3_list_buckets github.projects_list +q139,Create a Stripe invoice,verb_product,stripe.stripe_create_invoice,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,353,1.0,1,669,1.0,stripe.stripe_create_invoice stripe.stripe_create_customer stripe.stripe_create_product stripe.stripe_create_price stripe.stripe_create_refund stripe.stripe_create_payment_intent stripe.stripe_get_customer stripe.stripe_list_customers stripe.stripe_list_invoices stripe.stripe_list_subscriptions +q140,List GitHub Actions workflows,verb_product,github.actions_list,1.0,1.0,1.0,1.0000,1.0000,1,1,293,1.0,1,433,1.0,1,736,1.0,github.actions_list github.actions_get github.actions_run_trigger github.actions_get_job_logs git.git_branch git.git_checkout git.git_show git.git_diff git.git_commit git.git_status +q141,Get GitHub Actions job logs,verb_product,github.actions_get_job_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,450,1.0,1,747,1.0,github.actions_get_job_logs github.actions_get github.actions_list github.actions_run_trigger git.git_log git.git_checkout git.git_show git.git_branch git.git_diff git.git_commit +q142,Create a GitHub repository,verb_product,github.create_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,388,1.0,1,750,1.0,github.create_repository github.delete_repository github.fork_repository github.star_repository github.unstar_repository github.list_repository_collaborators github.get_repository_tree git.git_commit git.git_status git.git_checkout +q143,Checkout a git branch,verb_product,git.git_checkout,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,373,1.0,1,666,1.0,git.git_checkout git.git_create_branch git.git_branch github.create_branch git.git_show git.git_diff git.git_commit git.git_status git.git_add git.git_reset +q144,Show git log,identifier,git.git_log,1.0,1.0,1.0,1.0000,1.0000,1,1,206,1.0,1,320,1.0,1,600,1.0,git.git_log git.git_show git.git_diff git.git_status git.git_checkout git.git_branch git.git_diff_unstaged git.git_diff_staged git.git_commit git.git_add +q145,List postgres schemas,verb_product,postgres.list_schemas,1.0,1.0,1.0,1.0000,1.0000,1,1,158,1.0,1,277,1.0,1,555,1.0,postgres.list_schemas postgres.list_tables postgres.query postgres.describe_table postgres.explain_query clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments kubernetes.k8s_list_services aws_s3.s3_list_buckets +q146,Open nodes in the knowledge graph,verb_product,memory.open_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,163,1.0,1,262,1.0,1,549,1.0,memory.open_nodes memory.search_nodes memory.read_graph memory.create_entities memory.create_relations memory.add_observations memory.delete_relations memory.delete_entities memory.delete_observations figma.figma_get_file_nodes +q147,Get file metadata from the filesystem,paraphrase,filesystem.read_file,0.0,0.0,1.0,0.2500,0.4307,4,2,462,0.0,1,348,1.0,1,688,1.0,filesystem.read_file filesystem.read_text_file filesystem.read_media_file filesystem.get_file_info filesystem.write_file filesystem.edit_file filesystem.move_file filesystem.list_allowed_directories google_drive.gdrive_get_file gitlab.gitlab_get_file +q148,Write a new local file,paraphrase,filesystem.write_file,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,366,1.0,1,714,1.0,filesystem.write_file playwright.browser_tab_new brave_search.brave_local_search filesystem.edit_file filesystem.read_file filesystem.read_text_file filesystem.read_media_file figma.figma_get_file slack.create_file slack.get_file +q149,Search GitHub issues,verb_product,github.search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,270,1.0,1,377,1.0,1,670,1.0,github.search_issues github.list_issues git.git_checkout git.git_show git.git_branch git.git_diff git.git_commit git.git_status git.git_add git.git_reset +q150,Get the authenticated GitHub user,paraphrase,github.get_teams,0.0,0.0,0.0,0.1667,0.3562,6,2,382,0.0,2,615,0.0,1,593,1.0,github.get_teams git.git_commit git.git_status git.git_add github.list_notifications github.get_me git.git_checkout git.git_show git.git_branch git.git_diff diff --git a/experiments/mcp-tool-search/results/lexical_rerank_weighted_top20.json b/experiments/mcp-tool-search/results/lexical_rerank_weighted_top20.json new file mode 100644 index 00000000..432cc9c5 --- /dev/null +++ b/experiments/mcp-tool-search/results/lexical_rerank_weighted_top20.json @@ -0,0 +1,6969 @@ +{ + "summary": { + "n": 150, + "index_s": 0.00289805900013107, + "rss_mb": 0.01171875, + "p50_ms": 0.5752700000130062, + "p95_ms": 0.8243450499321623, + "recall@1": 0.88, + "recall@1_lo": 0.8266666666666667, + "recall@1_hi": 0.9333333333333333, + "recall@3": 0.9733333333333334, + "recall@3_lo": 0.9466666666666667, + "recall@3_hi": 0.9933333333333333, + "recall@5": 0.9866666666666667, + "recall@5_lo": 0.9666666666666667, + "recall@5_hi": 1.0, + "mrr@10": 0.928888888888889, + "mrr@10_lo": 0.8966666666666666, + "mrr@10_hi": 0.96, + "ndcg@10": 0.9395892628155625, + "ndcg@10_lo": 0.9144213810520128, + "ndcg@10_hi": 0.9630889375006223 + }, + "meta": { + "name": "lexical_rerank_weighted_top20", + "needs_model": false, + "needs_api": false, + "notes": "lexical overlap rerank of weighted BM25 top-20; not a cross-encoder" + }, + "rows": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "github.issue_read", + "github.issue_write", + "github.add_issue_comment", + "github.update_issue_comment", + "github.assign_copilot_to_issue", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "git.git_create_branch" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 388, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 598, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 955, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "kind": "verb_product", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_create_issue", + "jira.jira_create_sprint", + "jira.jira_assign_issue", + "jira.jira_list_projects", + "jira.jira_update_issue", + "jira.jira_get_issue", + "jira.jira_search_issues", + "jira.jira_add_comment", + "jira.jira_add_worklog", + "jira.jira_transition_issue" + ], + "hit1": "jira.jira_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 312, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "kind": "verb_product", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "ranking": [ + "slack.slack_post_message", + "slack.slack_add_reaction", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_create_channel", + "slack.slack_search_messages", + "slack.slack_upload_file", + "slack.slack_list_users", + "slack.slack_get_users", + "slack.slack_list_channels" + ], + "hit1": "slack.slack_post_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "kind": "resource", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "ranking": [ + "github.list_pull_requests", + "github.search_pull_requests", + "gitlab.gitlab_list_merge_requests", + "github.merge_pull_request", + "github.pull_request_read", + "github.create_pull_request", + "github.update_pull_request", + "github.update_pull_request_branch", + "github.pull_request_review_write", + "bitbucket.bitbucket_list_prs" + ], + "hit1": "github.list_pull_requests", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1035, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "ranking": [ + "github.create_pull_request", + "github.merge_pull_request", + "github.pull_request_read", + "github.update_pull_request", + "github.pull_request_review_write", + "github.request_copilot_review", + "github.update_pull_request_branch", + "github.search_pull_requests", + "github.list_pull_requests", + "git.git_checkout" + ], + "hit1": "github.create_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 368, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 653, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1052, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "kind": "verb_product", + "gold": [ + "postgres.query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "postgres.explain_query", + "postgres.list_schemas", + "postgres.list_tables", + "postgres.describe_table", + "mysql.mysql_query", + "sqlite.sqlite_query", + "salesforce.salesforce_query", + "mysql.mysql_list_tables", + "prometheus.prom_query" + ], + "hit1": "postgres.query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 167, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 544, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "kind": "paraphrase", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "ranking": [ + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "filesystem.write_file", + "filesystem.edit_file", + "filesystem.get_file_info", + "filesystem.move_file", + "brave_search.brave_local_search", + "github.delete_file", + "gitlab.gitlab_get_file" + ], + "hit1": "filesystem.read_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 782, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "kind": "paraphrase", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "ranking": [ + "filesystem.search_files", + "google_drive.gdrive_list_folder", + "slack.list_files", + "github.push_files", + "filesystem.move_file", + "filesystem.list_directory", + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree", + "filesystem.read_file", + "filesystem.read_text_file" + ], + "hit1": "filesystem.search_files", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.16666666666666666, + "ndcg@10": 0.43476305489842704, + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 473, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 730, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 708, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "ranking": [ + "memory.read_graph", + "memory.create_entities", + "memory.create_relations", + "memory.add_observations", + "memory.open_nodes", + "memory.delete_relations", + "memory.search_nodes", + "memory.delete_entities", + "memory.delete_observations", + "time.get_current_time" + ], + "hit1": "memory.read_graph", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.5672074169568709, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 156, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 524, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "kind": "verb_product", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "ranking": [ + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "filesystem.list_directory_with_sizes", + "jira.jira_search_issues", + "github.update_pull_request_branch", + "memory.search_nodes", + "puppeteer.puppeteer_screenshot", + "puppeteer.puppeteer_navigate", + "puppeteer.puppeteer_click", + "puppeteer.puppeteer_fill" + ], + "hit1": "brave_search.brave_web_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "kind": "verb_product", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "ranking": [ + "fetch.fetch", + "github.get_file_contents", + "playwright.browser_navigate", + "filesystem.read_file", + "puppeteer.puppeteer_navigate", + "git.git_add", + "slack.slack_set_channel_topic", + "github.get_repository_tree", + "git.git_status", + "playwright.browser_snapshot" + ], + "hit1": "fetch.fetch", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 372, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "kind": "identifier", + "gold": [ + "git.git_status" + ], + "alts": [], + "ranking": [ + "git.git_status", + "git.git_show", + "git.git_diff", + "git.git_checkout", + "git.git_branch", + "git.git_log", + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_commit", + "git.git_add" + ], + "hit1": "git.git_status", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 304, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "kind": "paraphrase", + "gold": [ + "git.git_commit" + ], + "alts": [], + "ranking": [ + "git.git_commit", + "github.get_commit", + "git.git_diff_staged", + "git.git_reset", + "git.git_log", + "git.git_show", + "github.push_files" + ], + "hit1": "git.git_commit", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_issue", + "linear.linear_update_issue", + "linear.linear_assign_issue", + "linear.linear_get_issue", + "linear.linear_create_comment", + "linear.linear_create_project", + "linear.linear_list_teams", + "linear.linear_list_projects", + "linear.linear_list_issues", + "linear.linear_search_issues" + ], + "hit1": "linear.linear_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 340, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "kind": "paraphrase", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_project", + "linear.linear_create_issue", + "linear.linear_update_issue", + "linear.linear_list_teams", + "linear.linear_list_projects", + "linear.linear_list_issues", + "linear.linear_search_issues", + "linear.linear_create_comment", + "linear.linear_assign_issue", + "linear.linear_get_issue" + ], + "hit1": "linear.linear_create_project", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 352, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 432, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 769, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "ranking": [ + "github.update_issue_comment", + "github.add_issue_comment", + "github.issue_read", + "github.assign_copilot_to_issue", + "github.issue_write", + "github.create_issue", + "github.add_comment_to_pending_review", + "git.git_checkout", + "git.git_show", + "git.git_branch" + ], + "hit1": "github.update_issue_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 573, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1028, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "kind": "ambiguous", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "ranking": [ + "github.merge_pull_request", + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_get_merge_request", + "github.update_pull_request_branch", + "github.pull_request_read", + "github.create_pull_request", + "github.update_pull_request", + "gitlab.gitlab_create_merge_request", + "github.pull_request_review_write", + "github.request_copilot_review" + ], + "hit1": "github.merge_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 468, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1121, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "kind": "verb_product", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "ranking": [ + "github.actions_run_trigger", + "github.actions_get", + "github.actions_list", + "github.actions_get_job_logs", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "git.git_diff", + "git.git_commit", + "git.git_status" + ], + "hit1": "github.actions_run_trigger", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 450, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 737, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "kind": "resource", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "ranking": [ + "github.list_dependabot_alerts", + "github.get_dependabot_alert", + "github.list_code_scanning_alerts", + "github.list_secret_scanning_alerts", + "kubernetes.k8s_get_pod", + "redis.redis_get", + "github.get_discussion", + "github.projects_get", + "github.get_label", + "aws_s3.s3_get_object" + ], + "hit1": "github.list_dependabot_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 287, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 445, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 798, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "kind": "verb_product", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_issues", + "sentry.sentry_list_projects", + "sentry.sentry_list_events", + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "github.list_issues", + "github.search_issues", + "gitlab.gitlab_list_issues", + "linear.linear_list_issues", + "linear.linear_search_issues" + ], + "hit1": "sentry.sentry_list_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_list_subscriptions", + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "stripe.stripe_create_refund", + "stripe.stripe_list_customers" + ], + "hit1": "stripe.stripe_create_customer", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "kind": "paraphrase", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_refund", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_list_customers", + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_create_product", + "stripe.stripe_create_price" + ], + "hit1": "stripe.stripe_create_refund", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "kind": "verb_product", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "ranking": [ + "gmail.gmail_send_email", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "gmail.gmail_modify_email", + "gmail.gmail_search_emails", + "gmail.gmail_create_draft", + "gmail.gmail_list_labels", + "filesystem.list_directory_with_sizes", + "puppeteer.puppeteer_click", + "puppeteer.puppeteer_fill" + ], + "hit1": "gmail.gmail_send_email", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 685, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "kind": "verb_product", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "ranking": [ + "gmail.gmail_search_emails", + "gmail.gmail_create_draft", + "gmail.gmail_list_labels", + "gmail.gmail_send_email", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "gmail.gmail_modify_email", + "mysql.mysql_list_tables", + "brave_search.brave_web_search", + "brave_search.brave_local_search" + ], + "hit1": "gmail.gmail_search_emails", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "kind": "paraphrase", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_update_file", + "google_drive.gdrive_share_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "slack.slack_upload_file", + "gitlab.gitlab_get_file", + "filesystem.edit_file", + "slack.slack_reply_to_thread" + ], + "hit1": "google_drive.gdrive_update_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 496, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 829, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_create_event", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment", + "github.create_package" + ], + "hit1": "google_calendar.gcal_create_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 895, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_list_pods", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_apply", + "kubernetes.k8s_delete", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_pod_logs", + "clickup.clickup_list_tasks" + ], + "hit1": "kubernetes.k8s_list_pods", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_get_pod", + "docker.docker_logs", + "datadog.datadog_list_logs", + "github.actions_get_job_logs", + "aws_s3.s3_get_object", + "elasticsearch.es_get_doc", + "git.git_log", + "gitlab.gitlab_get_file", + "github.get_tag" + ], + "hit1": "kubernetes.k8s_pod_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 262, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 834, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "ranking": [ + "docker.docker_run", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_exec", + "docker.docker_logs", + "docker.docker_images", + "docker.docker_build", + "docker.docker_ps", + "github.create_check_run", + "github.get_check_run" + ], + "hit1": "docker.docker_run", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 644, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "kind": "verb_product", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "ranking": [ + "docker.docker_logs", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_run", + "docker.docker_exec", + "docker.docker_images", + "docker.docker_build", + "docker.docker_ps", + "kubernetes.k8s_pod_logs", + "datadog.datadog_list_logs" + ], + "hit1": "docker.docker_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "kind": "ambiguous", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "ranking": [ + "playwright.browser_navigate", + "playwright.browser_tab_new", + "puppeteer.puppeteer_navigate", + "playwright.browser_click", + "playwright.browser_evaluate", + "playwright.browser_tab_close", + "playwright.browser_snapshot", + "playwright.browser_tab_list", + "playwright.browser_press_key", + "playwright.browser_wait_for" + ], + "hit1": "playwright.browser_navigate", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 536, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "kind": "ambiguous", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "ranking": [ + "playwright.browser_take_screenshot", + "puppeteer.puppeteer_screenshot", + "notion.notion_get_page", + "notion.notion_update_page", + "confluence.confluence_get_page", + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "notion.notion_create_page", + "playwright.browser_snapshot", + "playwright.browser_press_key" + ], + "hit1": "playwright.browser_take_screenshot", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 711, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_get_merge_request", + "gitlab.gitlab_create_merge_request", + "github.merge_pull_request", + "gitlab.gitlab_list_merge_requests", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "git.git_create_branch", + "git.git_diff" + ], + "hit1": "gitlab.gitlab_merge_merge_request", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 792, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "kind": "verb_product", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "ranking": [ + "notion.notion_search", + "notion.notion_get_page", + "notion.notion_get_block", + "notion.notion_query_database", + "notion.notion_create_database", + "notion.notion_update_page", + "notion.notion_list_users", + "notion.notion_get_comments", + "notion.notion_append_block", + "notion.notion_create_page" + ], + "hit1": "notion.notion_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "kind": "verb_product", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "ranking": [ + "notion.notion_get_page", + "notion.notion_update_page", + "notion.notion_create_page", + "notion.notion_get_comments", + "notion.notion_create_database", + "notion.notion_search", + "notion.notion_get_block", + "notion.notion_query_database", + "notion.notion_list_users", + "notion.notion_append_block" + ], + "hit1": "notion.notion_get_page", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "kind": "verb_product", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "ranking": [ + "notion.notion_query_database", + "notion.notion_create_database", + "notion.notion_create_page", + "notion.notion_search", + "notion.notion_get_page", + "notion.notion_get_block", + "notion.notion_update_page", + "notion.notion_list_users", + "notion.notion_get_comments", + "notion.notion_append_block" + ], + "hit1": "notion.notion_query_database", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 279, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_search", + "zendesk.zendesk_list_tickets", + "jira.jira_create_issue", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment" + ], + "hit1": "zendesk.zendesk_create_ticket", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 856, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_create_deal", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_search_contacts", + "github.create_gist", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.create_pull_request" + ], + "hit1": "hubspot.hubspot_create_contact", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 904, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "kind": "verb_product", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_convert_lead", + "salesforce.salesforce_create_lead", + "salesforce.salesforce_create_record", + "salesforce.salesforce_query", + "salesforce.salesforce_update_record", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_list_opportunities", + "salesforce.salesforce_get_record", + "github.create_gist", + "github.create_milestone" + ], + "hit1": "salesforce.salesforce_convert_lead", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 787, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "kind": "identifier", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_query", + "postgres.query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "prometheus.prom_query", + "docker.docker_run", + "postgres.explain_query", + "prometheus.prom_query_range", + "datadog.datadog_query_metrics", + "notion.notion_query_database" + ], + "hit1": "salesforce.salesforce_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "kind": "verb_product", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "ranking": [ + "shopify.shopify_list_orders", + "shopify.shopify_list_products", + "shopify.shopify_list_customers", + "shopify.shopify_get_product", + "shopify.shopify_create_product", + "shopify.shopify_update_product", + "shopify.shopify_get_order", + "shopify.shopify_create_order", + "clickup.clickup_list_tasks", + "trello.trello_list_cards" + ], + "hit1": "shopify.shopify_list_orders", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment" + ], + "hit1": "pagerduty.pagerduty_create_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 838, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "kind": "paraphrase", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_list_oncalls", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "twilio.twilio_make_call", + "memory.search_nodes", + "filesystem.list_allowed_directories", + "trello.trello_add_comment" + ], + "hit1": "pagerduty.pagerduty_list_oncalls", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 723, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "kind": "verb_product", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "ranking": [ + "datadog.datadog_query_metrics", + "datadog.datadog_list_monitors", + "datadog.datadog_create_monitor", + "datadog.datadog_list_logs", + "datadog.datadog_get_monitor", + "prometheus.prom_list_metrics", + "prometheus.prom_query", + "postgres.query", + "postgres.explain_query", + "salesforce.salesforce_query" + ], + "hit1": "datadog.datadog_query_metrics", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "kind": "verb_product", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "ranking": [ + "datadog.datadog_list_logs", + "datadog.datadog_list_monitors", + "datadog.datadog_get_monitor", + "datadog.datadog_create_monitor", + "datadog.datadog_query_metrics", + "docker.docker_logs", + "kubernetes.k8s_pod_logs", + "github.actions_get_job_logs", + "git.git_log", + "brave_search.brave_web_search" + ], + "hit1": "datadog.datadog_list_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 863, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_list_buckets", + "aws_s3.s3_list_objects", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "github.projects_list", + "github.list_milestones", + "github.list_hooks", + "github.list_environments" + ], + "hit1": "aws_s3.s3_list_buckets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 318, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "kind": "paraphrase", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_put_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_get_object", + "slack.slack_upload_file", + "github.assign_copilot_to_issue", + "slack.slack_reply_to_thread", + "github.add_comment_to_pending_review", + "slack.slack_add_reaction", + "twilio.twilio_send_sms", + "gmail.gmail_send_email" + ], + "hit1": "aws_s3.s3_put_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 950, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns" + ], + "hit1": "cloudflare.cf_purge_cache", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "cloudflare.cf_list_zones", + "cloudflare.cf_purge_cache", + "salesforce.salesforce_create_record", + "salesforce.salesforce_update_record", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_get_record", + "git.git_commit", + "github.create_gist" + ], + "hit1": "cloudflare.cf_create_dns", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_deployments", + "vercel.vercel_list_projects", + "vercel.vercel_get_deployment", + "vercel.vercel_create_project", + "kubernetes.k8s_list_deployments", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_services", + "aws_s3.s3_list_buckets", + "github.projects_list" + ], + "hit1": "vercel.vercel_list_deployments", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 579, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "ranking": [ + "supabase.supabase_insert", + "supabase.supabase_select", + "supabase.supabase_update", + "supabase.supabase_invoke_fn", + "supabase.supabase_list_tables", + "mongodb.mongo_insert", + "playwright.browser_type" + ], + "hit1": "supabase.supabase_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "kind": "paraphrase", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "ranking": [ + "figma.figma_get_images", + "figma.figma_get_file_nodes", + "figma.figma_get_file", + "figma.figma_get_comments", + "figma.figma_post_comment", + "docker.docker_images", + "elasticsearch.es_get_doc", + "aws_s3.s3_get_object", + "memory.delete_observations", + "docker.docker_logs" + ], + "hit1": "figma.figma_get_images", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "kind": "verb_product", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "ranking": [ + "twilio.twilio_send_sms", + "twilio.twilio_list_messages", + "twilio.twilio_make_call", + "filesystem.list_directory_with_sizes", + "puppeteer.puppeteer_click", + "puppeteer.puppeteer_fill", + "gmail.gmail_send_email", + "puppeteer.puppeteer_screenshot", + "jira.jira_search_issues", + "puppeteer.puppeteer_navigate" + ], + "hit1": "twilio.twilio_send_sms", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 730, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "kind": "verb_product", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "ranking": [ + "okta.okta_get_user", + "okta.okta_deactivate_user", + "okta.okta_create_user", + "okta.okta_list_users", + "linear.create_user", + "linear.get_user", + "linear.update_user", + "linear.delete_user", + "slack.slack_get_user_profile", + "asana.asana_create_task" + ], + "hit1": "okta.okta_get_user", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 656, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query", + "prometheus.prom_query_range", + "prometheus.prom_list_metrics", + "postgres.query", + "postgres.explain_query", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "datadog.datadog_query_metrics", + "notion.notion_query_database" + ], + "hit1": "prometheus.prom_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "kind": "verb_product", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "ranking": [ + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "grafana.grafana_create_annotation", + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "confluence.confluence_search", + "github.search_issues", + "github.search_users", + "github.search_orgs", + "elasticsearch.es_search" + ], + "hit1": "grafana.grafana_search_dashboards", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 690, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "kind": "verb_product", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "ranking": [ + "circleci.circleci_trigger_pipeline", + "circleci.circleci_get_pipeline", + "circleci.circleci_list_pipelines", + "gitlab.gitlab_create_pipeline", + "github.actions_run_trigger", + "gitlab.create_pipeline", + "gitlab.get_pipeline", + "gitlab.update_pipeline", + "gitlab.delete_pipeline", + "hubspot.hubspot_create_deal" + ], + "hit1": "circleci.circleci_trigger_pipeline", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 390, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "kind": "identifier", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "ranking": [ + "terraform.tf_plan", + "terraform.tf_apply", + "terraform.tf_state_list", + "postgres.explain_query", + "docker.docker_run", + "github.create_check_run", + "github.get_check_run", + "github.update_check_run", + "github.delete_check_run", + "github.actions_run_trigger" + ], + "hit1": "terraform.tf_plan", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "kind": "verb_product", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "ranking": [ + "asana.asana_get_task", + "asana.asana_create_task", + "asana.asana_update_task", + "asana.asana_create_project", + "asana.asana_list_tasks", + "clickup.clickup_get_task", + "clickup.clickup_create_task", + "clickup.clickup_update_task", + "okta.okta_create_user", + "github.create_issue" + ], + "hit1": "asana.asana_get_task", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "ranking": [ + "trello.trello_create_card", + "trello.trello_add_comment", + "trello.trello_create_board", + "trello.trello_list_cards", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment", + "github.create_package", + "github.create_artifact" + ], + "hit1": "trello.trello_create_card", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 394, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 884, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_get_task", + "clickup.clickup_create_task", + "clickup.clickup_update_task", + "clickup.clickup_list_tasks", + "puppeteer.puppeteer_click", + "playwright.browser_click", + "asana.asana_get_task", + "asana.asana_create_task", + "asana.asana_update_task", + "github.create_gist" + ], + "hit1": "clickup.clickup_get_task", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "kind": "verb_product", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "ranking": [ + "confluence.confluence_create_page", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "confluence.confluence_search", + "notion.notion_create_page", + "notion.notion_get_page", + "notion.notion_update_page", + "playwright.browser_take_screenshot", + "notion.notion_get_comments", + "playwright.browser_press_key" + ], + "hit1": "confluence.confluence_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "kind": "verb_product", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "discord.discord_add_reaction", + "discord.discord_get_channel", + "discord.discord_list_channels", + "discord.discord_list_members", + "slack.slack_post_message", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "slack.slack_add_reaction", + "gmail.gmail_modify_email" + ], + "hit1": "discord.discord_send_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_search", + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "confluence.confluence_search", + "github.search_issues", + "github.search_users", + "github.search_orgs", + "github.search_commits" + ], + "hit1": "elasticsearch.es_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "redis.redis_set", + "redis.redis_delete", + "redis.redis_keys", + "playwright.browser_press_key", + "jira.jira_get_issue", + "aws_s3.s3_get_object", + "figma.figma_get_file", + "figma.figma_get_comments", + "figma.figma_get_file_nodes" + ], + "hit1": "redis.redis_get", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 589, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_find", + "mongodb.mongo_insert", + "mongodb.mongo_update", + "mongodb.mongo_delete", + "mongodb.mongo_list_collections" + ], + "hit1": "mongodb.mongo_find", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "kind": "verb_product", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "ranking": [ + "sqlite.sqlite_list_tables", + "postgres.list_tables", + "mysql.mysql_list_tables", + "supabase.supabase_list_tables", + "sqlite.sqlite_describe_table", + "sqlite.sqlite_query", + "postgres.list_schemas", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments" + ], + "hit1": "sqlite.sqlite_list_tables", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 151, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 240, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 514, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "kind": "verb_product", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "ranking": [ + "postgres.describe_table", + "sqlite.sqlite_describe_table", + "postgres.query", + "postgres.list_tables", + "postgres.list_schemas", + "postgres.explain_query", + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update" + ], + "hit1": "postgres.describe_table", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 272, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "github.issue_write", + "github.issue_read", + "github.add_issue_comment", + "github.update_issue_comment", + "github.assign_copilot_to_issue", + "bitbucket.bitbucket_create_issue", + "gitlab.gitlab_create_issue", + "linear.linear_create_issue", + "sentry.sentry_get_issue" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 386, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 596, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1084, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "ranking": [ + "github.create_issue", + "bitbucket.bitbucket_create_issue", + "github.issue_write", + "gitlab.gitlab_create_issue", + "linear.linear_create_issue", + "github.issue_read", + "jira.jira_create_issue", + "sentry.sentry_get_issue", + "jira.jira_assign_issue", + "jira.jira_get_issue" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8688109777822237, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 377, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 600, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1003, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "kind": "identifier", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "ranking": [ + "filesystem.create_directory", + "filesystem.list_directory", + "filesystem.directory_tree", + "filesystem.list_directory_with_sizes", + "filesystem.get_file_info", + "filesystem.list_allowed_directories", + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "filesystem.write_file" + ], + "hit1": "filesystem.create_directory", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 607, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "kind": "paraphrase", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "ranking": [ + "sequential_thinking.sequentialthinking" + ], + "hit1": "sequential_thinking.sequentialthinking", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "kind": "paraphrase", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "ranking": [ + "time.convert_time", + "time.get_current_time", + "playwright.browser_wait_for", + "filesystem.list_allowed_directories", + "slack.slack_create_channel", + "google_calendar.gcal_list_events", + "grafana.grafana_create_annotation", + "jira.jira_add_worklog", + "elasticsearch.es_index_doc", + "github.create_milestone" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8597186998521972, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 783, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "kind": "verb_product", + "gold": [ + "time.convert_time" + ], + "alts": [], + "ranking": [ + "time.convert_time", + "time.get_current_time", + "salesforce.salesforce_convert_lead", + "playwright.browser_wait_for", + "git.git_diff", + "memory.create_relations", + "google_calendar.gcal_list_events", + "grafana.grafana_create_annotation", + "jira.jira_add_worklog" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "kind": "verb_product", + "gold": [ + "github.star_repository" + ], + "alts": [], + "ranking": [ + "github.star_repository", + "git.git_branch", + "git.git_show", + "git.git_checkout", + "git.git_status", + "git.git_diff", + "git.git_commit", + "git.git_add", + "git.git_reset", + "git.git_diff_unstaged" + ], + "hit1": "github.star_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 309, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 604, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "kind": "resource", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "ranking": [ + "github.fork_repository", + "github.delete_repository", + "github.star_repository", + "github.unstar_repository", + "github.create_repository", + "github.list_repository_collaborators", + "github.get_repository_tree", + "github.list_dependabot_alerts", + "github.list_branches", + "github.list_releases" + ], + "hit1": "github.fork_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 792, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "kind": "verb_product", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "ranking": [ + "github.get_latest_release", + "github.get_release_by_tag", + "git.git_commit", + "git.git_status", + "git.git_add", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "git.git_diff", + "git.git_reset" + ], + "hit1": "github.get_latest_release", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "kind": "resource", + "gold": [ + "github.create_gist" + ], + "alts": [], + "ranking": [ + "github.create_gist", + "github.update_gist", + "github.get_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment", + "github.create_package", + "github.create_artifact", + "github.create_runner", + "gitlab.create_pipeline" + ], + "hit1": "github.create_gist", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 912, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "kind": "verb_product", + "gold": [ + "github.search_code" + ], + "alts": [], + "ranking": [ + "github.search_code", + "github.get_code_scanning_alert", + "github.list_code_scanning_alerts", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "git.git_diff", + "git.git_commit", + "git.git_status", + "git.git_add" + ], + "hit1": "github.search_code", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 406, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 697, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "kind": "verb_product", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "ranking": [ + "github.list_notifications", + "github.mark_all_notifications_read", + "git.git_branch", + "git.git_checkout", + "git.git_show", + "git.git_diff", + "git.git_commit", + "git.git_status", + "git.git_add", + "git.git_reset" + ], + "hit1": "github.list_notifications", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 169, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 585, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "kind": "verb_product", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "ranking": [ + "github.assign_copilot_to_issue", + "linear.linear_assign_issue", + "jira.jira_assign_issue", + "github.request_copilot_review", + "jira.jira_update_issue", + "jira.jira_transition_issue", + "github.create_issue", + "github.add_issue_comment", + "slack.slack_reply_to_thread", + "github.issue_read" + ], + "hit1": "github.assign_copilot_to_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 434, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 928, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "kind": "resource", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "ranking": [ + "github.list_secret_scanning_alerts", + "github.get_secret_scanning_alert", + "github.list_code_scanning_alerts", + "github.list_dependabot_alerts", + "github.get_code_scanning_alert", + "kubernetes.k8s_get_pod", + "redis.redis_get", + "github.get_discussion", + "github.projects_get", + "github.get_label" + ], + "hit1": "github.list_secret_scanning_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 830, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "ranking": [ + "github.create_or_update_file", + "github.get_file_contents", + "github.delete_file", + "git.git_diff", + "git.git_add", + "github.get_file_blame", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "git.git_create_branch" + ], + "hit1": "github.create_or_update_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 390, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 519, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 869, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "kind": "resource", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "ranking": [ + "github.list_repository_collaborators", + "github.list_commits", + "github.get_discussion_comments", + "github.update_issue_comment", + "github.request_copilot_review", + "github.create_pull_request", + "trello.trello_list_cards", + "github.assign_copilot_to_issue", + "pagerduty.pagerduty_list_oncalls", + "trello.trello_add_comment" + ], + "hit1": "github.list_repository_collaborators", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 501, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 964, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "kind": "verb_product", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "ranking": [ + "slack.slack_reply_to_thread", + "slack.slack_get_thread_replies", + "slack.slack_list_channels", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_create_channel", + "slack.slack_search_messages", + "slack.slack_upload_file", + "slack.slack_list_users", + "slack.slack_post_message" + ], + "hit1": "slack.slack_reply_to_thread", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "kind": "verb_product", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "ranking": [ + "slack.slack_add_reaction", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_create_channel", + "slack.slack_search_messages", + "slack.slack_upload_file", + "slack.slack_list_users", + "slack.slack_post_message", + "slack.slack_get_users", + "slack.slack_list_channels" + ], + "hit1": "slack.slack_add_reaction", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 357, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "kind": "verb_product", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "ranking": [ + "slack.slack_search_messages", + "slack.slack_get_channel_history", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_create_channel", + "slack.slack_upload_file", + "slack.slack_list_users", + "slack.slack_post_message", + "slack.slack_get_users", + "slack.slack_list_channels" + ], + "hit1": "slack.slack_search_messages", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 689, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "kind": "verb_product", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "ranking": [ + "slack.slack_create_channel", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_get_channel_history", + "slack.slack_set_channel_topic", + "slack.slack_post_message", + "slack.slack_upload_file", + "slack.slack_add_reaction", + "slack.slack_reply_to_thread", + "slack.slack_get_thread_replies" + ], + "hit1": "slack.slack_create_channel", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "kind": "verb_product", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_transition_issue", + "jira.jira_assign_issue", + "jira.jira_get_issue", + "jira.jira_update_issue", + "jira.jira_create_issue", + "jira.jira_add_comment", + "jira.jira_add_worklog", + "jira.jira_list_projects", + "jira.jira_create_sprint", + "jira.jira_search_issues" + ], + "hit1": "jira.jira_transition_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "kind": "identifier", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "ranking": [ + "jira.jira_search_issues", + "jira.jira_assign_issue", + "jira.jira_list_projects", + "jira.jira_create_sprint", + "jira.jira_update_issue", + "jira.jira_get_issue", + "jira.jira_add_comment", + "jira.jira_add_worklog", + "jira.jira_transition_issue", + "jira.jira_create_issue" + ], + "hit1": "jira.jira_search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "kind": "verb_product", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "ranking": [ + "jira.jira_add_worklog", + "jira.jira_add_comment", + "jira.jira_transition_issue", + "jira.jira_assign_issue", + "jira.jira_list_projects", + "jira.jira_create_sprint", + "jira.jira_update_issue", + "jira.jira_get_issue", + "jira.jira_search_issues", + "jira.jira_create_issue" + ], + "hit1": "jira.jira_add_worklog", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "kind": "verb_product", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "ranking": [ + "linear.linear_list_teams", + "linear.linear_list_projects", + "linear.linear_list_issues", + "linear.linear_update_issue", + "linear.linear_create_project", + "linear.linear_create_issue", + "linear.linear_search_issues", + "linear.linear_create_comment", + "linear.linear_assign_issue", + "linear.linear_get_issue" + ], + "hit1": "linear.linear_list_teams", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "ranking": [ + "linear.linear_create_comment", + "linear.linear_update_issue", + "linear.linear_create_issue", + "linear.linear_assign_issue", + "linear.linear_get_issue", + "linear.create_comment", + "linear.get_comment", + "linear.update_comment", + "linear.delete_comment", + "github.update_issue_comment" + ], + "hit1": "linear.linear_create_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 334, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 464, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 831, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "kind": "paraphrase", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "ranking": [ + "filesystem.move_file", + "brave_search.brave_local_search", + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "figma.figma_get_file", + "filesystem.write_file", + "slack.create_file", + "slack.get_file", + "slack.update_file" + ], + "hit1": "filesystem.move_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "kind": "paraphrase", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "ranking": [ + "filesystem.search_files", + "slack.list_files", + "github.push_files", + "github.get_release_by_tag", + "filesystem.move_file", + "filesystem.list_directory", + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree", + "redis.redis_keys", + "google_drive.gdrive_search" + ], + "hit1": "filesystem.search_files", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 280, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 433, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "kind": "paraphrase", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "ranking": [ + "filesystem.edit_file", + "filesystem.list_directory_with_sizes", + "playwright.browser_tab_new", + "filesystem.write_file", + "slack.create_file", + "slack.get_file", + "slack.update_file", + "slack.delete_file", + "filesystem.move_file", + "figma.figma_get_file" + ], + "hit1": "filesystem.edit_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 394, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "ranking": [ + "memory.read_graph", + "memory.delete_relations", + "memory.create_entities", + "memory.create_relations", + "memory.add_observations", + "memory.delete_entities", + "memory.open_nodes", + "memory.search_nodes", + "memory.delete_observations", + "filesystem.read_file" + ], + "hit1": "memory.read_graph", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 259, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 529, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "kind": "verb_product", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "memory.open_nodes", + "figma.figma_get_file_nodes", + "memory.create_entities", + "memory.create_relations", + "memory.add_observations", + "memory.delete_entities", + "memory.delete_observations", + "memory.delete_relations", + "memory.read_graph" + ], + "hit1": "memory.search_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 546, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "kind": "verb_product", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "ranking": [ + "git.git_diff_unstaged", + "git.git_diff", + "git.git_diff_staged", + "git.git_show", + "git.git_status", + "git.git_checkout", + "git.git_branch", + "git.git_log", + "git.git_commit", + "git.git_add" + ], + "hit1": "git.git_diff_unstaged", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 602, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "kind": "identifier", + "gold": [ + "git.git_add" + ], + "alts": [], + "ranking": [ + "filesystem.list_directory_with_sizes", + "git.git_add", + "filesystem.search_files", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "github.push_files", + "git.git_diff", + "git.git_commit", + "git.git_status" + ], + "hit1": "filesystem.list_directory_with_sizes", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 336, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 677, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "kind": "verb_product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "ranking": [ + "shopify.shopify_create_product", + "shopify.shopify_get_product", + "shopify.shopify_update_product", + "shopify.shopify_create_order", + "shopify.shopify_list_products", + "shopify.shopify_list_orders", + "shopify.shopify_get_order", + "shopify.shopify_list_customers", + "stripe.stripe_create_product", + "stripe.stripe_create_price" + ], + "hit1": "shopify.shopify_create_product", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "kind": "verb_product", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "ranking": [ + "stripe.stripe_cancel_subscription", + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_list_customers", + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "stripe.stripe_list_subscriptions", + "stripe.stripe_create_refund" + ], + "hit1": "stripe.stripe_cancel_subscription", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "kind": "resource", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_refund", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment", + "github.create_package", + "github.create_artifact", + "github.create_runner" + ], + "hit1": "stripe.stripe_create_payment_intent", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 897, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "kind": "verb_product", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "ranking": [ + "datadog.datadog_get_monitor", + "datadog.datadog_create_monitor", + "datadog.datadog_list_monitors", + "datadog.datadog_list_logs", + "datadog.datadog_query_metrics", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment", + "github.create_package" + ], + "hit1": "datadog.datadog_get_monitor", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 927, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "kind": "paraphrase", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "ranking": [ + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "sentry.sentry_list_events", + "sentry.sentry_list_projects", + "sentry.sentry_list_issues", + "pagerduty.pagerduty_resolve_incident", + "github.issue_read", + "jira.jira_assign_issue", + "github.issue_write", + "jira.jira_get_issue" + ], + "hit1": "sentry.sentry_get_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 817, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_apply", + "kubernetes.k8s_delete", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_pod_logs", + "vercel.vercel_get_deployment" + ], + "hit1": "kubernetes.k8s_scale_deployment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 263, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_apply", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_delete", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_pod_logs", + "terraform.tf_apply" + ], + "hit1": "kubernetes.k8s_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 253, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "ranking": [ + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_run", + "docker.docker_exec", + "docker.docker_images", + "docker.docker_build", + "docker.docker_ps", + "kubernetes.k8s_pod_logs" + ], + "hit1": "docker.docker_stop", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "kind": "verb_product", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "ranking": [ + "docker.docker_build", + "docker.docker_run", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_images", + "docker.docker_ps", + "docker.docker_logs", + "docker.docker_exec", + "filesystem.read_media_file" + ], + "hit1": "docker.docker_build", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "kind": "verb_product", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "ranking": [ + "playwright.browser_click", + "puppeteer.puppeteer_click", + "playwright.browser_type", + "playwright.browser_take_screenshot", + "playwright.browser_snapshot", + "playwright.browser_navigate", + "playwright.browser_fill_form", + "playwright.browser_evaluate", + "playwright.browser_tab_list", + "playwright.browser_tab_new" + ], + "hit1": "playwright.browser_click", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 566, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "kind": "paraphrase", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "ranking": [ + "playwright.browser_type", + "playwright.browser_wait_for", + "playwright.browser_navigate", + "playwright.browser_tab_close", + "playwright.browser_tab_list", + "playwright.browser_tab_new", + "playwright.browser_snapshot", + "playwright.browser_click", + "playwright.browser_evaluate", + "playwright.browser_fill_form" + ], + "hit1": "playwright.browser_type", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 554, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_list_projects", + "git.git_branch", + "github.projects_list", + "git.git_checkout", + "git.git_show", + "git.git_diff", + "git.git_commit", + "git.git_status", + "git.git_add", + "git.git_reset" + ], + "hit1": "gitlab.gitlab_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 306, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 599, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "kind": "verb_product", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "ranking": [ + "bitbucket.bitbucket_create_pr", + "bitbucket.bitbucket_list_prs", + "bitbucket.bitbucket_create_issue", + "bitbucket.bitbucket_list_repos", + "github.merge_pull_request", + "github.create_pull_request", + "github.pull_request_read", + "github.update_pull_request", + "github.pull_request_review_write", + "github.update_pull_request_branch" + ], + "hit1": "bitbucket.bitbucket_create_pr", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 471, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1091, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "kind": "verb_product", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "ranking": [ + "gmail.gmail_create_draft", + "gmail.gmail_search_emails", + "gmail.gmail_list_labels", + "gmail.gmail_send_email", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "gmail.gmail_modify_email", + "github.create_pull_request", + "github.create_gist", + "github.create_milestone" + ], + "hit1": "gmail.gmail_create_draft", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 835, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "kind": "verb_product", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_share_file", + "google_drive.gdrive_update_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "gitlab.gitlab_get_file", + "figma.figma_get_file", + "google_calendar.gcal_list_calendars", + "filesystem.write_file" + ], + "hit1": "google_drive.gdrive_share_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "kind": "verb_product", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_list_events", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_list_calendars", + "kubernetes.k8s_get_events", + "sentry.sentry_list_events", + "google_drive.gdrive_list_folder", + "google_drive.gdrive_update_file", + "google_drive.gdrive_share_file" + ], + "hit1": "google_calendar.gcal_list_events", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 784, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "kubernetes.k8s_delete", + "redis.redis_delete", + "aws_s3.s3_delete_object", + "github.delete_repository", + "github.delete_milestone" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_projects", + "vercel.vercel_list_deployments", + "vercel.vercel_get_deployment", + "vercel.vercel_create_project", + "github.projects_list", + "linear.linear_list_projects", + "jira.jira_list_projects", + "sentry.sentry_list_projects", + "gitlab.gitlab_list_projects", + "github.projects_get" + ], + "hit1": "vercel.vercel_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "ranking": [ + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update", + "supabase.supabase_invoke_fn", + "supabase.supabase_list_tables", + "elasticsearch.es_get_doc", + "aws_s3.s3_get_object", + "memory.delete_observations", + "docker.docker_logs", + "kubernetes.k8s_pod_logs" + ], + "hit1": "supabase.supabase_select", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "kind": "verb_product", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "figma.figma_get_file", + "figma.figma_get_comments", + "figma.figma_get_images", + "figma.figma_get_file_nodes", + "linear.create_comment", + "linear.get_comment", + "linear.update_comment", + "linear.delete_comment", + "notion.create_comment" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "kind": "verb_product", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "ranking": [ + "okta.okta_list_users", + "okta.okta_get_user", + "okta.okta_create_user", + "okta.okta_deactivate_user", + "linear.list_users", + "slack.slack_list_users", + "notion.notion_list_users", + "github.search_users", + "notion.list_users_tokens", + "slack.slack_get_users" + ], + "hit1": "okta.okta_list_users", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 306, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query_range", + "prometheus.prom_query", + "prometheus.prom_list_metrics", + "postgres.query", + "postgres.explain_query", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "datadog.datadog_query_metrics", + "notion.notion_query_database" + ], + "hit1": "prometheus.prom_query_range", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "kind": "verb_product", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "ranking": [ + "grafana.grafana_create_annotation", + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment", + "github.create_package", + "github.create_artifact", + "github.create_runner" + ], + "hit1": "grafana.grafana_create_annotation", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 881, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "kind": "identifier", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "ranking": [ + "terraform.tf_apply", + "terraform.tf_plan", + "terraform.tf_state_list", + "kubernetes.k8s_apply" + ], + "hit1": "terraform.tf_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "kind": "verb_product", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "ranking": [ + "asana.asana_list_tasks", + "asana.asana_create_task", + "asana.asana_get_task", + "asana.asana_update_task", + "asana.asana_create_project", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "aws_s3.s3_list_buckets" + ], + "hit1": "asana.asana_list_tasks", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "ranking": [ + "trello.trello_add_comment", + "trello.trello_create_card", + "trello.trello_list_cards", + "trello.trello_create_board", + "github.update_issue_comment", + "figma.figma_post_comment", + "linear.create_comment", + "linear.get_comment", + "linear.update_comment", + "linear.delete_comment" + ], + "hit1": "trello.trello_add_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 422, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_update_task", + "clickup.clickup_get_task", + "clickup.clickup_create_task", + "clickup.clickup_list_tasks", + "puppeteer.puppeteer_click", + "playwright.browser_click", + "asana.asana_update_task", + "asana.asana_get_task", + "asana.asana_create_task", + "mongodb.mongo_update" + ], + "hit1": "clickup.clickup_update_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 761, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "kind": "verb_product", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "ranking": [ + "confluence.confluence_search", + "confluence.confluence_get_page", + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "github.search_issues", + "github.search_users", + "github.search_orgs", + "elasticsearch.es_search" + ], + "hit1": "confluence.confluence_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 731, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_deal", + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_search_contacts", + "github.create_gist", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.create_pull_request" + ], + "hit1": "hubspot.hubspot_create_deal", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 904, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_search", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_create_ticket", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "aws_s3.s3_list_buckets" + ], + "hit1": "zendesk.zendesk_list_tickets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "kind": "verb_product", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "ranking": [ + "discord.discord_list_channels", + "discord.discord_list_members", + "discord.discord_get_channel", + "discord.discord_send_message", + "discord.discord_add_reaction", + "slack.slack_list_channels", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services" + ], + "hit1": "discord.discord_list_channels", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 166, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 625, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "ranking": [ + "redis.redis_set", + "redis.redis_get", + "redis.redis_delete", + "redis.redis_keys", + "playwright.browser_press_key", + "jira.jira_get_issue", + "slack.slack_set_channel_topic", + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "figma.figma_get_file" + ], + "hit1": "redis.redis_set", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert", + "mongodb.mongo_update", + "mongodb.mongo_delete", + "mongodb.mongo_find", + "mongodb.mongo_list_collections", + "supabase.supabase_insert" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "kind": "verb_product", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "mysql.mysql_query", + "postgres.explain_query", + "mysql.mysql_list_tables", + "postgres.list_schemas", + "postgres.list_tables", + "postgres.describe_table", + "sqlite.sqlite_query", + "salesforce.salesforce_query", + "prometheus.prom_query" + ], + "hit1": "postgres.query", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 266, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 543, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "elasticsearch.es_search", + "playwright.browser_tab_close", + "github.create_milestone", + "github.get_milestone", + "github.list_milestones", + "github.update_milestone", + "github.delete_milestone", + "github.create_hook" + ], + "hit1": "elasticsearch.es_index_doc", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_put_object", + "okta.okta_get_user", + "asana.asana_get_task", + "aws_s3.s3_list_objects", + "elasticsearch.es_search", + "okta.okta_create_user", + "okta.okta_deactivate_user", + "asana.asana_create_task" + ], + "hit1": "aws_s3.s3_get_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones", + "cloudflare.cf_list_dns", + "cloudflare.cf_purge_cache", + "cloudflare.cf_create_dns", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "aws_s3.s3_list_buckets", + "github.projects_list" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 629, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_create_customer", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "stripe.stripe_create_refund", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_customer", + "stripe.stripe_list_customers", + "stripe.stripe_list_invoices", + "stripe.stripe_list_subscriptions" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "kind": "verb_product", + "gold": [ + "github.actions_list" + ], + "alts": [], + "ranking": [ + "github.actions_list", + "github.actions_get", + "github.actions_run_trigger", + "github.actions_get_job_logs", + "git.git_branch", + "git.git_checkout", + "git.git_show", + "git.git_diff", + "git.git_commit", + "git.git_status" + ], + "hit1": "github.actions_list", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 433, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 736, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "kind": "verb_product", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "ranking": [ + "github.actions_get_job_logs", + "github.actions_get", + "github.actions_list", + "github.actions_run_trigger", + "git.git_log", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "git.git_diff", + "git.git_commit" + ], + "hit1": "github.actions_get_job_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 450, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 747, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "kind": "verb_product", + "gold": [ + "github.create_repository" + ], + "alts": [], + "ranking": [ + "github.create_repository", + "github.delete_repository", + "github.fork_repository", + "github.star_repository", + "github.unstar_repository", + "github.list_repository_collaborators", + "github.get_repository_tree", + "git.git_commit", + "git.git_status", + "git.git_checkout" + ], + "hit1": "github.create_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 750, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "kind": "verb_product", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "ranking": [ + "git.git_checkout", + "git.git_create_branch", + "git.git_branch", + "github.create_branch", + "git.git_show", + "git.git_diff", + "git.git_commit", + "git.git_status", + "git.git_add", + "git.git_reset" + ], + "hit1": "git.git_checkout", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 666, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "kind": "identifier", + "gold": [ + "git.git_log" + ], + "alts": [], + "ranking": [ + "git.git_log", + "git.git_show", + "git.git_diff", + "git.git_status", + "git.git_checkout", + "git.git_branch", + "git.git_diff_unstaged", + "git.git_diff_staged", + "git.git_commit", + "git.git_add" + ], + "hit1": "git.git_log", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "kind": "verb_product", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "ranking": [ + "postgres.list_schemas", + "postgres.list_tables", + "postgres.query", + "postgres.describe_table", + "postgres.explain_query", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "aws_s3.s3_list_buckets" + ], + "hit1": "postgres.list_schemas", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 158, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 555, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "kind": "verb_product", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "ranking": [ + "memory.open_nodes", + "memory.search_nodes", + "memory.read_graph", + "memory.create_entities", + "memory.create_relations", + "memory.add_observations", + "memory.delete_relations", + "memory.delete_entities", + "memory.delete_observations", + "figma.figma_get_file_nodes" + ], + "hit1": "memory.open_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 549, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "kind": "paraphrase", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "ranking": [ + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "filesystem.get_file_info", + "filesystem.write_file", + "filesystem.edit_file", + "filesystem.move_file", + "filesystem.list_allowed_directories", + "google_drive.gdrive_get_file", + "gitlab.gitlab_get_file" + ], + "hit1": "filesystem.read_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 462, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 688, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "kind": "paraphrase", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "ranking": [ + "filesystem.write_file", + "playwright.browser_tab_new", + "brave_search.brave_local_search", + "filesystem.edit_file", + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "figma.figma_get_file", + "slack.create_file", + "slack.get_file" + ], + "hit1": "filesystem.write_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "kind": "verb_product", + "gold": [ + "github.search_issues" + ], + "alts": [], + "ranking": [ + "github.search_issues", + "github.list_issues", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "git.git_diff", + "git.git_commit", + "git.git_status", + "git.git_add", + "git.git_reset" + ], + "hit1": "github.search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 270, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 670, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "kind": "paraphrase", + "gold": [ + "github.get_me" + ], + "alts": [], + "ranking": [ + "github.get_teams", + "git.git_commit", + "git.git_status", + "git.git_add", + "github.list_notifications", + "github.get_me", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "git.git_diff" + ], + "hit1": "github.get_teams", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.16666666666666666, + "ndcg@10": 0.3562071871080222, + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 382, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 615, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 593, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "cost": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.0266666666666666, + "p95_calls": 1.0, + "success_within_1": 0.9733333333333334, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 240.99333333333334, + "p95_tokens": 372.9499999999998, + "total_tokens": 36149, + "tokens_per_successful_query": 240.99333333333334, + "total_tokens_successful": 36149 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.0133333333333334, + "p95_calls": 1.0, + "success_within_1": 0.9866666666666667, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 380.41333333333336, + "p95_tokens": 510.89999999999975, + "total_tokens": 57062, + "tokens_per_successful_query": 380.41333333333336, + "total_tokens_successful": 57062 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.0, + "p95_calls": 1.0, + "success_within_1": 1.0, + "success_within_2": 1.0, + "success_within_3": 1.0, + "mean_tokens": 730.9933333333333, + "p95_tokens": 959.9499999999998, + "total_tokens": 109649, + "tokens_per_successful_query": 730.9933333333333, + "total_tokens_successful": 109649 + } + } +} diff --git a/experiments/mcp-tool-search/results/live_agent.json b/experiments/mcp-tool-search/results/live_agent.json new file mode 100644 index 00000000..d1f26a3e --- /dev/null +++ b/experiments/mcp-tool-search/results/live_agent.json @@ -0,0 +1,17 @@ +{ + "skipped": true, + "reason": "No LLM API key in the environment (checked OPENAI_API_KEY, ANTHROPIC_API_KEY, AZURE_OPENAI_API_KEY, GOOGLE_API_KEY, GEMINI_API_KEY, COHERE_API_KEY, TOGETHER_API_KEY, GROQ_API_KEY, MISTRAL_API_KEY, OPENROUTER_API_KEY). Offline cost model still ran. To execute the live loop: OPENAI_API_KEY=... python live_agent.py", + "checked_keys": [ + "OPENAI_API_KEY", + "ANTHROPIC_API_KEY", + "AZURE_OPENAI_API_KEY", + "GOOGLE_API_KEY", + "GEMINI_API_KEY", + "COHERE_API_KEY", + "TOGETHER_API_KEY", + "GROQ_API_KEY", + "MISTRAL_API_KEY", + "OPENROUTER_API_KEY" + ], + "command": "OPENAI_API_KEY=... python live_agent.py" +} diff --git a/experiments/mcp-tool-search/results/name_exact_substr.csv b/experiments/mcp-tool-search/results/name_exact_substr.csv new file mode 100644 index 00000000..dddb0347 --- /dev/null +++ b/experiments/mcp-tool-search/results/name_exact_substr.csv @@ -0,0 +1,151 @@ +id,query,kind,hit1,recall@1,recall@3,recall@5,mrr@10,ndcg@10,hit_rank,k3_calls,k3_tokens,k3_success1,k5_calls,k5_tokens,k5_success1,k10_calls,k10_tokens,k10_success1,ranking +q001,Create a GitHub issue,verb_product,,0.0,0.0,0.0,0.0000,0.0000,29,10,2782,0.0,6,2698,0.0,3,2635,0.0, +q002,Create a Jira ticket,verb_product,,0.0,0.0,0.0,0.0000,0.0000,146,49,12183,0.0,30,11995,0.0,15,11695,0.0, +q003,Send a Slack message,verb_product,,0.0,0.0,0.0,0.0000,0.0000,121,41,10292,0.0,25,10122,0.0,13,10175,0.0, +q004,List open pull requests,resource,,0.0,0.0,0.0,0.0000,0.0000,37,13,3710,0.0,8,3752,0.0,4,3672,0.0, +q005,Open a pull request on GitHub,verb_product,,0.0,0.0,0.0,0.0000,0.0000,36,12,3495,0.0,8,3776,0.0,4,3684,0.0, +q006,Run a SQL query against postgres,verb_product,,0.0,0.0,0.0,0.0000,0.0000,98,33,8598,0.0,20,8376,0.0,10,8156,0.0, +q007,Read a local file from disk,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,76,26,7282,0.0,16,7224,0.0,8,7048,0.0, +q008,What files are in this folder,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,82,28,7711,0.0,17,7530,0.0,9,7616,0.0, +q009,Remember this fact in the knowledge graph,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,89,30,8108,0.0,18,7832,0.0,9,7625,0.0, +q010,Search the web with Brave,verb_product,,0.0,0.0,0.0,0.0000,0.0000,119,40,10063,0.0,24,9727,0.0,12,9475,0.0, +q011,Fetch the contents of a URL,verb_product,,0.0,0.0,0.0,0.0000,0.0000,103,35,8982,0.0,21,8674,0.0,11,8746,0.0, +q012,Show git status,identifier,,0.0,0.0,0.0,0.0000,0.0000,104,35,8877,0.0,21,8611,0.0,11,8713,0.0, +q013,Commit staged changes,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,108,36,9075,0.0,22,8922,0.0,11,8713,0.0, +q014,Create a Linear issue,verb_product,,0.0,0.0,0.0,0.0000,0.0000,136,46,11528,0.0,28,11279,0.0,14,10999,0.0, +q015,Make a task in my project tracker Linear,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,136,46,11712,0.0,28,11391,0.0,14,11055,0.0, +q016,Post a comment on a GitHub issue,verb_product,,0.0,0.0,0.0,0.0000,0.0000,34,12,3507,0.0,7,3237,0.0,4,3688,0.0, +q017,Merge the pull request,ambiguous,,0.0,0.0,0.0,0.0000,0.0000,41,14,4072,0.0,9,4373,0.0,5,4766,0.0, +q018,Trigger a GitHub Actions workflow,verb_product,,0.0,0.0,0.0,0.0000,0.0000,62,21,6123,0.0,13,6050,0.0,7,6293,0.0, +q019,Get Dependabot alerts,resource,,0.0,0.0,0.0,0.0000,0.0000,51,17,5097,0.0,11,5225,0.0,6,5482,0.0, +q020,List Sentry issues,verb_product,,0.0,0.0,0.0,0.0000,0.0000,228,76,18006,0.0,46,17548,0.0,23,17088,0.0, +q021,Create a Stripe customer,verb_product,,0.0,0.0,0.0,0.0000,0.0000,200,67,15974,0.0,40,15384,0.0,20,14984,0.0, +q022,Refund a Stripe payment,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,211,71,16820,0.0,43,16424,0.0,22,16307,0.0, +q023,Send an email with Gmail,verb_product,,0.0,0.0,0.0,0.0000,0.0000,244,82,19588,0.0,49,18841,0.0,25,18655,0.0, +q024,Search my Gmail inbox,verb_product,,0.0,0.0,0.0,0.0000,0.0000,245,82,19506,0.0,49,18792,0.0,25,18630,0.0, +q025,Upload a file to Google Drive,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,253,85,20366,0.0,51,19618,0.0,26,19523,0.0, +q026,Create a calendar event,resource,,0.0,0.0,0.0,0.0000,0.0000,258,86,20511,0.0,52,19991,0.0,26,19471,0.0, +q027,List Kubernetes pods,verb_product,,0.0,0.0,0.0,0.0000,0.0000,191,64,15221,0.0,39,14999,0.0,20,14964,0.0, +q028,Get logs from a k8s pod,verb_product,,0.0,0.0,0.0,0.0000,0.0000,193,65,15818,0.0,39,15194,0.0,20,15064,0.0, +q029,Run a docker container,verb_product,,0.0,0.0,0.0,0.0000,0.0000,185,62,14888,0.0,37,14342,0.0,19,14250,0.0, +q030,Show docker container logs,verb_product,,0.0,0.0,0.0,0.0000,0.0000,184,62,14888,0.0,37,14342,0.0,19,14250,0.0, +q031,Navigate the browser to a URL,ambiguous,,0.0,0.0,0.0,0.0000,0.0000,166,56,13816,0.0,34,13458,0.0,17,13084,0.0, +q032,Take a screenshot of the page,ambiguous,,0.0,0.0,0.0,0.0000,0.0000,171,57,14038,0.0,35,13740,0.0,18,13672,0.0, +q033,Create a GitLab merge request,verb_product,,0.0,0.0,0.0,0.0000,0.0000,265,89,21452,0.0,53,20508,0.0,27,20309,0.0, +q034,Search Notion pages,verb_product,,0.0,0.0,0.0,0.0000,0.0000,156,52,12874,0.0,32,12736,0.0,16,12416,0.0, +q035,Create a Notion page,verb_product,,0.0,0.0,0.0,0.0000,0.0000,158,53,13156,0.0,32,12768,0.0,16,12432,0.0, +q036,Query a Notion database,verb_product,,0.0,0.0,0.0,0.0000,0.0000,162,54,13386,0.0,33,13124,0.0,17,13067,0.0, +q037,Create a Zendesk ticket,verb_product,,0.0,0.0,0.0,0.0000,0.0000,357,119,28019,0.0,72,27234,0.0,36,26478,0.0, +q038,Create a HubSpot contact,verb_product,,0.0,0.0,0.0,0.0000,0.0000,352,118,27762,0.0,71,26866,0.0,36,26478,0.0, +q039,Create a Salesforce lead,verb_product,,0.0,0.0,0.0,0.0000,0.0000,226,76,18006,0.0,46,17548,0.0,23,17088,0.0, +q040,Run a SOQL query,identifier,,0.0,0.0,0.0,0.0000,0.0000,220,74,17580,0.0,44,16791,0.0,22,16329,0.0, +q041,List Shopify orders,verb_product,,0.0,0.0,0.0,0.0000,0.0000,216,72,16998,0.0,44,16703,0.0,22,16285,0.0, +q042,Create a PagerDuty incident,verb_product,,0.0,0.0,0.0,0.0000,0.0000,240,80,19130,0.0,48,18458,0.0,24,17954,0.0, +q043,Who is on call in PagerDuty,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,243,81,19511,0.0,49,18939,0.0,25,18705,0.0, +q044,Query Datadog metrics,verb_product,,0.0,0.0,0.0,0.0000,0.0000,233,78,18569,0.0,47,17977,0.0,24,17954,0.0, +q045,Search Datadog logs,verb_product,,0.0,0.0,0.0,0.0000,0.0000,237,79,18863,0.0,48,18458,0.0,24,17954,0.0, +q046,List S3 buckets,verb_product,,0.0,0.0,0.0,0.0000,0.0000,293,98,23231,0.0,59,22522,0.0,30,22294,0.0, +q047,Upload an object to S3,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,296,99,23677,0.0,60,23014,0.0,30,22354,0.0, +q048,Purge Cloudflare cache,verb_product,,0.0,0.0,0.0,0.0000,0.0000,301,101,23981,0.0,61,23259,0.0,31,22962,0.0, +q049,Create a Cloudflare DNS record,verb_product,,0.0,0.0,0.0,0.0000,0.0000,300,100,23894,0.0,60,23014,0.0,30,22354,0.0, +q050,List Vercel deployments,verb_product,,0.0,0.0,0.0,0.0000,0.0000,303,101,23981,0.0,61,23259,0.0,31,22962,0.0, +q051,Insert a row into Supabase,verb_product,,0.0,0.0,0.0,0.0000,0.0000,308,103,24500,0.0,62,23675,0.0,31,22993,0.0, +q052,Export images from Figma,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,315,105,24863,0.0,63,23981,0.0,32,23662,0.0, +q053,Send an SMS with Twilio,verb_product,,0.0,0.0,0.0,0.0000,0.0000,316,106,25217,0.0,64,24398,0.0,32,23694,0.0, +q054,Create an Okta user,verb_product,,0.0,0.0,0.0,0.0000,0.0000,321,107,25289,0.0,65,24664,0.0,33,24298,0.0, +q055,Instant query Prometheus,verb_product,,0.0,0.0,0.0,0.0000,0.0000,323,108,25409,0.0,65,24599,0.0,33,24265,0.0, +q056,Search Grafana dashboards,verb_product,,0.0,0.0,0.0,0.0000,0.0000,326,109,25591,0.0,66,24925,0.0,33,24265,0.0, +q057,Trigger a CircleCI pipeline,verb_product,,0.0,0.0,0.0,0.0000,0.0000,331,111,26111,0.0,67,25340,0.0,34,25017,0.0, +q058,Run terraform plan,identifier,,0.0,0.0,0.0,0.0000,0.0000,332,111,25889,0.0,67,25206,0.0,34,24949,0.0, +q059,Create an Asana task,verb_product,,0.0,0.0,0.0,0.0000,0.0000,335,112,26352,0.0,67,25340,0.0,34,25017,0.0, +q060,Create a Trello card,verb_product,,0.0,0.0,0.0,0.0000,0.0000,340,114,26830,0.0,68,25731,0.0,34,25017,0.0, +q061,Create a ClickUp task,verb_product,,0.0,0.0,0.0,0.0000,0.0000,344,115,27066,0.0,69,26100,0.0,35,25770,0.0, +q062,Create a Confluence page,verb_product,,0.0,0.0,0.0,0.0000,0.0000,349,117,27546,0.0,70,26505,0.0,35,25770,0.0, +q063,Send a Discord message,verb_product,,0.0,0.0,0.0,0.0000,0.0000,362,121,28315,0.0,73,27492,0.0,37,27133,0.0, +q064,Search Elasticsearch,verb_product,,0.0,0.0,0.0,0.0000,0.0000,290,97,22834,0.0,58,22047,0.0,29,21525,0.0, +q065,Get a Redis key,verb_product,,0.0,0.0,0.0,0.0000,0.0000,286,96,22810,0.0,58,22163,0.0,29,21583,0.0, +q066,Find MongoDB documents,verb_product,,0.0,0.0,0.0,0.0000,0.0000,281,94,22397,0.0,57,21868,0.0,29,21583,0.0, +q067,List SQLite tables,verb_product,,0.0,0.0,0.0,0.0000,0.0000,277,93,22087,0.0,56,21431,0.0,28,20899,0.0, +q068,Describe a postgres table,verb_product,,0.0,0.0,0.0,0.0000,0.0000,100,34,8708,0.0,20,8336,0.0,10,8136,0.0, +q069,github_create_issue,identifier,,0.0,0.0,0.0,0.0000,0.0000,29,10,2762,0.0,6,2686,0.0,3,2629,0.0, +q070,createIssue,identifier,,0.0,0.0,0.0,0.0000,0.0000,29,10,2752,0.0,6,2680,0.0,3,2626,0.0, +q071,list-directory,identifier,,0.0,0.0,0.0,0.0000,0.0000,82,28,7599,0.0,17,7462,0.0,9,7580,0.0, +q072,sequential thinking next thought,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,116,39,9799,0.0,24,9727,0.0,12,9475,0.0, +q073,What time is it in Tokyo,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,117,39,9838,0.0,24,9751,0.0,12,9487,0.0, +q074,Convert time between timezones,verb_product,,0.0,0.0,0.0,0.0000,0.0000,118,40,10063,0.0,24,9727,0.0,12,9475,0.0, +q075,Star a GitHub repo,verb_product,,0.0,0.0,0.0,0.0000,0.0000,25,9,2387,0.0,5,2154,0.0,3,2635,0.0, +q076,Fork a repository,resource,,0.0,0.0,0.0,0.0000,0.0000,20,7,1831,0.0,4,1674,0.0,2,1636,0.0, +q077,Get the latest GitHub release,verb_product,,0.0,0.0,0.0,0.0000,0.0000,15,5,1242,0.0,3,1198,0.0,2,1642,0.0, +q078,Create a gist,resource,,0.0,0.0,0.0,0.0000,0.0000,66,22,6260,0.0,14,6405,0.0,7,6272,0.0, +q079,Search GitHub code,verb_product,,0.0,0.0,0.0,0.0000,0.0000,7,3,725,0.0,2,802,0.0,1,782,1.0, +q080,List GitHub notifications,verb_product,,0.0,0.0,0.0,0.0000,0.0000,53,18,5313,0.0,11,5225,0.0,6,5482,0.0, +q081,Assign Copilot to an issue,verb_product,,0.0,0.0,0.0,0.0000,0.0000,45,15,4523,0.0,9,4391,0.0,5,4776,0.0, +q082,Get secret scanning alerts,resource,,0.0,0.0,0.0,0.0000,0.0000,49,17,5097,0.0,10,4866,0.0,5,4766,0.0, +q083,Create or update a file on GitHub,verb_product,,0.0,0.0,0.0,0.0000,0.0000,17,6,1593,0.0,4,1694,0.0,2,1646,0.0, +q084,List collaborators on a repo,resource,,0.0,0.0,0.0,0.0000,0.0000,27,9,2387,0.0,6,2698,0.0,3,2635,0.0, +q085,Reply in a Slack thread,verb_product,,0.0,0.0,0.0,0.0000,0.0000,122,41,10333,0.0,25,10147,0.0,13,10188,0.0, +q086,Add an emoji reaction on Slack,verb_product,,0.0,0.0,0.0,0.0000,0.0000,123,41,10374,0.0,25,10172,0.0,13,10201,0.0, +q087,Search Slack messages,verb_product,,0.0,0.0,0.0,0.0000,0.0000,134,45,11132,0.0,27,10790,0.0,14,10985,0.0, +q088,Create a Slack channel,verb_product,,0.0,0.0,0.0,0.0000,0.0000,131,44,10940,0.0,27,10817,0.0,14,10999,0.0, +q089,Transition a Jira issue,verb_product,,0.0,0.0,0.0,0.0000,0.0000,151,51,12585,0.0,31,12385,0.0,16,12416,0.0, +q090,Search Jira with JQL,identifier,,0.0,0.0,0.0,0.0000,0.0000,149,50,12445,0.0,30,12025,0.0,15,11710,0.0, +q091,Add a worklog to Jira,verb_product,,0.0,0.0,0.0,0.0000,0.0000,155,52,12978,0.0,31,12447,0.0,16,12448,0.0, +q092,List Linear teams,verb_product,,0.0,0.0,0.0,0.0000,0.0000,142,48,11860,0.0,29,11573,0.0,15,11680,0.0, +q093,Comment on a Linear issue,verb_product,,0.0,0.0,0.0,0.0000,0.0000,141,47,11773,0.0,29,11631,0.0,15,11710,0.0, +q094,Move a local file,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,85,29,7859,0.0,17,7496,0.0,9,7598,0.0, +q095,Recursively search files by pattern,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,86,29,7917,0.0,18,7814,0.0,9,7616,0.0, +q096,Edit a file in place with old and new string,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,80,27,7629,0.0,16,7288,0.0,8,7080,0.0, +q097,Read the whole knowledge graph,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,95,32,8392,0.0,19,8062,0.0,10,8146,0.0, +q098,Search memory nodes,verb_product,,0.0,0.0,0.0,0.0000,0.0000,96,32,8328,0.0,20,8316,0.0,10,8126,0.0, +q099,Show unstaged git diff,verb_product,,0.0,0.0,0.0,0.0000,0.0000,105,35,8947,0.0,21,8653,0.0,11,8735,0.0, +q100,Stage files with git add,identifier,,0.0,0.0,0.0,0.0000,0.0000,109,37,9343,0.0,22,8966,0.0,11,8735,0.0, +q101,Create a Shopify product,verb_product,,0.0,0.0,0.0,0.0000,0.0000,214,72,17070,0.0,43,16424,0.0,22,16307,0.0, +q102,Cancel a Stripe subscription,verb_product,,0.0,0.0,0.0,0.0000,0.0000,210,70,16604,0.0,42,16044,0.0,21,15624,0.0, +q103,Create a payment intent,resource,,0.0,0.0,0.0,0.0000,0.0000,203,68,16204,0.0,41,15712,0.0,21,15624,0.0, +q104,Acknowledge a PagerDuty incident,verb_product,,0.0,0.0,0.0,0.0000,0.0000,241,81,19511,0.0,49,18939,0.0,25,18705,0.0, +q105,Create a Datadog monitor,verb_product,,0.0,0.0,0.0,0.0000,0.0000,236,79,18942,0.0,48,18506,0.0,24,17978,0.0, +q106,Resolve a Sentry issue,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,231,77,18333,0.0,47,17977,0.0,24,17954,0.0, +q107,Scale a Kubernetes deployment,verb_product,,0.0,0.0,0.0,0.0000,0.0000,197,66,15784,0.0,40,15384,0.0,20,14984,0.0, +q108,Apply a Kubernetes manifest,verb_product,,0.0,0.0,0.0,0.0000,0.0000,194,65,15558,0.0,39,15038,0.0,20,14984,0.0, +q109,Stop a docker container,verb_product,,0.0,0.0,0.0,0.0000,0.0000,186,62,14888,0.0,38,14630,0.0,19,14250,0.0, +q110,Build a docker image,verb_product,,0.0,0.0,0.0,0.0000,0.0000,189,63,15066,0.0,38,14630,0.0,19,14250,0.0, +q111,Click an element in Playwright,verb_product,,0.0,0.0,0.0,0.0000,0.0000,168,56,13816,0.0,34,13458,0.0,17,13084,0.0, +q112,Type text into a browser input,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,169,57,14038,0.0,34,13458,0.0,17,13084,0.0, +q113,List GitLab projects,verb_product,,0.0,0.0,0.0,0.0000,0.0000,269,90,21515,0.0,54,20795,0.0,27,20255,0.0, +q114,Create a Bitbucket pull request,verb_product,,0.0,0.0,0.0,0.0000,0.0000,272,91,21988,0.0,55,21340,0.0,28,20983,0.0, +q115,Create a Gmail draft,verb_product,,0.0,0.0,0.0,0.0000,0.0000,247,83,19734,0.0,50,19130,0.0,25,18630,0.0, +q116,Share a Google Drive file,verb_product,,0.0,0.0,0.0,0.0000,0.0000,256,86,20597,0.0,52,20043,0.0,26,19497,0.0, +q117,List Google Calendar events,verb_product,,0.0,0.0,0.0,0.0000,0.0000,257,86,20511,0.0,52,19991,0.0,26,19471,0.0, +q118,Delete a calendar event,resource,,0.0,0.0,0.0,0.0000,0.0000,260,87,20729,0.0,52,19991,0.0,26,19471,0.0, +q119,List Vercel projects,verb_product,,0.0,0.0,0.0,0.0000,0.0000,302,101,23981,0.0,61,23259,0.0,31,22962,0.0, +q120,Select rows from Supabase,verb_product,,0.0,0.0,0.0,0.0000,0.0000,307,103,24397,0.0,62,23613,0.0,31,22962,0.0, +q121,Post a Figma comment,verb_product,,0.0,0.0,0.0,0.0000,0.0000,314,105,24863,0.0,63,23981,0.0,32,23662,0.0, +q122,List Okta users,verb_product,,0.0,0.0,0.0,0.0000,0.0000,319,107,25182,0.0,64,24270,0.0,32,23630,0.0, +q123,Range query Prometheus,verb_product,,0.0,0.0,0.0,0.0000,0.0000,324,108,25409,0.0,65,24599,0.0,33,24265,0.0, +q124,Create a Grafana annotation,verb_product,,0.0,0.0,0.0,0.0000,0.0000,328,110,25915,0.0,66,24991,0.0,33,24298,0.0, +q125,Apply terraform,identifier,,0.0,0.0,0.0,0.0000,0.0000,333,111,25778,0.0,67,25139,0.0,34,24915,0.0, +q126,List Asana tasks,verb_product,,0.0,0.0,0.0,0.0000,0.0000,336,112,26240,0.0,68,25663,0.0,34,24983,0.0, +q127,Comment on a Trello card,verb_product,,0.0,0.0,0.0,0.0000,0.0000,343,115,27181,0.0,69,26169,0.0,35,25805,0.0, +q128,Update a ClickUp task,verb_product,,0.0,0.0,0.0,0.0000,0.0000,347,116,27289,0.0,70,26505,0.0,35,25770,0.0, +q129,Search Confluence,verb_product,,0.0,0.0,0.0,0.0000,0.0000,351,117,27312,0.0,71,26724,0.0,36,26406,0.0, +q130,Create a HubSpot deal,verb_product,,0.0,0.0,0.0,0.0000,0.0000,355,119,28019,0.0,71,26866,0.0,36,26478,0.0, +q131,List Zendesk tickets,verb_product,,0.0,0.0,0.0,0.0000,0.0000,359,120,28122,0.0,72,27162,0.0,36,26442,0.0, +q132,List Discord channels,verb_product,,0.0,0.0,0.0,0.0000,0.0000,363,121,28194,0.0,73,27419,0.0,37,27096,0.0, +q133,Set a Redis key,verb_product,,0.0,0.0,0.0,0.0000,0.0000,287,96,22810,0.0,58,22163,0.0,29,21583,0.0, +q134,Insert MongoDB documents,verb_product,,0.0,0.0,0.0,0.0000,0.0000,282,94,22397,0.0,57,21868,0.0,29,21583,0.0, +q135,Run a MySQL query,verb_product,,0.0,0.0,0.0,0.0000,0.0000,279,93,22273,0.0,56,21543,0.0,28,20955,0.0, +q136,Index a document in Elasticsearch,verb_product,,0.0,0.0,0.0,0.0000,0.0000,291,97,23125,0.0,59,22581,0.0,30,22324,0.0, +q137,Get an S3 object,verb_product,,0.0,0.0,0.0,0.0000,0.0000,295,99,23578,0.0,59,22581,0.0,30,22324,0.0, +q138,List Cloudflare zones,verb_product,,0.0,0.0,0.0,0.0000,0.0000,298,100,23694,0.0,60,22894,0.0,30,22294,0.0, +q139,Create a Stripe invoice,verb_product,,0.0,0.0,0.0,0.0000,0.0000,205,69,16389,0.0,41,15712,0.0,21,15624,0.0, +q140,List GitHub Actions workflows,verb_product,,0.0,0.0,0.0,0.0000,0.0000,60,20,5782,0.0,12,5614,0.0,6,5488,0.0, +q141,Get GitHub Actions job logs,verb_product,,0.0,0.0,0.0,0.0000,0.0000,63,21,6123,0.0,13,6050,0.0,7,6293,0.0, +q142,Create a GitHub repository,verb_product,,0.0,0.0,0.0,0.0000,0.0000,18,6,1575,0.0,4,1682,0.0,2,1640,0.0, +q143,Checkout a git branch,verb_product,,0.0,0.0,0.0,0.0000,0.0000,113,38,9533,0.0,23,9279,0.0,12,9463,0.0, +q144,Show git log,identifier,,0.0,0.0,0.0,0.0000,0.0000,111,37,9269,0.0,23,9256,0.0,12,9451,0.0, +q145,List postgres schemas,verb_product,,0.0,0.0,0.0,0.0000,0.0000,101,34,8674,0.0,21,8611,0.0,11,8713,0.0, +q146,Open nodes in the knowledge graph,verb_product,,0.0,0.0,0.0,0.0000,0.0000,97,33,8598,0.0,20,8376,0.0,10,8156,0.0, +q147,Get file metadata from the filesystem,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,87,29,7917,0.0,18,7814,0.0,9,7616,0.0, +q148,Write a new local file,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,79,27,7494,0.0,16,7208,0.0,8,7040,0.0, +q149,Search GitHub issues,verb_product,,0.0,0.0,0.0,0.0000,0.0000,32,11,3079,0.0,7,3209,0.0,4,3672,0.0, +q150,Get the authenticated GitHub user,paraphrase,,0.0,0.0,0.0,0.0000,0.0000,1,1,181,1.0,1,379,1.0,1,784,1.0, diff --git a/experiments/mcp-tool-search/results/name_exact_substr.json b/experiments/mcp-tool-search/results/name_exact_substr.json new file mode 100644 index 00000000..fddacec8 --- /dev/null +++ b/experiments/mcp-tool-search/results/name_exact_substr.json @@ -0,0 +1,5363 @@ +{ + "summary": { + "n": 150, + "index_s": 0.0, + "rss_mb": 0.01953125, + "p50_ms": 0.31149599993796073, + "p95_ms": 0.35911225004383596, + "recall@1": 0.0, + "recall@1_lo": 0.0, + "recall@1_hi": 0.0, + "recall@3": 0.0, + "recall@3_lo": 0.0, + "recall@3_hi": 0.0, + "recall@5": 0.0, + "recall@5_lo": 0.0, + "recall@5_hi": 0.0, + "mrr@10": 0.0, + "mrr@10_lo": 0.0, + "mrr@10_hi": 0.0, + "ndcg@10": 0.0, + "ndcg@10_lo": 0.0, + "ndcg@10_hi": 0.0 + }, + "meta": { + "name": "name_exact_substr", + "needs_model": false, + "needs_api": false, + "notes": "exact/substring on tool name" + }, + "rows": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 29, + "k3_calls": 10, + "k3_tokens": 2782, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 6, + "k5_tokens": 2698, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 3, + "k10_tokens": 2635, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "kind": "verb_product", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 146, + "k3_calls": 49, + "k3_tokens": 12183, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 30, + "k5_tokens": 11995, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11695, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "kind": "verb_product", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 121, + "k3_calls": 41, + "k3_tokens": 10292, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 25, + "k5_tokens": 10122, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 13, + "k10_tokens": 10175, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "kind": "resource", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 37, + "k3_calls": 13, + "k3_tokens": 3710, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 8, + "k5_tokens": 3752, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3672, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 36, + "k3_calls": 12, + "k3_tokens": 3495, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 8, + "k5_tokens": 3776, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3684, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "kind": "verb_product", + "gold": [ + "postgres.query" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 98, + "k3_calls": 33, + "k3_tokens": 8598, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 20, + "k5_tokens": 8376, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 10, + "k10_tokens": 8156, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "kind": "paraphrase", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 76, + "k3_calls": 26, + "k3_tokens": 7282, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 16, + "k5_tokens": 7224, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 8, + "k10_tokens": 7048, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "kind": "paraphrase", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 82, + "k3_calls": 28, + "k3_tokens": 7711, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 17, + "k5_tokens": 7530, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7616, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 89, + "k3_calls": 30, + "k3_tokens": 8108, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7832, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7625, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "kind": "verb_product", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 119, + "k3_calls": 40, + "k3_tokens": 10063, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 24, + "k5_tokens": 9727, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9475, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "kind": "verb_product", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 103, + "k3_calls": 35, + "k3_tokens": 8982, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 21, + "k5_tokens": 8674, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 11, + "k10_tokens": 8746, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q012", + "query": "Show git status", + "kind": "identifier", + "gold": [ + "git.git_status" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 104, + "k3_calls": 35, + "k3_tokens": 8877, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 21, + "k5_tokens": 8611, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 11, + "k10_tokens": 8713, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "kind": "paraphrase", + "gold": [ + "git.git_commit" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 108, + "k3_calls": 36, + "k3_tokens": 9075, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 22, + "k5_tokens": 8922, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 11, + "k10_tokens": 8713, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 136, + "k3_calls": 46, + "k3_tokens": 11528, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 28, + "k5_tokens": 11279, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 14, + "k10_tokens": 10999, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "kind": "paraphrase", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 136, + "k3_calls": 46, + "k3_tokens": 11712, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 28, + "k5_tokens": 11391, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 14, + "k10_tokens": 11055, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 34, + "k3_calls": 12, + "k3_tokens": 3507, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 7, + "k5_tokens": 3237, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3688, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "kind": "ambiguous", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 41, + "k3_calls": 14, + "k3_tokens": 4072, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 9, + "k5_tokens": 4373, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 5, + "k10_tokens": 4766, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "kind": "verb_product", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 62, + "k3_calls": 21, + "k3_tokens": 6123, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 13, + "k5_tokens": 6050, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 7, + "k10_tokens": 6293, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "kind": "resource", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 51, + "k3_calls": 17, + "k3_tokens": 5097, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 11, + "k5_tokens": 5225, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 6, + "k10_tokens": 5482, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "kind": "verb_product", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 228, + "k3_calls": 76, + "k3_tokens": 18006, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 46, + "k5_tokens": 17548, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 23, + "k10_tokens": 17088, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 200, + "k3_calls": 67, + "k3_tokens": 15974, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 40, + "k5_tokens": 15384, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 20, + "k10_tokens": 14984, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "kind": "paraphrase", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 211, + "k3_calls": 71, + "k3_tokens": 16820, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 43, + "k5_tokens": 16424, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 22, + "k10_tokens": 16307, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "kind": "verb_product", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 244, + "k3_calls": 82, + "k3_tokens": 19588, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 49, + "k5_tokens": 18841, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 25, + "k10_tokens": 18655, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "kind": "verb_product", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 245, + "k3_calls": 82, + "k3_tokens": 19506, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 49, + "k5_tokens": 18792, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 25, + "k10_tokens": 18630, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "kind": "paraphrase", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 253, + "k3_calls": 85, + "k3_tokens": 20366, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 51, + "k5_tokens": 19618, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 26, + "k10_tokens": 19523, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 258, + "k3_calls": 86, + "k3_tokens": 20511, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 52, + "k5_tokens": 19991, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 26, + "k10_tokens": 19471, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 191, + "k3_calls": 64, + "k3_tokens": 15221, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 39, + "k5_tokens": 14999, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 20, + "k10_tokens": 14964, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 193, + "k3_calls": 65, + "k3_tokens": 15818, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 39, + "k5_tokens": 15194, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 20, + "k10_tokens": 15064, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 185, + "k3_calls": 62, + "k3_tokens": 14888, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 37, + "k5_tokens": 14342, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 19, + "k10_tokens": 14250, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "kind": "verb_product", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 184, + "k3_calls": 62, + "k3_tokens": 14888, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 37, + "k5_tokens": 14342, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 19, + "k10_tokens": 14250, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "kind": "ambiguous", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 166, + "k3_calls": 56, + "k3_tokens": 13816, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 34, + "k5_tokens": 13458, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 17, + "k10_tokens": 13084, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "kind": "ambiguous", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 171, + "k3_calls": 57, + "k3_tokens": 14038, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 35, + "k5_tokens": 13740, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 18, + "k10_tokens": 13672, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 265, + "k3_calls": 89, + "k3_tokens": 21452, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 53, + "k5_tokens": 20508, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 27, + "k10_tokens": 20309, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "kind": "verb_product", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 156, + "k3_calls": 52, + "k3_tokens": 12874, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 32, + "k5_tokens": 12736, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 16, + "k10_tokens": 12416, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "kind": "verb_product", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 158, + "k3_calls": 53, + "k3_tokens": 13156, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 32, + "k5_tokens": 12768, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 16, + "k10_tokens": 12432, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "kind": "verb_product", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 162, + "k3_calls": 54, + "k3_tokens": 13386, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 33, + "k5_tokens": 13124, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 17, + "k10_tokens": 13067, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 357, + "k3_calls": 119, + "k3_tokens": 28019, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 72, + "k5_tokens": 27234, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 36, + "k10_tokens": 26478, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 352, + "k3_calls": 118, + "k3_tokens": 27762, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 71, + "k5_tokens": 26866, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 36, + "k10_tokens": 26478, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "kind": "verb_product", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 226, + "k3_calls": 76, + "k3_tokens": 18006, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 46, + "k5_tokens": 17548, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 23, + "k10_tokens": 17088, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "kind": "identifier", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 220, + "k3_calls": 74, + "k3_tokens": 17580, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 44, + "k5_tokens": 16791, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 22, + "k10_tokens": 16329, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "kind": "verb_product", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 216, + "k3_calls": 72, + "k3_tokens": 16998, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 44, + "k5_tokens": 16703, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 22, + "k10_tokens": 16285, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 240, + "k3_calls": 80, + "k3_tokens": 19130, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 48, + "k5_tokens": 18458, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 24, + "k10_tokens": 17954, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "kind": "paraphrase", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 243, + "k3_calls": 81, + "k3_tokens": 19511, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 49, + "k5_tokens": 18939, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 25, + "k10_tokens": 18705, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "kind": "verb_product", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 233, + "k3_calls": 78, + "k3_tokens": 18569, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 47, + "k5_tokens": 17977, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 24, + "k10_tokens": 17954, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "kind": "verb_product", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 237, + "k3_calls": 79, + "k3_tokens": 18863, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 48, + "k5_tokens": 18458, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 24, + "k10_tokens": 17954, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 293, + "k3_calls": 98, + "k3_tokens": 23231, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 59, + "k5_tokens": 22522, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 30, + "k10_tokens": 22294, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "kind": "paraphrase", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 296, + "k3_calls": 99, + "k3_tokens": 23677, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 60, + "k5_tokens": 23014, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 30, + "k10_tokens": 22354, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 301, + "k3_calls": 101, + "k3_tokens": 23981, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 61, + "k5_tokens": 23259, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 31, + "k10_tokens": 22962, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 300, + "k3_calls": 100, + "k3_tokens": 23894, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 60, + "k5_tokens": 23014, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 30, + "k10_tokens": 22354, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 303, + "k3_calls": 101, + "k3_tokens": 23981, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 61, + "k5_tokens": 23259, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 31, + "k10_tokens": 22962, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 308, + "k3_calls": 103, + "k3_tokens": 24500, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 62, + "k5_tokens": 23675, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 31, + "k10_tokens": 22993, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "kind": "paraphrase", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 315, + "k3_calls": 105, + "k3_tokens": 24863, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 63, + "k5_tokens": 23981, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 32, + "k10_tokens": 23662, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "kind": "verb_product", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 316, + "k3_calls": 106, + "k3_tokens": 25217, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 64, + "k5_tokens": 24398, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 32, + "k10_tokens": 23694, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "kind": "verb_product", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 321, + "k3_calls": 107, + "k3_tokens": 25289, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 65, + "k5_tokens": 24664, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 33, + "k10_tokens": 24298, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 323, + "k3_calls": 108, + "k3_tokens": 25409, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 65, + "k5_tokens": 24599, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 33, + "k10_tokens": 24265, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "kind": "verb_product", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 326, + "k3_calls": 109, + "k3_tokens": 25591, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 66, + "k5_tokens": 24925, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 33, + "k10_tokens": 24265, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "kind": "verb_product", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 331, + "k3_calls": 111, + "k3_tokens": 26111, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 67, + "k5_tokens": 25340, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 34, + "k10_tokens": 25017, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "kind": "identifier", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 332, + "k3_calls": 111, + "k3_tokens": 25889, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 67, + "k5_tokens": 25206, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 34, + "k10_tokens": 24949, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "kind": "verb_product", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 335, + "k3_calls": 112, + "k3_tokens": 26352, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 67, + "k5_tokens": 25340, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 34, + "k10_tokens": 25017, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 340, + "k3_calls": 114, + "k3_tokens": 26830, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 68, + "k5_tokens": 25731, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 34, + "k10_tokens": 25017, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 344, + "k3_calls": 115, + "k3_tokens": 27066, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 69, + "k5_tokens": 26100, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 35, + "k10_tokens": 25770, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "kind": "verb_product", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 349, + "k3_calls": 117, + "k3_tokens": 27546, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 70, + "k5_tokens": 26505, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 35, + "k10_tokens": 25770, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "kind": "verb_product", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 362, + "k3_calls": 121, + "k3_tokens": 28315, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 73, + "k5_tokens": 27492, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 37, + "k10_tokens": 27133, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 290, + "k3_calls": 97, + "k3_tokens": 22834, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 58, + "k5_tokens": 22047, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 29, + "k10_tokens": 21525, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 286, + "k3_calls": 96, + "k3_tokens": 22810, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 58, + "k5_tokens": 22163, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 29, + "k10_tokens": 21583, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 281, + "k3_calls": 94, + "k3_tokens": 22397, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 57, + "k5_tokens": 21868, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 29, + "k10_tokens": 21583, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "kind": "verb_product", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 277, + "k3_calls": 93, + "k3_tokens": 22087, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 56, + "k5_tokens": 21431, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 28, + "k10_tokens": 20899, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "kind": "verb_product", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 100, + "k3_calls": 34, + "k3_tokens": 8708, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 20, + "k5_tokens": 8336, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 10, + "k10_tokens": 8136, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 29, + "k3_calls": 10, + "k3_tokens": 2762, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 6, + "k5_tokens": 2686, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 3, + "k10_tokens": 2629, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 29, + "k3_calls": 10, + "k3_tokens": 2752, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 6, + "k5_tokens": 2680, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 3, + "k10_tokens": 2626, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "kind": "identifier", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 82, + "k3_calls": 28, + "k3_tokens": 7599, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 17, + "k5_tokens": 7462, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7580, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "kind": "paraphrase", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 116, + "k3_calls": 39, + "k3_tokens": 9799, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 24, + "k5_tokens": 9727, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9475, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "kind": "paraphrase", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 117, + "k3_calls": 39, + "k3_tokens": 9838, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 24, + "k5_tokens": 9751, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9487, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "kind": "verb_product", + "gold": [ + "time.convert_time" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 118, + "k3_calls": 40, + "k3_tokens": 10063, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 24, + "k5_tokens": 9727, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9475, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "kind": "verb_product", + "gold": [ + "github.star_repository" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 25, + "k3_calls": 9, + "k3_tokens": 2387, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 5, + "k5_tokens": 2154, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 3, + "k10_tokens": 2635, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "kind": "resource", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 20, + "k3_calls": 7, + "k3_tokens": 1831, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 4, + "k5_tokens": 1674, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 2, + "k10_tokens": 1636, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "kind": "verb_product", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 15, + "k3_calls": 5, + "k3_tokens": 1242, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 3, + "k5_tokens": 1198, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 1.0, + "k10_calls": 2, + "k10_tokens": 1642, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "kind": "resource", + "gold": [ + "github.create_gist" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 66, + "k3_calls": 22, + "k3_tokens": 6260, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 14, + "k5_tokens": 6405, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 7, + "k10_tokens": 6272, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "kind": "verb_product", + "gold": [ + "github.search_code" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 7, + "k3_calls": 3, + "k3_tokens": 725, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 802, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 782, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "kind": "verb_product", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 53, + "k3_calls": 18, + "k3_tokens": 5313, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 11, + "k5_tokens": 5225, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 6, + "k10_tokens": 5482, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "kind": "verb_product", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 45, + "k3_calls": 15, + "k3_tokens": 4523, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 9, + "k5_tokens": 4391, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 5, + "k10_tokens": 4776, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "kind": "resource", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 49, + "k3_calls": 17, + "k3_tokens": 5097, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 10, + "k5_tokens": 4866, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 5, + "k10_tokens": 4766, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 17, + "k3_calls": 6, + "k3_tokens": 1593, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 4, + "k5_tokens": 1694, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 2, + "k10_tokens": 1646, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "kind": "resource", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 27, + "k3_calls": 9, + "k3_tokens": 2387, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 6, + "k5_tokens": 2698, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 3, + "k10_tokens": 2635, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "kind": "verb_product", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 122, + "k3_calls": 41, + "k3_tokens": 10333, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 25, + "k5_tokens": 10147, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 13, + "k10_tokens": 10188, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "kind": "verb_product", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 123, + "k3_calls": 41, + "k3_tokens": 10374, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 25, + "k5_tokens": 10172, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 13, + "k10_tokens": 10201, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "kind": "verb_product", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 134, + "k3_calls": 45, + "k3_tokens": 11132, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 27, + "k5_tokens": 10790, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 14, + "k10_tokens": 10985, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "kind": "verb_product", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 131, + "k3_calls": 44, + "k3_tokens": 10940, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 27, + "k5_tokens": 10817, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 14, + "k10_tokens": 10999, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "kind": "verb_product", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 151, + "k3_calls": 51, + "k3_tokens": 12585, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 31, + "k5_tokens": 12385, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 16, + "k10_tokens": 12416, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "kind": "identifier", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 149, + "k3_calls": 50, + "k3_tokens": 12445, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 30, + "k5_tokens": 12025, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11710, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "kind": "verb_product", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 155, + "k3_calls": 52, + "k3_tokens": 12978, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 31, + "k5_tokens": 12447, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 16, + "k10_tokens": 12448, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "kind": "verb_product", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 142, + "k3_calls": 48, + "k3_tokens": 11860, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 29, + "k5_tokens": 11573, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11680, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 141, + "k3_calls": 47, + "k3_tokens": 11773, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 29, + "k5_tokens": 11631, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11710, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q094", + "query": "Move a local file", + "kind": "paraphrase", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 85, + "k3_calls": 29, + "k3_tokens": 7859, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 17, + "k5_tokens": 7496, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7598, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "kind": "paraphrase", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 86, + "k3_calls": 29, + "k3_tokens": 7917, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7814, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7616, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "kind": "paraphrase", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 80, + "k3_calls": 27, + "k3_tokens": 7629, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 16, + "k5_tokens": 7288, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 8, + "k10_tokens": 7080, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 95, + "k3_calls": 32, + "k3_tokens": 8392, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 19, + "k5_tokens": 8062, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 10, + "k10_tokens": 8146, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "kind": "verb_product", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 96, + "k3_calls": 32, + "k3_tokens": 8328, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 20, + "k5_tokens": 8316, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 10, + "k10_tokens": 8126, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "kind": "verb_product", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 105, + "k3_calls": 35, + "k3_tokens": 8947, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 21, + "k5_tokens": 8653, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 11, + "k10_tokens": 8735, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "kind": "identifier", + "gold": [ + "git.git_add" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 109, + "k3_calls": 37, + "k3_tokens": 9343, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 22, + "k5_tokens": 8966, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 11, + "k10_tokens": 8735, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "kind": "verb_product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 214, + "k3_calls": 72, + "k3_tokens": 17070, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 43, + "k5_tokens": 16424, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 22, + "k10_tokens": 16307, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "kind": "verb_product", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 210, + "k3_calls": 70, + "k3_tokens": 16604, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 42, + "k5_tokens": 16044, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 21, + "k10_tokens": 15624, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "kind": "resource", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 203, + "k3_calls": 68, + "k3_tokens": 16204, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 41, + "k5_tokens": 15712, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 21, + "k10_tokens": 15624, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 241, + "k3_calls": 81, + "k3_tokens": 19511, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 49, + "k5_tokens": 18939, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 25, + "k10_tokens": 18705, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "kind": "verb_product", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 236, + "k3_calls": 79, + "k3_tokens": 18942, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 48, + "k5_tokens": 18506, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 24, + "k10_tokens": 17978, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "kind": "paraphrase", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 231, + "k3_calls": 77, + "k3_tokens": 18333, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 47, + "k5_tokens": 17977, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 24, + "k10_tokens": 17954, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 197, + "k3_calls": 66, + "k3_tokens": 15784, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 40, + "k5_tokens": 15384, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 20, + "k10_tokens": 14984, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 194, + "k3_calls": 65, + "k3_tokens": 15558, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 39, + "k5_tokens": 15038, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 20, + "k10_tokens": 14984, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 186, + "k3_calls": 62, + "k3_tokens": 14888, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 38, + "k5_tokens": 14630, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 19, + "k10_tokens": 14250, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "kind": "verb_product", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 189, + "k3_calls": 63, + "k3_tokens": 15066, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 38, + "k5_tokens": 14630, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 19, + "k10_tokens": 14250, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "kind": "verb_product", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 168, + "k3_calls": 56, + "k3_tokens": 13816, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 34, + "k5_tokens": 13458, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 17, + "k10_tokens": 13084, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "kind": "paraphrase", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 169, + "k3_calls": 57, + "k3_tokens": 14038, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 34, + "k5_tokens": 13458, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 17, + "k10_tokens": 13084, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 269, + "k3_calls": 90, + "k3_tokens": 21515, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 54, + "k5_tokens": 20795, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 27, + "k10_tokens": 20255, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "kind": "verb_product", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 272, + "k3_calls": 91, + "k3_tokens": 21988, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 55, + "k5_tokens": 21340, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 28, + "k10_tokens": 20983, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "kind": "verb_product", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 247, + "k3_calls": 83, + "k3_tokens": 19734, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 50, + "k5_tokens": 19130, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 25, + "k10_tokens": 18630, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "kind": "verb_product", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 256, + "k3_calls": 86, + "k3_tokens": 20597, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 52, + "k5_tokens": 20043, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 26, + "k10_tokens": 19497, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "kind": "verb_product", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 257, + "k3_calls": 86, + "k3_tokens": 20511, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 52, + "k5_tokens": 19991, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 26, + "k10_tokens": 19471, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 260, + "k3_calls": 87, + "k3_tokens": 20729, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 52, + "k5_tokens": 19991, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 26, + "k10_tokens": 19471, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 302, + "k3_calls": 101, + "k3_tokens": 23981, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 61, + "k5_tokens": 23259, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 31, + "k10_tokens": 22962, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 307, + "k3_calls": 103, + "k3_tokens": 24397, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 62, + "k5_tokens": 23613, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 31, + "k10_tokens": 22962, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "kind": "verb_product", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 314, + "k3_calls": 105, + "k3_tokens": 24863, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 63, + "k5_tokens": 23981, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 32, + "k10_tokens": 23662, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q122", + "query": "List Okta users", + "kind": "verb_product", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 319, + "k3_calls": 107, + "k3_tokens": 25182, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 64, + "k5_tokens": 24270, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 32, + "k10_tokens": 23630, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 324, + "k3_calls": 108, + "k3_tokens": 25409, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 65, + "k5_tokens": 24599, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 33, + "k10_tokens": 24265, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "kind": "verb_product", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 328, + "k3_calls": 110, + "k3_tokens": 25915, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 66, + "k5_tokens": 24991, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 33, + "k10_tokens": 24298, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "kind": "identifier", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 333, + "k3_calls": 111, + "k3_tokens": 25778, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 67, + "k5_tokens": 25139, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 34, + "k10_tokens": 24915, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "kind": "verb_product", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 336, + "k3_calls": 112, + "k3_tokens": 26240, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 68, + "k5_tokens": 25663, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 34, + "k10_tokens": 24983, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 343, + "k3_calls": 115, + "k3_tokens": 27181, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 69, + "k5_tokens": 26169, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 35, + "k10_tokens": 25805, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 347, + "k3_calls": 116, + "k3_tokens": 27289, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 70, + "k5_tokens": 26505, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 35, + "k10_tokens": 25770, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "kind": "verb_product", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 351, + "k3_calls": 117, + "k3_tokens": 27312, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 71, + "k5_tokens": 26724, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 36, + "k10_tokens": 26406, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 355, + "k3_calls": 119, + "k3_tokens": 28019, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 71, + "k5_tokens": 26866, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 36, + "k10_tokens": 26478, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 359, + "k3_calls": 120, + "k3_tokens": 28122, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 72, + "k5_tokens": 27162, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 36, + "k10_tokens": 26442, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "kind": "verb_product", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 363, + "k3_calls": 121, + "k3_tokens": 28194, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 73, + "k5_tokens": 27419, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 37, + "k10_tokens": 27096, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 287, + "k3_calls": 96, + "k3_tokens": 22810, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 58, + "k5_tokens": 22163, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 29, + "k10_tokens": 21583, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 282, + "k3_calls": 94, + "k3_tokens": 22397, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 57, + "k5_tokens": 21868, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 29, + "k10_tokens": 21583, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "kind": "verb_product", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 279, + "k3_calls": 93, + "k3_tokens": 22273, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 56, + "k5_tokens": 21543, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 28, + "k10_tokens": 20955, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 291, + "k3_calls": 97, + "k3_tokens": 23125, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 59, + "k5_tokens": 22581, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 30, + "k10_tokens": 22324, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 295, + "k3_calls": 99, + "k3_tokens": 23578, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 59, + "k5_tokens": 22581, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 30, + "k10_tokens": 22324, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 298, + "k3_calls": 100, + "k3_tokens": 23694, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 60, + "k5_tokens": 22894, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 30, + "k10_tokens": 22294, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 205, + "k3_calls": 69, + "k3_tokens": 16389, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 41, + "k5_tokens": 15712, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 21, + "k10_tokens": 15624, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "kind": "verb_product", + "gold": [ + "github.actions_list" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 60, + "k3_calls": 20, + "k3_tokens": 5782, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 12, + "k5_tokens": 5614, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 6, + "k10_tokens": 5488, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "kind": "verb_product", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 63, + "k3_calls": 21, + "k3_tokens": 6123, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 13, + "k5_tokens": 6050, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 7, + "k10_tokens": 6293, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "kind": "verb_product", + "gold": [ + "github.create_repository" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 18, + "k3_calls": 6, + "k3_tokens": 1575, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 4, + "k5_tokens": 1682, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 2, + "k10_tokens": 1640, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "kind": "verb_product", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 113, + "k3_calls": 38, + "k3_tokens": 9533, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 23, + "k5_tokens": 9279, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9463, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q144", + "query": "Show git log", + "kind": "identifier", + "gold": [ + "git.git_log" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 111, + "k3_calls": 37, + "k3_tokens": 9269, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 23, + "k5_tokens": 9256, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 12, + "k10_tokens": 9451, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "kind": "verb_product", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 101, + "k3_calls": 34, + "k3_tokens": 8674, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 21, + "k5_tokens": 8611, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 11, + "k10_tokens": 8713, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "kind": "verb_product", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 97, + "k3_calls": 33, + "k3_tokens": 8598, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 20, + "k5_tokens": 8376, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 10, + "k10_tokens": 8156, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "kind": "paraphrase", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 87, + "k3_calls": 29, + "k3_tokens": 7917, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 18, + "k5_tokens": 7814, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 9, + "k10_tokens": 7616, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "kind": "paraphrase", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 79, + "k3_calls": 27, + "k3_tokens": 7494, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 16, + "k5_tokens": 7208, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 8, + "k10_tokens": 7040, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "kind": "verb_product", + "gold": [ + "github.search_issues" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 32, + "k3_calls": 11, + "k3_tokens": 3079, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 7, + "k5_tokens": 3209, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 4, + "k10_tokens": 3672, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "kind": "paraphrase", + "gold": [ + "github.get_me" + ], + "alts": [], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 181, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 379, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 784, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "cost": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 63.74666666666667, + "p95_calls": 116.54999999999998, + "success_within_1": 0.006666666666666667, + "success_within_2": 0.006666666666666667, + "success_within_3": 0.013333333333333334, + "mean_tokens": 15418.586666666666, + "p95_tokens": 27301.65, + "total_tokens": 2312788, + "tokens_per_successful_query": 15418.586666666666, + "total_tokens_successful": 2312788 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 38.48, + "p95_calls": 70.0, + "success_within_1": 0.006666666666666667, + "success_within_2": 0.013333333333333334, + "success_within_3": 0.02, + "mean_tokens": 14978.92, + "p95_tokens": 26505.0, + "total_tokens": 2246838, + "tokens_per_successful_query": 14978.92, + "total_tokens_successful": 2246838 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 19.486666666666668, + "p95_calls": 35.0, + "success_within_1": 0.013333333333333334, + "success_within_2": 0.04, + "success_within_3": 0.07333333333333333, + "mean_tokens": 14759.053333333333, + "p95_tokens": 25789.25, + "total_tokens": 2213858, + "tokens_per_successful_query": 14759.053333333333, + "total_tokens_successful": 2213858 + } + } +} diff --git a/experiments/mcp-tool-search/results/query_rewrite_weighted_bm25.csv b/experiments/mcp-tool-search/results/query_rewrite_weighted_bm25.csv new file mode 100644 index 00000000..6e88dd10 --- /dev/null +++ b/experiments/mcp-tool-search/results/query_rewrite_weighted_bm25.csv @@ -0,0 +1,151 @@ +id,query,kind,hit1,recall@1,recall@3,recall@5,mrr@10,ndcg@10,hit_rank,k3_calls,k3_tokens,k3_success1,k5_calls,k5_tokens,k5_success1,k10_calls,k10_tokens,k10_success1,ranking +q001,Create a GitHub issue,verb_product,github.create_issue,1.0,1.0,1.0,1.0000,0.9502,1,1,388,1.0,1,598,1.0,1,1021,1.0,github.create_issue github.issue_read github.issue_write github.add_issue_comment github.update_issue_comment github.assign_copilot_to_issue bitbucket.bitbucket_create_issue sentry.sentry_get_issue gitlab.gitlab_create_issue jira.jira_assign_issue +q002,Create a Jira ticket,verb_product,jira.jira_assign_issue,0.0,0.0,1.0,0.2000,0.3869,5,2,477,0.0,1,389,1.0,1,763,1.0,jira.jira_assign_issue jira.jira_get_issue jira.jira_update_issue jira.jira_transition_issue jira.jira_create_issue jira.jira_add_comment jira.jira_add_worklog jira.jira_create_sprint jira.jira_list_projects jira.jira_search_issues +q003,Send a Slack message,verb_product,slack.slack_post_message,1.0,1.0,1.0,1.0000,1.0000,1,1,232,1.0,1,356,1.0,1,707,1.0,slack.slack_post_message slack.slack_add_reaction slack.slack_join_channel slack.slack_leave_channel slack.slack_create_channel slack.slack_search_messages slack.slack_upload_file slack.slack_list_users slack.slack_get_users slack.slack_list_channels +q004,List open pull requests,resource,github.list_pull_requests,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,429,1.0,1,1035,1.0,github.list_pull_requests github.search_pull_requests gitlab.gitlab_list_merge_requests github.merge_pull_request github.pull_request_read github.create_pull_request github.update_pull_request github.update_pull_request_branch github.pull_request_review_write bitbucket.bitbucket_list_prs +q005,Open a pull request on GitHub,verb_product,github.create_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,368,1.0,1,605,1.0,1,1069,1.0,github.create_pull_request github.merge_pull_request github.pull_request_read github.update_pull_request github.update_pull_request_branch github.pull_request_review_write github.request_copilot_review github.search_pull_requests github.list_pull_requests gitlab.gitlab_get_merge_request +q006,Run a SQL query against postgres,verb_product,postgres.query,1.0,1.0,1.0,1.0000,1.0000,1,1,167,1.0,1,280,1.0,1,590,1.0,postgres.query postgres.explain_query postgres.list_schemas postgres.list_tables postgres.describe_table mysql.mysql_query sqlite.sqlite_query salesforce.salesforce_query prometheus.prom_query docker.docker_run +q007,Read a local file from disk,paraphrase,filesystem.read_file,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,385,1.0,1,782,1.0,filesystem.read_file filesystem.read_text_file filesystem.read_media_file filesystem.write_file filesystem.edit_file filesystem.get_file_info filesystem.move_file brave_search.brave_local_search github.delete_file gitlab.gitlab_get_file +q008,What files are in this folder,paraphrase,filesystem.search_files,0.0,0.0,0.0,0.1667,0.4348,6,2,473,0.0,2,730,0.0,1,708,1.0,filesystem.search_files google_drive.gdrive_list_folder slack.list_files github.push_files filesystem.move_file filesystem.list_directory filesystem.list_directory_with_sizes filesystem.directory_tree filesystem.read_file filesystem.read_text_file +q009,Remember this fact in the knowledge graph,paraphrase,memory.read_graph,0.0,1.0,1.0,0.5000,0.5339,2,1,154,1.0,1,261,1.0,1,524,1.0,memory.read_graph memory.create_entities memory.delete_relations memory.create_relations memory.add_observations memory.open_nodes memory.search_nodes memory.delete_entities memory.delete_observations time.get_current_time +q010,Search the web with Brave,verb_product,brave_search.brave_web_search,1.0,1.0,1.0,1.0000,1.0000,1,1,245,1.0,1,412,1.0,1,726,1.0,brave_search.brave_web_search brave_search.brave_local_search filesystem.list_directory_with_sizes jira.jira_search_issues github.update_pull_request_branch memory.search_nodes puppeteer.puppeteer_screenshot puppeteer.puppeteer_navigate puppeteer.puppeteer_click puppeteer.puppeteer_fill +q011,Fetch the contents of a URL,verb_product,fetch.fetch,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,372,1.0,1,707,1.0,fetch.fetch github.get_file_contents playwright.browser_navigate filesystem.read_file puppeteer.puppeteer_navigate git.git_add slack.slack_set_channel_topic github.get_repository_tree git.git_status playwright.browser_snapshot +q012,Show git status,identifier,git.git_status,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,304,1.0,1,600,1.0,git.git_status git.git_show git.git_diff git.git_checkout git.git_branch git.git_commit git.git_add git.git_log git.git_diff_unstaged git.git_diff_staged +q013,Commit staged changes,paraphrase,git.git_commit,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,328,1.0,1,663,1.0,git.git_commit github.get_commit git.git_diff_staged git.git_reset git.git_log git.git_show github.push_files +q014,Create a Linear issue,verb_product,linear.linear_create_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,340,1.0,1,463,1.0,1,765,1.0,linear.linear_create_issue linear.linear_update_issue linear.linear_assign_issue linear.linear_get_issue linear.linear_create_comment linear.linear_create_project linear.linear_list_teams linear.linear_list_projects linear.linear_list_issues linear.linear_search_issues +q015,Make a task in my project tracker Linear,paraphrase,linear.linear_create_project,0.0,1.0,1.0,0.5000,0.6309,2,1,352,1.0,1,432,1.0,1,769,1.0,linear.linear_create_project linear.linear_create_issue linear.linear_update_issue linear.linear_list_teams linear.linear_list_projects linear.linear_list_issues linear.linear_search_issues linear.linear_create_comment linear.linear_assign_issue linear.linear_get_issue +q016,Post a comment on a GitHub issue,verb_product,github.update_issue_comment,0.0,1.0,1.0,0.5000,0.6309,2,1,333,1.0,1,573,1.0,1,1067,1.0,github.update_issue_comment github.add_issue_comment github.issue_read github.issue_write github.assign_copilot_to_issue github.create_issue figma.figma_post_comment github.add_comment_to_pending_review linear.linear_create_comment jira.jira_add_comment +q017,Merge the pull request,ambiguous,github.merge_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,283,1.0,1,523,1.0,1,1121,1.0,github.merge_pull_request gitlab.gitlab_merge_merge_request gitlab.gitlab_get_merge_request github.pull_request_read github.create_pull_request github.update_pull_request_branch github.update_pull_request gitlab.gitlab_create_merge_request github.pull_request_review_write github.request_copilot_review +q018,Trigger a GitHub Actions workflow,verb_product,github.actions_run_trigger,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,458,1.0,1,825,1.0,github.actions_run_trigger github.actions_get github.actions_list github.actions_get_job_logs circleci.circleci_trigger_pipeline github.create_milestone github.get_milestone github.list_milestones github.update_milestone github.delete_milestone +q019,Get Dependabot alerts,resource,github.list_dependabot_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,287,1.0,1,445,1.0,1,798,1.0,github.list_dependabot_alerts github.get_dependabot_alert github.list_code_scanning_alerts github.list_secret_scanning_alerts kubernetes.k8s_get_pod redis.redis_get github.get_discussion github.projects_get github.get_label aws_s3.s3_get_object +q020,List Sentry issues,verb_product,sentry.sentry_list_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,396,1.0,1,796,1.0,sentry.sentry_list_issues sentry.sentry_list_projects sentry.sentry_list_events sentry.sentry_get_issue sentry.sentry_update_issue github.list_issues github.search_issues gitlab.gitlab_list_issues linear.linear_list_issues linear.linear_search_issues +q021,Create a Stripe customer,verb_product,stripe.stripe_get_customer,0.0,1.0,1.0,0.5000,0.6309,2,1,186,1.0,1,300,1.0,1,669,1.0,stripe.stripe_get_customer stripe.stripe_create_customer stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_list_subscriptions stripe.stripe_create_product stripe.stripe_create_price stripe.stripe_create_refund stripe.stripe_create_payment_intent stripe.stripe_list_customers +q022,Refund a Stripe payment,paraphrase,stripe.stripe_create_refund,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,352,1.0,1,662,1.0,stripe.stripe_create_refund stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_list_customers stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_create_product stripe.stripe_create_price +q023,Send an email with Gmail,verb_product,gmail.gmail_send_email,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,389,1.0,1,703,1.0,gmail.gmail_send_email gmail.gmail_get_email gmail.gmail_trash_email gmail.gmail_modify_email gmail.gmail_search_emails gmail.gmail_create_draft gmail.gmail_list_labels filesystem.list_directory_with_sizes twilio.twilio_send_sms puppeteer.puppeteer_click +q024,Search my Gmail inbox,verb_product,gmail.gmail_search_emails,1.0,1.0,1.0,1.0000,1.0000,1,1,205,1.0,1,352,1.0,1,714,1.0,gmail.gmail_search_emails gmail.gmail_create_draft gmail.gmail_list_labels gmail.gmail_send_email gmail.gmail_get_email gmail.gmail_trash_email gmail.gmail_modify_email brave_search.brave_web_search brave_search.brave_local_search mysql.mysql_list_tables +q025,Upload a file to Google Drive,paraphrase,google_drive.gdrive_update_file,0.0,0.0,1.0,0.2500,0.4307,4,2,496,0.0,1,418,1.0,1,791,1.0,google_drive.gdrive_update_file google_drive.gdrive_share_file google_drive.gdrive_get_file google_drive.gdrive_create_file google_drive.gdrive_search google_drive.gdrive_list_folder slack.slack_upload_file gitlab.gitlab_get_file slack.slack_reply_to_thread figma.figma_get_file +q026,Create a calendar event,resource,google_calendar.gcal_delete_event,0.0,1.0,1.0,0.5000,0.6309,2,1,300,1.0,1,427,1.0,1,895,1.0,google_calendar.gcal_delete_event google_calendar.gcal_create_event google_calendar.gcal_update_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars github.create_gist github.create_milestone github.create_hook github.create_environment github.create_package +q027,List Kubernetes pods,verb_product,kubernetes.k8s_list_pods,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,314,1.0,1,718,1.0,kubernetes.k8s_list_pods kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_get_pod kubernetes.k8s_apply kubernetes.k8s_delete kubernetes.k8s_scale_deployment kubernetes.k8s_pod_logs clickup.clickup_list_tasks +q028,Get logs from a k8s pod,verb_product,kubernetes.k8s_pod_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,252,1.0,1,407,1.0,1,732,1.0,kubernetes.k8s_pod_logs kubernetes.k8s_get_pod kubernetes.k8s_get_events kubernetes.k8s_apply kubernetes.k8s_delete kubernetes.k8s_list_deployments kubernetes.k8s_scale_deployment kubernetes.k8s_list_services kubernetes.k8s_list_pods docker.docker_logs +q029,Run a docker container,verb_product,docker.docker_run,1.0,1.0,1.0,1.0000,1.0000,1,1,219,1.0,1,345,1.0,1,644,1.0,docker.docker_run docker.docker_stop docker.docker_rm docker.docker_exec docker.docker_logs docker.docker_images docker.docker_build docker.docker_ps github.create_check_run github.get_check_run +q030,Show docker container logs,verb_product,docker.docker_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,345,1.0,1,686,1.0,docker.docker_logs docker.docker_stop docker.docker_rm docker.docker_run docker.docker_exec docker.docker_images docker.docker_build docker.docker_ps kubernetes.k8s_pod_logs datadog.datadog_list_logs +q031,Navigate the browser to a URL,ambiguous,playwright.browser_navigate,1.0,1.0,1.0,1.0000,0.9502,1,1,180,1.0,1,265,1.0,1,536,1.0,playwright.browser_navigate playwright.browser_tab_new puppeteer.puppeteer_navigate playwright.browser_tab_close playwright.browser_tab_list playwright.browser_click playwright.browser_evaluate playwright.browser_snapshot playwright.browser_press_key playwright.browser_wait_for +q032,Take a screenshot of the page,ambiguous,playwright.browser_take_screenshot,1.0,1.0,1.0,1.0000,1.0000,1,1,226,1.0,1,352,1.0,1,711,1.0,playwright.browser_take_screenshot puppeteer.puppeteer_screenshot notion.notion_get_page notion.notion_update_page confluence.confluence_get_page confluence.confluence_create_page confluence.confluence_update_page notion.notion_create_page playwright.browser_snapshot playwright.browser_press_key +q033,Create a GitLab merge request,verb_product,gitlab.gitlab_merge_merge_request,0.0,1.0,1.0,0.3333,0.5000,3,1,297,1.0,1,475,1.0,1,833,1.0,gitlab.gitlab_merge_merge_request gitlab.gitlab_get_merge_request gitlab.gitlab_create_merge_request gitlab.gitlab_list_merge_requests github.merge_pull_request gitlab.gitlab_create_issue gitlab.gitlab_create_pipeline gitlab.gitlab_list_issues gitlab.gitlab_get_issue gitlab.gitlab_list_projects +q034,Search Notion pages,verb_product,notion.notion_search,1.0,1.0,1.0,1.0000,1.0000,1,1,195,1.0,1,366,1.0,1,686,1.0,notion.notion_search notion.notion_get_page notion.notion_get_block notion.notion_query_database notion.notion_create_database notion.notion_update_page notion.notion_list_users notion.notion_get_comments notion.notion_append_block notion.notion_create_page +q035,Create a Notion page,verb_product,notion.notion_get_page,0.0,1.0,1.0,0.3333,0.5000,3,1,230,1.0,1,371,1.0,1,687,1.0,notion.notion_get_page notion.notion_update_page notion.notion_create_page notion.notion_get_comments notion.notion_create_database notion.notion_search notion.notion_get_block notion.notion_query_database notion.notion_list_users notion.notion_append_block +q036,Query a Notion database,verb_product,notion.notion_query_database,1.0,1.0,1.0,1.0000,1.0000,1,1,279,1.0,1,401,1.0,1,687,1.0,notion.notion_query_database notion.notion_create_database notion.notion_create_page notion.notion_search notion.notion_get_page notion.notion_get_block notion.notion_update_page notion.notion_list_users notion.notion_get_comments notion.notion_append_block +q037,Create a Zendesk ticket,verb_product,zendesk.zendesk_search,0.0,1.0,1.0,0.5000,0.6309,2,1,224,1.0,1,368,1.0,1,896,1.0,zendesk.zendesk_search zendesk.zendesk_create_ticket zendesk.zendesk_get_ticket zendesk.zendesk_list_tickets zendesk.zendesk_update_ticket github.create_issue github.issue_read bitbucket.bitbucket_create_issue github.issue_write sentry.sentry_get_issue +q038,Create a HubSpot contact,verb_product,hubspot.hubspot_create_contact,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,832,1.0,hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_create_deal hubspot.hubspot_list_contacts hubspot.hubspot_search_contacts github.create_gist github.create_milestone github.create_hook github.create_environment github.create_package +q039,Create a Salesforce lead,verb_product,salesforce.salesforce_convert_lead,0.0,1.0,1.0,0.5000,0.6309,2,1,252,1.0,1,402,1.0,1,787,1.0,salesforce.salesforce_convert_lead salesforce.salesforce_create_lead salesforce.salesforce_create_record salesforce.salesforce_query salesforce.salesforce_update_record salesforce.salesforce_delete_record salesforce.salesforce_list_opportunities salesforce.salesforce_get_record github.create_gist github.create_milestone +q040,Run a SOQL query,identifier,salesforce.salesforce_query,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,291,1.0,1,724,1.0,salesforce.salesforce_query postgres.query sqlite.sqlite_query mysql.mysql_query prometheus.prom_query docker.docker_run postgres.explain_query prometheus.prom_query_range datadog.datadog_query_metrics notion.notion_query_database +q041,List Shopify orders,verb_product,shopify.shopify_list_orders,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,354,1.0,1,682,1.0,shopify.shopify_list_orders shopify.shopify_list_products shopify.shopify_list_customers shopify.shopify_get_product shopify.shopify_create_product shopify.shopify_update_product shopify.shopify_get_order shopify.shopify_create_order clickup.clickup_list_tasks trello.trello_list_cards +q042,Create a PagerDuty incident,verb_product,pagerduty.pagerduty_create_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,259,1.0,1,403,1.0,1,838,1.0,pagerduty.pagerduty_create_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls github.create_gist github.create_milestone github.create_hook github.create_environment +q043,Who is on call in PagerDuty,paraphrase,pagerduty.pagerduty_list_incidents,0.0,0.0,0.0,0.1667,0.3562,6,2,490,0.0,2,792,0.0,1,769,1.0,pagerduty.pagerduty_list_incidents pagerduty.pagerduty_get_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_oncalls filesystem.list_allowed_directories slack.slack_create_channel elasticsearch.es_index_doc github.create_milestone +q044,Query Datadog metrics,verb_product,datadog.datadog_query_metrics,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,458,1.0,1,728,1.0,datadog.datadog_query_metrics datadog.datadog_list_monitors datadog.datadog_create_monitor datadog.datadog_list_logs datadog.datadog_get_monitor prometheus.prom_list_metrics prometheus.prom_query postgres.query postgres.explain_query salesforce.salesforce_query +q045,Search Datadog logs,verb_product,datadog.datadog_list_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,458,1.0,1,877,1.0,datadog.datadog_list_logs datadog.datadog_list_monitors datadog.datadog_get_monitor datadog.datadog_create_monitor datadog.datadog_query_metrics docker.docker_logs kubernetes.k8s_pod_logs github.actions_get_job_logs brave_search.brave_web_search brave_search.brave_local_search +q046,List S3 buckets,verb_product,aws_s3.s3_list_buckets,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,318,1.0,1,667,1.0,aws_s3.s3_list_buckets aws_s3.s3_list_objects clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments kubernetes.k8s_list_services github.projects_list github.list_milestones github.list_hooks github.list_environments +q047,Upload an object to S3,paraphrase,aws_s3.s3_put_object,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,437,1.0,1,950,1.0,aws_s3.s3_put_object aws_s3.s3_delete_object aws_s3.s3_get_object slack.slack_upload_file github.assign_copilot_to_issue slack.slack_reply_to_thread github.add_comment_to_pending_review slack.slack_add_reaction twilio.twilio_send_sms gmail.gmail_send_email +q048,Purge Cloudflare cache,verb_product,cloudflare.cf_purge_cache,1.0,1.0,1.0,1.0000,1.0000,1,1,231,1.0,1,340,1.0,1,746,1.0,cloudflare.cf_purge_cache cloudflare.cf_list_zones cloudflare.cf_create_dns cloudflare.cf_list_dns +q049,Create a Cloudflare DNS record,verb_product,cloudflare.cf_create_dns,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,356,1.0,1,738,1.0,cloudflare.cf_create_dns cloudflare.cf_list_dns cloudflare.cf_list_zones cloudflare.cf_purge_cache salesforce.salesforce_create_record salesforce.salesforce_update_record salesforce.salesforce_delete_record salesforce.salesforce_get_record git.git_commit github.create_gist +q050,List Vercel deployments,verb_product,vercel.vercel_list_deployments,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,291,1.0,1,579,1.0,vercel.vercel_list_deployments vercel.vercel_list_projects vercel.vercel_get_deployment vercel.vercel_create_project kubernetes.k8s_list_deployments clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_services aws_s3.s3_list_buckets github.projects_list +q051,Insert a row into Supabase,verb_product,supabase.supabase_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,238,1.0,1,355,1.0,1,665,1.0,supabase.supabase_insert supabase.supabase_select supabase.supabase_update supabase.supabase_invoke_fn supabase.supabase_list_tables mongodb.mongo_insert playwright.browser_type +q052,Export images from Figma,paraphrase,figma.figma_get_images,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,368,1.0,1,658,1.0,figma.figma_get_images figma.figma_get_file figma.figma_get_file_nodes figma.figma_get_comments figma.figma_post_comment docker.docker_images elasticsearch.es_get_doc aws_s3.s3_get_object memory.delete_observations docker.docker_logs +q053,Send an SMS with Twilio,verb_product,twilio.twilio_send_sms,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,386,1.0,1,751,1.0,twilio.twilio_send_sms twilio.twilio_list_messages twilio.twilio_make_call discord.discord_send_message slack.slack_post_message filesystem.list_directory_with_sizes gmail.gmail_send_email puppeteer.puppeteer_click puppeteer.puppeteer_fill slack.slack_add_reaction +q054,Create an Okta user,verb_product,okta.okta_get_user,0.0,1.0,1.0,0.3333,0.5000,3,1,183,1.0,1,329,1.0,1,656,1.0,okta.okta_get_user okta.okta_deactivate_user okta.okta_create_user okta.okta_list_users linear.create_user linear.get_user linear.update_user linear.delete_user slack.slack_get_user_profile asana.asana_create_task +q055,Instant query Prometheus,verb_product,prometheus.prom_query,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,329,1.0,1,680,1.0,prometheus.prom_query prometheus.prom_query_range prometheus.prom_list_metrics postgres.query postgres.explain_query salesforce.salesforce_query sqlite.sqlite_query mysql.mysql_query datadog.datadog_query_metrics notion.notion_query_database +q056,Search Grafana dashboards,verb_product,grafana.grafana_search_dashboards,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,382,1.0,1,690,1.0,grafana.grafana_search_dashboards grafana.grafana_get_dashboard grafana.grafana_create_annotation brave_search.brave_web_search brave_search.brave_local_search confluence.confluence_search github.search_issues github.search_users github.search_orgs elasticsearch.es_search +q057,Trigger a CircleCI pipeline,verb_product,circleci.circleci_trigger_pipeline,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,417,1.0,1,755,1.0,circleci.circleci_trigger_pipeline circleci.circleci_get_pipeline circleci.circleci_list_pipelines github.actions_run_trigger gitlab.create_pipeline gitlab.get_pipeline gitlab.update_pipeline gitlab.delete_pipeline gitlab.gitlab_create_pipeline hubspot.hubspot_create_deal +q058,Run terraform plan,identifier,terraform.tf_plan,1.0,1.0,1.0,1.0000,1.0000,1,1,172,1.0,1,320,1.0,1,717,1.0,terraform.tf_plan terraform.tf_apply terraform.tf_state_list postgres.explain_query docker.docker_run github.create_check_run github.get_check_run github.update_check_run github.delete_check_run github.actions_run_trigger +q059,Create an Asana task,verb_product,asana.asana_get_task,0.0,1.0,1.0,0.5000,0.6309,2,1,267,1.0,1,405,1.0,1,825,1.0,asana.asana_get_task asana.asana_create_task asana.asana_update_task asana.asana_create_project asana.asana_list_tasks clickup.clickup_get_task clickup.clickup_create_task clickup.clickup_update_task okta.okta_create_user github.create_issue +q060,Create a Trello card,verb_product,trello.trello_create_card,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,394,1.0,1,884,1.0,trello.trello_create_card trello.trello_add_comment trello.trello_create_board trello.trello_list_cards github.create_gist github.create_milestone github.create_hook github.create_environment github.create_package github.create_artifact +q061,Create a ClickUp task,verb_product,clickup.clickup_get_task,0.0,1.0,1.0,0.5000,0.6309,2,1,257,1.0,1,362,1.0,1,834,1.0,clickup.clickup_get_task clickup.clickup_create_task clickup.clickup_update_task clickup.clickup_list_tasks asana.asana_get_task asana.asana_create_task asana.asana_update_task github.create_gist github.create_milestone github.create_hook +q062,Create a Confluence page,verb_product,confluence.confluence_create_page,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,367,1.0,1,703,1.0,confluence.confluence_create_page confluence.confluence_get_page confluence.confluence_update_page confluence.confluence_search notion.notion_get_page notion.notion_create_page notion.notion_update_page playwright.browser_take_screenshot notion.notion_get_comments playwright.browser_press_key +q063,Send a Discord message,verb_product,discord.discord_send_message,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,322,1.0,1,682,1.0,discord.discord_send_message discord.discord_add_reaction discord.discord_get_channel discord.discord_list_channels discord.discord_list_members slack.slack_post_message gmail.gmail_send_email twilio.twilio_send_sms gmail.gmail_get_email gmail.gmail_trash_email +q064,Search Elasticsearch,verb_product,elasticsearch.es_search,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,404,1.0,1,699,1.0,elasticsearch.es_search elasticsearch.es_index_doc elasticsearch.es_get_doc brave_search.brave_web_search brave_search.brave_local_search confluence.confluence_search github.search_issues github.search_users github.search_orgs github.search_commits +q065,Get a Redis key,verb_product,redis.redis_get,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,280,1.0,1,589,1.0,redis.redis_get redis.redis_set redis.redis_delete redis.redis_keys playwright.browser_press_key jira.jira_get_issue aws_s3.s3_get_object figma.figma_get_file figma.figma_get_comments figma.figma_get_file_nodes +q066,Find MongoDB documents,verb_product,mongodb.mongo_find,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,381,1.0,1,738,1.0,mongodb.mongo_find mongodb.mongo_insert mongodb.mongo_update mongodb.mongo_delete mongodb.mongo_list_collections +q067,List SQLite tables,verb_product,sqlite.sqlite_list_tables,1.0,1.0,1.0,1.0000,1.0000,1,1,156,1.0,1,252,1.0,1,514,1.0,sqlite.sqlite_list_tables sqlite.sqlite_query sqlite.sqlite_describe_table postgres.list_tables mysql.mysql_list_tables supabase.supabase_list_tables postgres.list_schemas clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments +q068,Describe a postgres table,verb_product,postgres.describe_table,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,272,1.0,1,597,1.0,postgres.describe_table sqlite.sqlite_describe_table postgres.query postgres.list_tables postgres.list_schemas postgres.explain_query supabase.supabase_select supabase.supabase_insert supabase.supabase_update +q069,github_create_issue,identifier,github.create_issue,1.0,1.0,1.0,1.0000,0.9502,1,1,386,1.0,1,596,1.0,1,1019,1.0,github.create_issue github.issue_read github.issue_write github.add_issue_comment github.update_issue_comment github.assign_copilot_to_issue bitbucket.bitbucket_create_issue sentry.sentry_get_issue gitlab.gitlab_create_issue jira.jira_assign_issue +q070,createIssue,identifier,,0.0,0.0,0.0,0.0000,0.0000,29,10,2752,0.0,6,2680,0.0,3,2626,0.0, +q071,list-directory,identifier,filesystem.create_directory,0.0,1.0,1.0,0.5000,0.6309,2,1,182,1.0,1,296,1.0,1,607,1.0,filesystem.create_directory filesystem.list_directory filesystem.directory_tree filesystem.list_directory_with_sizes filesystem.get_file_info filesystem.list_allowed_directories filesystem.read_file filesystem.read_text_file filesystem.read_media_file filesystem.write_file +q072,sequential thinking next thought,paraphrase,sequential_thinking.sequentialthinking,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,382,1.0,1,797,1.0,sequential_thinking.sequentialthinking +q073,What time is it in Tokyo,paraphrase,time.convert_time,1.0,1.0,1.0,1.0000,0.8597,1,1,217,1.0,1,329,1.0,1,783,1.0,time.convert_time time.get_current_time playwright.browser_wait_for filesystem.list_allowed_directories slack.slack_create_channel google_calendar.gcal_list_events grafana.grafana_create_annotation jira.jira_add_worklog elasticsearch.es_index_doc github.create_milestone +q074,Convert time between timezones,verb_product,time.convert_time,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,335,1.0,1,704,1.0,time.convert_time time.get_current_time salesforce.salesforce_convert_lead playwright.browser_wait_for git.git_diff memory.create_relations google_calendar.gcal_list_events grafana.grafana_create_annotation jira.jira_add_worklog +q075,Star a GitHub repo,verb_product,github.star_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,388,1.0,1,810,1.0,github.star_repository github.delete_repository github.fork_repository github.unstar_repository github.create_repository github.list_repository_collaborators github.get_repository_tree github.list_dependabot_alerts github.create_milestone github.get_milestone +q076,Fork a repository,resource,github.fork_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,235,1.0,1,386,1.0,1,792,1.0,github.fork_repository github.delete_repository github.star_repository github.unstar_repository github.create_repository github.list_repository_collaborators github.get_repository_tree github.list_dependabot_alerts github.list_branches github.list_releases +q077,Get the latest GitHub release,verb_product,github.get_latest_release,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,405,1.0,1,667,1.0,github.get_latest_release github.get_release_by_tag gitlab.get_release gitlab.create_release gitlab.update_release gitlab.delete_release github.get_milestone github.get_hook github.get_environment github.get_package +q078,Create a gist,resource,github.create_gist,1.0,1.0,1.0,1.0000,1.0000,1,1,229,1.0,1,429,1.0,1,912,1.0,github.create_gist github.update_gist github.get_gist github.create_milestone github.create_hook github.create_environment github.create_package github.create_artifact github.create_runner gitlab.create_pipeline +q079,Search GitHub code,verb_product,github.search_code,1.0,1.0,1.0,1.0000,1.0000,1,1,283,1.0,1,419,1.0,1,773,1.0,github.search_code github.get_code_scanning_alert github.list_code_scanning_alerts github.search_issues github.search_users github.search_orgs github.search_commits github.search_repositories github.search_pull_requests github.create_milestone +q080,List GitHub notifications,verb_product,github.list_notifications,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,354,1.0,1,764,1.0,github.list_notifications github.mark_all_notifications_read github.list_milestones github.list_hooks github.list_environments github.list_packages github.list_artifacts github.list_runners github.list_check_runs github.create_milestone +q081,Assign Copilot to an issue,verb_product,github.assign_copilot_to_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,434,1.0,1,928,1.0,github.assign_copilot_to_issue jira.jira_assign_issue linear.linear_assign_issue github.request_copilot_review jira.jira_update_issue jira.jira_transition_issue github.create_issue github.add_issue_comment slack.slack_reply_to_thread github.issue_read +q082,Get secret scanning alerts,resource,github.list_secret_scanning_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,295,1.0,1,470,1.0,1,830,1.0,github.list_secret_scanning_alerts github.get_secret_scanning_alert github.list_code_scanning_alerts github.list_dependabot_alerts github.get_code_scanning_alert kubernetes.k8s_get_pod redis.redis_get github.get_discussion github.projects_get github.get_label +q083,Create or update a file on GitHub,verb_product,github.create_or_update_file,1.0,1.0,1.0,1.0000,0.7602,1,1,390,1.0,1,557,1.0,1,922,1.0,github.create_or_update_file github.delete_file github.get_file_contents github.get_file_blame filesystem.write_file slack.update_file google_drive.gdrive_get_file slack.create_file google_drive.gdrive_update_file figma.figma_get_file +q084,List collaborators on a repo,resource,github.list_repository_collaborators,1.0,1.0,1.0,1.0000,1.0000,1,1,248,1.0,1,386,1.0,1,813,1.0,github.list_repository_collaborators github.delete_repository github.fork_repository github.star_repository github.unstar_repository github.create_repository github.get_repository_tree github.list_commits trello.trello_list_cards pagerduty.pagerduty_list_oncalls +q085,Reply in a Slack thread,verb_product,slack.slack_reply_to_thread,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,358,1.0,1,717,1.0,slack.slack_reply_to_thread slack.slack_get_thread_replies slack.slack_list_channels slack.slack_join_channel slack.slack_leave_channel slack.slack_create_channel slack.slack_search_messages slack.slack_upload_file slack.slack_list_users slack.slack_post_message +q086,Add an emoji reaction on Slack,verb_product,slack.slack_add_reaction,1.0,1.0,1.0,1.0000,0.7602,1,1,216,1.0,1,357,1.0,1,709,1.0,slack.slack_add_reaction slack.slack_join_channel slack.slack_leave_channel slack.slack_create_channel slack.slack_search_messages slack.slack_upload_file slack.slack_list_users slack.slack_post_message slack.slack_get_users slack.slack_list_channels +q087,Search Slack messages,verb_product,slack.slack_search_messages,1.0,1.0,1.0,1.0000,1.0000,1,1,213,1.0,1,337,1.0,1,689,1.0,slack.slack_search_messages slack.slack_get_channel_history slack.slack_join_channel slack.slack_leave_channel slack.slack_create_channel slack.slack_upload_file slack.slack_list_users slack.slack_post_message slack.slack_get_users slack.slack_list_channels +q088,Create a Slack channel,verb_product,slack.slack_join_channel,0.0,1.0,1.0,0.3333,0.5000,3,1,196,1.0,1,342,1.0,1,719,1.0,slack.slack_join_channel slack.slack_leave_channel slack.slack_create_channel slack.slack_get_channel_history slack.slack_set_channel_topic slack.slack_post_message slack.slack_upload_file slack.slack_add_reaction slack.slack_search_messages slack.slack_list_users +q089,Transition a Jira issue,verb_product,jira.jira_transition_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,389,1.0,1,763,1.0,jira.jira_transition_issue jira.jira_assign_issue jira.jira_get_issue jira.jira_update_issue jira.jira_create_issue jira.jira_add_comment jira.jira_add_worklog jira.jira_list_projects jira.jira_create_sprint jira.jira_search_issues +q090,Search Jira with JQL,identifier,jira.jira_search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,194,1.0,1,370,1.0,1,764,1.0,jira.jira_search_issues jira.jira_assign_issue jira.jira_list_projects jira.jira_create_sprint jira.jira_update_issue jira.jira_get_issue jira.jira_add_comment jira.jira_add_worklog jira.jira_transition_issue jira.jira_create_issue +q091,Add a worklog to Jira,verb_product,jira.jira_add_worklog,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,350,1.0,1,765,1.0,jira.jira_add_worklog jira.jira_add_comment jira.jira_transition_issue jira.jira_assign_issue jira.jira_list_projects jira.jira_create_sprint jira.jira_update_issue jira.jira_get_issue jira.jira_search_issues jira.jira_create_issue +q092,List Linear teams,verb_product,linear.linear_list_teams,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,380,1.0,1,764,1.0,linear.linear_list_teams linear.linear_list_projects linear.linear_list_issues linear.linear_update_issue linear.linear_create_project linear.linear_create_issue linear.linear_search_issues linear.linear_create_comment linear.linear_assign_issue linear.linear_get_issue +q093,Comment on a Linear issue,verb_product,linear.linear_create_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,334,1.0,1,464,1.0,1,760,1.0,linear.linear_create_comment linear.linear_update_issue linear.linear_create_issue linear.linear_assign_issue linear.linear_get_issue linear.create_comment linear.get_comment linear.update_comment linear.delete_comment linear.linear_list_teams +q094,Move a local file,paraphrase,filesystem.move_file,1.0,1.0,1.0,1.0000,1.0000,1,1,214,1.0,1,354,1.0,1,699,1.0,filesystem.move_file brave_search.brave_local_search filesystem.read_file filesystem.read_text_file figma.figma_get_file filesystem.read_media_file filesystem.write_file slack.create_file slack.get_file slack.update_file +q095,Recursively search files by pattern,paraphrase,filesystem.search_files,1.0,1.0,1.0,1.0000,1.0000,1,1,280,1.0,1,433,1.0,1,705,1.0,filesystem.search_files slack.list_files github.push_files github.get_release_by_tag filesystem.move_file filesystem.list_directory filesystem.list_directory_with_sizes filesystem.directory_tree redis.redis_keys filesystem.read_file +q096,Edit a file in place with old and new string,paraphrase,filesystem.edit_file,1.0,1.0,1.0,1.0000,1.0000,1,1,227,1.0,1,394,1.0,1,682,1.0,filesystem.edit_file playwright.browser_tab_new filesystem.list_directory_with_sizes filesystem.write_file slack.create_file slack.get_file slack.update_file slack.delete_file figma.figma_get_file filesystem.move_file +q097,Read the whole knowledge graph,paraphrase,memory.read_graph,1.0,1.0,1.0,1.0000,1.0000,1,1,152,1.0,1,259,1.0,1,529,1.0,memory.read_graph memory.delete_relations memory.create_entities memory.create_relations memory.add_observations memory.delete_entities memory.open_nodes memory.search_nodes memory.delete_observations filesystem.read_file +q098,Search memory nodes,verb_product,memory.search_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,202,1.0,1,301,1.0,1,546,1.0,memory.search_nodes memory.open_nodes figma.figma_get_file_nodes memory.create_entities memory.create_relations memory.add_observations memory.delete_entities memory.delete_observations memory.delete_relations memory.read_graph +q099,Show unstaged git diff,verb_product,git.git_diff_unstaged,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,303,1.0,1,602,1.0,git.git_diff_unstaged git.git_diff git.git_diff_staged git.git_show git.git_status git.git_checkout git.git_branch git.git_commit git.git_add git.git_log +q100,Stage files with git add,identifier,filesystem.search_files,0.0,1.0,1.0,0.3333,0.5000,3,1,213,1.0,1,336,1.0,1,677,1.0,filesystem.search_files filesystem.list_directory_with_sizes git.git_add git.git_checkout git.git_show git.git_branch git.git_diff git.git_commit git.git_status github.push_files +q101,Create a Shopify product,verb_product,shopify.shopify_create_product,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,380,1.0,1,702,1.0,shopify.shopify_create_product shopify.shopify_get_product shopify.shopify_update_product shopify.shopify_create_order shopify.shopify_list_products shopify.shopify_list_orders shopify.shopify_get_order shopify.shopify_list_customers stripe.stripe_create_product stripe.stripe_create_price +q102,Cancel a Stripe subscription,verb_product,stripe.stripe_cancel_subscription,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,301,1.0,1,632,1.0,stripe.stripe_cancel_subscription stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_list_customers stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_create_product stripe.stripe_create_price stripe.stripe_list_subscriptions stripe.stripe_create_refund +q103,Create a payment intent,resource,stripe.stripe_create_payment_intent,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,417,1.0,1,897,1.0,stripe.stripe_create_payment_intent stripe.stripe_get_payment_intent stripe.stripe_create_refund github.create_gist github.create_milestone github.create_hook github.create_environment github.create_package github.create_artifact github.create_runner +q104,Acknowledge a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,261,1.0,1,405,1.0,1,717,1.0,pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_incidents pagerduty.pagerduty_list_oncalls +q105,Create a Datadog monitor,verb_product,datadog.datadog_get_monitor,0.0,1.0,1.0,0.5000,0.6309,2,1,259,1.0,1,459,1.0,1,927,1.0,datadog.datadog_get_monitor datadog.datadog_create_monitor datadog.datadog_list_monitors datadog.datadog_list_logs datadog.datadog_query_metrics github.create_gist github.create_milestone github.create_hook github.create_environment github.create_package +q106,Resolve a Sentry issue,paraphrase,sentry.sentry_get_issue,0.0,1.0,1.0,0.5000,0.6309,2,1,251,1.0,1,397,1.0,1,817,1.0,sentry.sentry_get_issue sentry.sentry_update_issue sentry.sentry_list_events sentry.sentry_list_projects sentry.sentry_list_issues pagerduty.pagerduty_resolve_incident github.issue_read jira.jira_assign_issue github.issue_write jira.jira_get_issue +q107,Scale a Kubernetes deployment,verb_product,kubernetes.k8s_scale_deployment,1.0,1.0,1.0,1.0000,1.0000,1,1,263,1.0,1,407,1.0,1,726,1.0,kubernetes.k8s_scale_deployment kubernetes.k8s_get_pod kubernetes.k8s_apply kubernetes.k8s_delete kubernetes.k8s_list_deployments kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_list_pods kubernetes.k8s_pod_logs vercel.vercel_get_deployment +q108,Apply a Kubernetes manifest,verb_product,kubernetes.k8s_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,253,1.0,1,407,1.0,1,727,1.0,kubernetes.k8s_apply kubernetes.k8s_get_pod kubernetes.k8s_delete kubernetes.k8s_list_deployments kubernetes.k8s_scale_deployment kubernetes.k8s_list_services kubernetes.k8s_get_events kubernetes.k8s_list_pods kubernetes.k8s_pod_logs terraform.tf_apply +q109,Stop a docker container,verb_product,docker.docker_stop,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,345,1.0,1,641,1.0,docker.docker_stop docker.docker_rm docker.docker_logs docker.docker_run docker.docker_exec docker.docker_images docker.docker_build docker.docker_ps kubernetes.k8s_pod_logs +q110,Build a docker image,verb_product,docker.docker_build,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,317,1.0,1,601,1.0,docker.docker_build docker.docker_run docker.docker_stop docker.docker_rm docker.docker_images docker.docker_ps docker.docker_logs docker.docker_exec filesystem.read_media_file +q111,Click an element in Playwright,verb_product,playwright.browser_click,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,329,1.0,1,566,1.0,playwright.browser_click puppeteer.puppeteer_click playwright.browser_type playwright.browser_take_screenshot playwright.browser_snapshot playwright.browser_navigate playwright.browser_fill_form playwright.browser_evaluate playwright.browser_tab_list playwright.browser_tab_new +q112,Type text into a browser input,paraphrase,playwright.browser_type,1.0,1.0,1.0,1.0000,0.7602,1,1,217,1.0,1,302,1.0,1,554,1.0,playwright.browser_type playwright.browser_navigate playwright.browser_wait_for playwright.browser_tab_close playwright.browser_tab_list playwright.browser_tab_new playwright.browser_snapshot playwright.browser_click playwright.browser_evaluate playwright.browser_fill_form +q113,List GitLab projects,verb_product,gitlab.gitlab_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,398,1.0,1,817,1.0,gitlab.gitlab_list_projects gitlab.gitlab_list_issues gitlab.gitlab_create_issue gitlab.gitlab_get_issue gitlab.gitlab_get_file gitlab.gitlab_create_pipeline gitlab.gitlab_list_merge_requests gitlab.gitlab_create_merge_request gitlab.gitlab_get_merge_request gitlab.gitlab_merge_merge_request +q114,Create a Bitbucket pull request,verb_product,bitbucket.bitbucket_create_pr,1.0,1.0,1.0,1.0000,1.0000,1,1,313,1.0,1,471,1.0,1,1091,1.0,bitbucket.bitbucket_create_pr bitbucket.bitbucket_list_prs bitbucket.bitbucket_create_issue bitbucket.bitbucket_list_repos github.merge_pull_request github.pull_request_read github.create_pull_request github.update_pull_request github.update_pull_request_branch github.pull_request_review_write +q115,Create a Gmail draft,verb_product,gmail.gmail_create_draft,1.0,1.0,1.0,1.0000,1.0000,1,1,205,1.0,1,352,1.0,1,835,1.0,gmail.gmail_create_draft gmail.gmail_search_emails gmail.gmail_list_labels gmail.gmail_send_email gmail.gmail_get_email gmail.gmail_trash_email gmail.gmail_modify_email github.create_pull_request github.create_gist github.create_milestone +q116,Share a Google Drive file,verb_product,google_drive.gdrive_share_file,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,417,1.0,1,726,1.0,google_drive.gdrive_share_file google_drive.gdrive_update_file google_drive.gdrive_get_file google_drive.gdrive_create_file google_drive.gdrive_search google_drive.gdrive_list_folder gitlab.gitlab_get_file figma.figma_get_file google_calendar.gcal_list_calendars filesystem.write_file +q117,List Google Calendar events,verb_product,google_calendar.gcal_list_events,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,427,1.0,1,784,1.0,google_calendar.gcal_list_events google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event google_calendar.gcal_list_calendars kubernetes.k8s_get_events sentry.sentry_list_events google_drive.gdrive_list_folder google_drive.gdrive_update_file google_drive.gdrive_share_file +q118,Delete a calendar event,resource,google_calendar.gcal_delete_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,427,1.0,1,751,1.0,google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars kubernetes.k8s_delete redis.redis_delete aws_s3.s3_delete_object github.delete_repository github.delete_milestone +q119,List Vercel projects,verb_product,vercel.vercel_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,301,1.0,1,550,1.0,vercel.vercel_list_projects vercel.vercel_list_deployments vercel.vercel_get_deployment vercel.vercel_create_project github.projects_list linear.linear_list_projects jira.jira_list_projects sentry.sentry_list_projects gitlab.gitlab_list_projects github.projects_get +q120,Select rows from Supabase,verb_product,supabase.supabase_select,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,354,1.0,1,709,1.0,supabase.supabase_select supabase.supabase_insert supabase.supabase_update supabase.supabase_invoke_fn supabase.supabase_list_tables elasticsearch.es_get_doc aws_s3.s3_get_object memory.delete_observations docker.docker_logs kubernetes.k8s_pod_logs +q121,Post a Figma comment,verb_product,figma.figma_post_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,368,1.0,1,727,1.0,figma.figma_post_comment figma.figma_get_file figma.figma_get_comments figma.figma_get_images figma.figma_get_file_nodes linear.create_comment linear.get_comment linear.update_comment linear.delete_comment notion.create_comment +q122,List Okta users,verb_product,okta.okta_list_users,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,306,1.0,1,606,1.0,okta.okta_list_users okta.okta_get_user okta.okta_create_user okta.okta_deactivate_user linear.list_users github.search_users slack.slack_list_users notion.notion_list_users notion.list_users_tokens slack.slack_get_users +q123,Range query Prometheus,verb_product,prometheus.prom_query_range,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,329,1.0,1,680,1.0,prometheus.prom_query_range prometheus.prom_query prometheus.prom_list_metrics postgres.query postgres.explain_query salesforce.salesforce_query sqlite.sqlite_query mysql.mysql_query datadog.datadog_query_metrics notion.notion_query_database +q124,Create a Grafana annotation,verb_product,grafana.grafana_create_annotation,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,401,1.0,1,881,1.0,grafana.grafana_create_annotation grafana.grafana_search_dashboards grafana.grafana_get_dashboard github.create_gist github.create_milestone github.create_hook github.create_environment github.create_package github.create_artifact github.create_runner +q125,Apply terraform,identifier,terraform.tf_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,171,1.0,1,292,1.0,1,698,1.0,terraform.tf_apply terraform.tf_plan terraform.tf_state_list kubernetes.k8s_apply +q126,List Asana tasks,verb_product,asana.asana_list_tasks,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,404,1.0,1,682,1.0,asana.asana_list_tasks asana.asana_create_task asana.asana_get_task asana.asana_update_task asana.asana_create_project clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments kubernetes.k8s_list_services aws_s3.s3_list_buckets +q127,Comment on a Trello card,verb_product,trello.trello_add_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,260,1.0,1,422,1.0,1,757,1.0,trello.trello_add_comment trello.trello_create_card trello.trello_list_cards trello.trello_create_board github.update_issue_comment figma.figma_post_comment linear.create_comment linear.get_comment linear.update_comment linear.delete_comment +q128,Update a ClickUp task,verb_product,clickup.clickup_update_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,399,1.0,1,795,1.0,clickup.clickup_update_task clickup.clickup_get_task clickup.clickup_create_task clickup.clickup_list_tasks asana.asana_update_task asana.asana_get_task asana.asana_create_task mongodb.mongo_update github.update_gist github.update_milestone +q129,Search Confluence,verb_product,confluence.confluence_search,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,401,1.0,1,731,1.0,confluence.confluence_search confluence.confluence_get_page confluence.confluence_create_page confluence.confluence_update_page brave_search.brave_web_search brave_search.brave_local_search github.search_issues github.search_users github.search_orgs elasticsearch.es_search +q130,Create a HubSpot deal,verb_product,hubspot.hubspot_create_deal,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,832,1.0,hubspot.hubspot_create_deal hubspot.hubspot_create_contact hubspot.hubspot_get_contact hubspot.hubspot_list_contacts hubspot.hubspot_search_contacts github.create_gist github.create_milestone github.create_hook github.create_environment github.create_package +q131,List Zendesk tickets,verb_product,zendesk.zendesk_list_tickets,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,367,1.0,1,645,1.0,zendesk.zendesk_list_tickets zendesk.zendesk_search zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_create_ticket clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments kubernetes.k8s_list_services aws_s3.s3_list_buckets +q132,List Discord channels,verb_product,discord.discord_list_channels,1.0,1.0,1.0,1.0000,1.0000,1,1,166,1.0,1,321,1.0,1,625,1.0,discord.discord_list_channels discord.discord_list_members discord.discord_get_channel discord.discord_send_message discord.discord_add_reaction slack.slack_list_channels clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments kubernetes.k8s_list_services +q133,Set a Redis key,verb_product,redis.redis_set,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,280,1.0,1,600,1.0,redis.redis_set redis.redis_get redis.redis_delete redis.redis_keys playwright.browser_press_key jira.jira_get_issue slack.slack_set_channel_topic aws_s3.s3_get_object aws_s3.s3_delete_object figma.figma_get_file +q134,Insert MongoDB documents,verb_product,mongodb.mongo_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,256,1.0,1,399,1.0,1,698,1.0,mongodb.mongo_insert mongodb.mongo_update mongodb.mongo_delete mongodb.mongo_find supabase.supabase_insert mongodb.mongo_list_collections +q135,Run a MySQL query,verb_product,mysql.mysql_query,1.0,1.0,1.0,1.0000,1.0000,1,1,174,1.0,1,289,1.0,1,686,1.0,mysql.mysql_query mysql.mysql_list_tables postgres.query postgres.explain_query salesforce.salesforce_query sqlite.sqlite_query prometheus.prom_query docker.docker_run prometheus.prom_query_range datadog.datadog_query_metrics +q136,Index a document in Elasticsearch,verb_product,elasticsearch.es_index_doc,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,396,1.0,1,753,1.0,elasticsearch.es_index_doc elasticsearch.es_search elasticsearch.es_get_doc playwright.browser_tab_close github.create_milestone github.get_milestone github.list_milestones github.update_milestone github.delete_milestone github.create_hook +q137,Get an S3 object,verb_product,aws_s3.s3_get_object,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,355,1.0,1,706,1.0,aws_s3.s3_get_object aws_s3.s3_delete_object aws_s3.s3_put_object okta.okta_get_user asana.asana_get_task aws_s3.s3_list_objects elasticsearch.es_search okta.okta_create_user okta.okta_deactivate_user asana.asana_create_task +q138,List Cloudflare zones,verb_product,cloudflare.cf_list_zones,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,338,1.0,1,629,1.0,cloudflare.cf_list_zones cloudflare.cf_purge_cache cloudflare.cf_list_dns cloudflare.cf_create_dns clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments kubernetes.k8s_list_services aws_s3.s3_list_buckets github.projects_list +q139,Create a Stripe invoice,verb_product,stripe.stripe_create_invoice,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,353,1.0,1,632,1.0,stripe.stripe_create_invoice stripe.stripe_create_customer stripe.stripe_create_product stripe.stripe_create_price stripe.stripe_create_refund stripe.stripe_get_customer stripe.stripe_list_customers stripe.stripe_list_invoices stripe.stripe_list_subscriptions stripe.stripe_cancel_subscription +q140,List GitHub Actions workflows,verb_product,github.actions_list,1.0,1.0,1.0,1.0000,1.0000,1,1,293,1.0,1,468,1.0,1,845,1.0,github.actions_list github.actions_get github.actions_run_trigger github.actions_get_job_logs github.list_milestones github.list_hooks github.list_environments github.list_packages github.list_artifacts github.list_runners +q141,Get GitHub Actions job logs,verb_product,github.actions_get_job_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,442,1.0,1,822,1.0,github.actions_get_job_logs github.actions_get github.actions_list github.actions_run_trigger gitlab.get_job kubernetes.k8s_pod_logs docker.docker_logs gitlab.create_job gitlab.update_job gitlab.delete_job +q142,Create a GitHub repository,verb_product,github.create_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,388,1.0,1,867,1.0,github.create_repository github.delete_repository github.fork_repository github.star_repository github.unstar_repository github.list_repository_collaborators github.get_repository_tree github.create_milestone github.create_hook github.create_environment +q143,Checkout a git branch,verb_product,git.git_checkout,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,373,1.0,1,666,1.0,git.git_checkout git.git_create_branch git.git_branch git.git_show github.create_branch git.git_diff git.git_commit git.git_status git.git_add git.git_reset +q144,Show git log,identifier,git.git_log,1.0,1.0,1.0,1.0000,1.0000,1,1,206,1.0,1,320,1.0,1,600,1.0,git.git_log git.git_show git.git_diff git.git_status git.git_checkout git.git_branch git.git_commit git.git_add git.git_diff_unstaged git.git_diff_staged +q145,List postgres schemas,verb_product,postgres.list_schemas,1.0,1.0,1.0,1.0000,1.0000,1,1,158,1.0,1,277,1.0,1,555,1.0,postgres.list_schemas postgres.list_tables postgres.query postgres.describe_table postgres.explain_query clickup.clickup_list_tasks trello.trello_list_cards kubernetes.k8s_list_deployments kubernetes.k8s_list_services aws_s3.s3_list_buckets +q146,Open nodes in the knowledge graph,verb_product,memory.open_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,163,1.0,1,260,1.0,1,549,1.0,memory.open_nodes memory.read_graph memory.search_nodes memory.create_entities memory.delete_relations memory.create_relations memory.add_observations memory.delete_entities memory.delete_observations figma.figma_get_file_nodes +q147,Get file metadata from the filesystem,paraphrase,filesystem.read_file,0.0,0.0,1.0,0.2500,0.4307,4,2,462,0.0,1,348,1.0,1,688,1.0,filesystem.read_file filesystem.read_text_file filesystem.read_media_file filesystem.get_file_info filesystem.write_file filesystem.edit_file filesystem.move_file filesystem.list_allowed_directories google_drive.gdrive_get_file gitlab.gitlab_get_file +q148,Write a new local file,paraphrase,playwright.browser_tab_new,0.0,1.0,1.0,0.5000,0.6309,2,1,217,1.0,1,366,1.0,1,714,1.0,playwright.browser_tab_new filesystem.write_file brave_search.brave_local_search filesystem.edit_file filesystem.read_file filesystem.read_text_file figma.figma_get_file filesystem.read_media_file slack.create_file slack.get_file +q149,Search GitHub issues,verb_product,github.search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,263,1.0,1,407,1.0,1,742,1.0,github.search_issues github.list_issues linear.linear_search_issues jira.jira_search_issues gitlab.gitlab_list_issues linear.linear_list_issues sentry.sentry_list_issues github.search_users github.search_orgs github.search_commits +q150,Get the authenticated GitHub user,paraphrase,github.get_teams,0.0,0.0,0.0,0.1111,0.3010,9,3,585,0.0,2,628,0.0,1,606,1.0,github.get_teams linear.get_user okta.okta_get_user okta.okta_deactivate_user slack.slack_get_user_profile linear.create_user linear.update_user linear.delete_user github.get_me github.list_notifications diff --git a/experiments/mcp-tool-search/results/query_rewrite_weighted_bm25.json b/experiments/mcp-tool-search/results/query_rewrite_weighted_bm25.json new file mode 100644 index 00000000..4fe25c27 --- /dev/null +++ b/experiments/mcp-tool-search/results/query_rewrite_weighted_bm25.json @@ -0,0 +1,6958 @@ +{ + "summary": { + "n": 150, + "index_s": 0.00289805900013107, + "rss_mb": 0.0, + "p50_ms": 0.4016589999196185, + "p95_ms": 0.5639099500399425, + "recall@1": 0.8333333333333334, + "recall@1_lo": 0.7733333333333333, + "recall@1_hi": 0.8933333333333333, + "recall@3": 0.9533333333333334, + "recall@3_lo": 0.9133333333333333, + "recall@3_hi": 0.9866666666666667, + "recall@5": 0.9733333333333334, + "recall@5_lo": 0.9466666666666667, + "recall@5_hi": 0.9933333333333333, + "mrr@10": 0.8954074074074073, + "mrr@10_lo": 0.8554074074074074, + "mrr@10_hi": 0.9333333333333333, + "ndcg@10": 0.9129082726821834, + "ndcg@10_lo": 0.8808897532481432, + "ndcg@10_hi": 0.9425391523882402 + }, + "meta": { + "name": "query_rewrite_weighted_bm25", + "needs_model": false, + "needs_api": false, + "notes": "tiny synonym rewrite then weighted BM25 (not an LLM)" + }, + "rows": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "github.issue_read", + "github.issue_write", + "github.add_issue_comment", + "github.update_issue_comment", + "github.assign_copilot_to_issue", + "bitbucket.bitbucket_create_issue", + "sentry.sentry_get_issue", + "gitlab.gitlab_create_issue", + "jira.jira_assign_issue" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 388, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 598, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1021, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "kind": "verb_product", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_assign_issue", + "jira.jira_get_issue", + "jira.jira_update_issue", + "jira.jira_transition_issue", + "jira.jira_create_issue", + "jira.jira_add_comment", + "jira.jira_add_worklog", + "jira.jira_create_sprint", + "jira.jira_list_projects", + "jira.jira_search_issues" + ], + "hit1": "jira.jira_assign_issue", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.2, + "ndcg@10": 0.38685280723454163, + "hit_rank": 5, + "k3_calls": 2, + "k3_tokens": 477, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "kind": "verb_product", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "ranking": [ + "slack.slack_post_message", + "slack.slack_add_reaction", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_create_channel", + "slack.slack_search_messages", + "slack.slack_upload_file", + "slack.slack_list_users", + "slack.slack_get_users", + "slack.slack_list_channels" + ], + "hit1": "slack.slack_post_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "kind": "resource", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "ranking": [ + "github.list_pull_requests", + "github.search_pull_requests", + "gitlab.gitlab_list_merge_requests", + "github.merge_pull_request", + "github.pull_request_read", + "github.create_pull_request", + "github.update_pull_request", + "github.update_pull_request_branch", + "github.pull_request_review_write", + "bitbucket.bitbucket_list_prs" + ], + "hit1": "github.list_pull_requests", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1035, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "ranking": [ + "github.create_pull_request", + "github.merge_pull_request", + "github.pull_request_read", + "github.update_pull_request", + "github.update_pull_request_branch", + "github.pull_request_review_write", + "github.request_copilot_review", + "github.search_pull_requests", + "github.list_pull_requests", + "gitlab.gitlab_get_merge_request" + ], + "hit1": "github.create_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 368, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 605, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1069, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "kind": "verb_product", + "gold": [ + "postgres.query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "postgres.explain_query", + "postgres.list_schemas", + "postgres.list_tables", + "postgres.describe_table", + "mysql.mysql_query", + "sqlite.sqlite_query", + "salesforce.salesforce_query", + "prometheus.prom_query", + "docker.docker_run" + ], + "hit1": "postgres.query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 167, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 590, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "kind": "paraphrase", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "ranking": [ + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "filesystem.write_file", + "filesystem.edit_file", + "filesystem.get_file_info", + "filesystem.move_file", + "brave_search.brave_local_search", + "github.delete_file", + "gitlab.gitlab_get_file" + ], + "hit1": "filesystem.read_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 385, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 782, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "kind": "paraphrase", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "ranking": [ + "filesystem.search_files", + "google_drive.gdrive_list_folder", + "slack.list_files", + "github.push_files", + "filesystem.move_file", + "filesystem.list_directory", + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree", + "filesystem.read_file", + "filesystem.read_text_file" + ], + "hit1": "filesystem.search_files", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.16666666666666666, + "ndcg@10": 0.43476305489842704, + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 473, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 730, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 708, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "ranking": [ + "memory.read_graph", + "memory.create_entities", + "memory.delete_relations", + "memory.create_relations", + "memory.add_observations", + "memory.open_nodes", + "memory.search_nodes", + "memory.delete_entities", + "memory.delete_observations", + "time.get_current_time" + ], + "hit1": "memory.read_graph", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.5338931479009518, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 154, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 261, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 524, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "kind": "verb_product", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "ranking": [ + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "filesystem.list_directory_with_sizes", + "jira.jira_search_issues", + "github.update_pull_request_branch", + "memory.search_nodes", + "puppeteer.puppeteer_screenshot", + "puppeteer.puppeteer_navigate", + "puppeteer.puppeteer_click", + "puppeteer.puppeteer_fill" + ], + "hit1": "brave_search.brave_web_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 245, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 412, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "kind": "verb_product", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "ranking": [ + "fetch.fetch", + "github.get_file_contents", + "playwright.browser_navigate", + "filesystem.read_file", + "puppeteer.puppeteer_navigate", + "git.git_add", + "slack.slack_set_channel_topic", + "github.get_repository_tree", + "git.git_status", + "playwright.browser_snapshot" + ], + "hit1": "fetch.fetch", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 372, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "kind": "identifier", + "gold": [ + "git.git_status" + ], + "alts": [], + "ranking": [ + "git.git_status", + "git.git_show", + "git.git_diff", + "git.git_checkout", + "git.git_branch", + "git.git_commit", + "git.git_add", + "git.git_log", + "git.git_diff_unstaged", + "git.git_diff_staged" + ], + "hit1": "git.git_status", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 304, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "kind": "paraphrase", + "gold": [ + "git.git_commit" + ], + "alts": [], + "ranking": [ + "git.git_commit", + "github.get_commit", + "git.git_diff_staged", + "git.git_reset", + "git.git_log", + "git.git_show", + "github.push_files" + ], + "hit1": "git.git_commit", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_issue", + "linear.linear_update_issue", + "linear.linear_assign_issue", + "linear.linear_get_issue", + "linear.linear_create_comment", + "linear.linear_create_project", + "linear.linear_list_teams", + "linear.linear_list_projects", + "linear.linear_list_issues", + "linear.linear_search_issues" + ], + "hit1": "linear.linear_create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 340, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "kind": "paraphrase", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_project", + "linear.linear_create_issue", + "linear.linear_update_issue", + "linear.linear_list_teams", + "linear.linear_list_projects", + "linear.linear_list_issues", + "linear.linear_search_issues", + "linear.linear_create_comment", + "linear.linear_assign_issue", + "linear.linear_get_issue" + ], + "hit1": "linear.linear_create_project", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 352, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 432, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 769, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "ranking": [ + "github.update_issue_comment", + "github.add_issue_comment", + "github.issue_read", + "github.issue_write", + "github.assign_copilot_to_issue", + "github.create_issue", + "figma.figma_post_comment", + "github.add_comment_to_pending_review", + "linear.linear_create_comment", + "jira.jira_add_comment" + ], + "hit1": "github.update_issue_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 573, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1067, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "kind": "ambiguous", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "ranking": [ + "github.merge_pull_request", + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_get_merge_request", + "github.pull_request_read", + "github.create_pull_request", + "github.update_pull_request_branch", + "github.update_pull_request", + "gitlab.gitlab_create_merge_request", + "github.pull_request_review_write", + "github.request_copilot_review" + ], + "hit1": "github.merge_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 523, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1121, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "kind": "verb_product", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "ranking": [ + "github.actions_run_trigger", + "github.actions_get", + "github.actions_list", + "github.actions_get_job_logs", + "circleci.circleci_trigger_pipeline", + "github.create_milestone", + "github.get_milestone", + "github.list_milestones", + "github.update_milestone", + "github.delete_milestone" + ], + "hit1": "github.actions_run_trigger", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "kind": "resource", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "ranking": [ + "github.list_dependabot_alerts", + "github.get_dependabot_alert", + "github.list_code_scanning_alerts", + "github.list_secret_scanning_alerts", + "kubernetes.k8s_get_pod", + "redis.redis_get", + "github.get_discussion", + "github.projects_get", + "github.get_label", + "aws_s3.s3_get_object" + ], + "hit1": "github.list_dependabot_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 287, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 445, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 798, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "kind": "verb_product", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_issues", + "sentry.sentry_list_projects", + "sentry.sentry_list_events", + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "github.list_issues", + "github.search_issues", + "gitlab.gitlab_list_issues", + "linear.linear_list_issues", + "linear.linear_search_issues" + ], + "hit1": "sentry.sentry_list_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "ranking": [ + "stripe.stripe_get_customer", + "stripe.stripe_create_customer", + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_list_subscriptions", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "stripe.stripe_create_refund", + "stripe.stripe_create_payment_intent", + "stripe.stripe_list_customers" + ], + "hit1": "stripe.stripe_get_customer", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "kind": "paraphrase", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_refund", + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_list_customers", + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_create_product", + "stripe.stripe_create_price" + ], + "hit1": "stripe.stripe_create_refund", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "kind": "verb_product", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "ranking": [ + "gmail.gmail_send_email", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "gmail.gmail_modify_email", + "gmail.gmail_search_emails", + "gmail.gmail_create_draft", + "gmail.gmail_list_labels", + "filesystem.list_directory_with_sizes", + "twilio.twilio_send_sms", + "puppeteer.puppeteer_click" + ], + "hit1": "gmail.gmail_send_email", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "kind": "verb_product", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "ranking": [ + "gmail.gmail_search_emails", + "gmail.gmail_create_draft", + "gmail.gmail_list_labels", + "gmail.gmail_send_email", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "gmail.gmail_modify_email", + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "mysql.mysql_list_tables" + ], + "hit1": "gmail.gmail_search_emails", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "kind": "paraphrase", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_update_file", + "google_drive.gdrive_share_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "slack.slack_upload_file", + "gitlab.gitlab_get_file", + "slack.slack_reply_to_thread", + "figma.figma_get_file" + ], + "hit1": "google_drive.gdrive_update_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 496, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 418, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 791, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment", + "github.create_package" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 895, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_list_pods", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_apply", + "kubernetes.k8s_delete", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_pod_logs", + "clickup.clickup_list_tasks" + ], + "hit1": "kubernetes.k8s_list_pods", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 314, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_get_events", + "kubernetes.k8s_apply", + "kubernetes.k8s_delete", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_list_services", + "kubernetes.k8s_list_pods", + "docker.docker_logs" + ], + "hit1": "kubernetes.k8s_pod_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 732, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "ranking": [ + "docker.docker_run", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_exec", + "docker.docker_logs", + "docker.docker_images", + "docker.docker_build", + "docker.docker_ps", + "github.create_check_run", + "github.get_check_run" + ], + "hit1": "docker.docker_run", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 219, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 644, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "kind": "verb_product", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "ranking": [ + "docker.docker_logs", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_run", + "docker.docker_exec", + "docker.docker_images", + "docker.docker_build", + "docker.docker_ps", + "kubernetes.k8s_pod_logs", + "datadog.datadog_list_logs" + ], + "hit1": "docker.docker_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "kind": "ambiguous", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "ranking": [ + "playwright.browser_navigate", + "playwright.browser_tab_new", + "puppeteer.puppeteer_navigate", + "playwright.browser_tab_close", + "playwright.browser_tab_list", + "playwright.browser_click", + "playwright.browser_evaluate", + "playwright.browser_snapshot", + "playwright.browser_press_key", + "playwright.browser_wait_for" + ], + "hit1": "playwright.browser_navigate", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 265, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 536, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "kind": "ambiguous", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "ranking": [ + "playwright.browser_take_screenshot", + "puppeteer.puppeteer_screenshot", + "notion.notion_get_page", + "notion.notion_update_page", + "confluence.confluence_get_page", + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "notion.notion_create_page", + "playwright.browser_snapshot", + "playwright.browser_press_key" + ], + "hit1": "playwright.browser_take_screenshot", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 711, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_get_merge_request", + "gitlab.gitlab_create_merge_request", + "gitlab.gitlab_list_merge_requests", + "github.merge_pull_request", + "gitlab.gitlab_create_issue", + "gitlab.gitlab_create_pipeline", + "gitlab.gitlab_list_issues", + "gitlab.gitlab_get_issue", + "gitlab.gitlab_list_projects" + ], + "hit1": "gitlab.gitlab_merge_merge_request", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 833, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "kind": "verb_product", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "ranking": [ + "notion.notion_search", + "notion.notion_get_page", + "notion.notion_get_block", + "notion.notion_query_database", + "notion.notion_create_database", + "notion.notion_update_page", + "notion.notion_list_users", + "notion.notion_get_comments", + "notion.notion_append_block", + "notion.notion_create_page" + ], + "hit1": "notion.notion_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 195, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "kind": "verb_product", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "ranking": [ + "notion.notion_get_page", + "notion.notion_update_page", + "notion.notion_create_page", + "notion.notion_get_comments", + "notion.notion_create_database", + "notion.notion_search", + "notion.notion_get_block", + "notion.notion_query_database", + "notion.notion_list_users", + "notion.notion_append_block" + ], + "hit1": "notion.notion_get_page", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 371, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "kind": "verb_product", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "ranking": [ + "notion.notion_query_database", + "notion.notion_create_database", + "notion.notion_create_page", + "notion.notion_search", + "notion.notion_get_page", + "notion.notion_get_block", + "notion.notion_update_page", + "notion.notion_list_users", + "notion.notion_get_comments", + "notion.notion_append_block" + ], + "hit1": "notion.notion_query_database", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 279, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_search", + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_update_ticket", + "github.create_issue", + "github.issue_read", + "bitbucket.bitbucket_create_issue", + "github.issue_write", + "sentry.sentry_get_issue" + ], + "hit1": "zendesk.zendesk_search", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 896, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_create_deal", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_search_contacts", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment", + "github.create_package" + ], + "hit1": "hubspot.hubspot_create_contact", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 832, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "kind": "verb_product", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_convert_lead", + "salesforce.salesforce_create_lead", + "salesforce.salesforce_create_record", + "salesforce.salesforce_query", + "salesforce.salesforce_update_record", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_list_opportunities", + "salesforce.salesforce_get_record", + "github.create_gist", + "github.create_milestone" + ], + "hit1": "salesforce.salesforce_convert_lead", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 402, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 787, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "kind": "identifier", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_query", + "postgres.query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "prometheus.prom_query", + "docker.docker_run", + "postgres.explain_query", + "prometheus.prom_query_range", + "datadog.datadog_query_metrics", + "notion.notion_query_database" + ], + "hit1": "salesforce.salesforce_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 724, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "kind": "verb_product", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "ranking": [ + "shopify.shopify_list_orders", + "shopify.shopify_list_products", + "shopify.shopify_list_customers", + "shopify.shopify_get_product", + "shopify.shopify_create_product", + "shopify.shopify_update_product", + "shopify.shopify_get_order", + "shopify.shopify_create_order", + "clickup.clickup_list_tasks", + "trello.trello_list_cards" + ], + "hit1": "shopify.shopify_list_orders", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment" + ], + "hit1": "pagerduty.pagerduty_create_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 403, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 838, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "kind": "paraphrase", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_oncalls", + "filesystem.list_allowed_directories", + "slack.slack_create_channel", + "elasticsearch.es_index_doc", + "github.create_milestone" + ], + "hit1": "pagerduty.pagerduty_list_incidents", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.16666666666666666, + "ndcg@10": 0.3562071871080222, + "hit_rank": 6, + "k3_calls": 2, + "k3_tokens": 490, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 792, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 769, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "kind": "verb_product", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "ranking": [ + "datadog.datadog_query_metrics", + "datadog.datadog_list_monitors", + "datadog.datadog_create_monitor", + "datadog.datadog_list_logs", + "datadog.datadog_get_monitor", + "prometheus.prom_list_metrics", + "prometheus.prom_query", + "postgres.query", + "postgres.explain_query", + "salesforce.salesforce_query" + ], + "hit1": "datadog.datadog_query_metrics", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 728, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "kind": "verb_product", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "ranking": [ + "datadog.datadog_list_logs", + "datadog.datadog_list_monitors", + "datadog.datadog_get_monitor", + "datadog.datadog_create_monitor", + "datadog.datadog_query_metrics", + "docker.docker_logs", + "kubernetes.k8s_pod_logs", + "github.actions_get_job_logs", + "brave_search.brave_web_search", + "brave_search.brave_local_search" + ], + "hit1": "datadog.datadog_list_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 877, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_list_buckets", + "aws_s3.s3_list_objects", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "github.projects_list", + "github.list_milestones", + "github.list_hooks", + "github.list_environments" + ], + "hit1": "aws_s3.s3_list_buckets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 318, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "kind": "paraphrase", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_put_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_get_object", + "slack.slack_upload_file", + "github.assign_copilot_to_issue", + "slack.slack_reply_to_thread", + "github.add_comment_to_pending_review", + "slack.slack_add_reaction", + "twilio.twilio_send_sms", + "gmail.gmail_send_email" + ], + "hit1": "aws_s3.s3_put_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 950, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns" + ], + "hit1": "cloudflare.cf_purge_cache", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "cloudflare.cf_list_zones", + "cloudflare.cf_purge_cache", + "salesforce.salesforce_create_record", + "salesforce.salesforce_update_record", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_get_record", + "git.git_commit", + "github.create_gist" + ], + "hit1": "cloudflare.cf_create_dns", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 356, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_deployments", + "vercel.vercel_list_projects", + "vercel.vercel_get_deployment", + "vercel.vercel_create_project", + "kubernetes.k8s_list_deployments", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_services", + "aws_s3.s3_list_buckets", + "github.projects_list" + ], + "hit1": "vercel.vercel_list_deployments", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 579, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "ranking": [ + "supabase.supabase_insert", + "supabase.supabase_select", + "supabase.supabase_update", + "supabase.supabase_invoke_fn", + "supabase.supabase_list_tables", + "mongodb.mongo_insert", + "playwright.browser_type" + ], + "hit1": "supabase.supabase_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 238, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "kind": "paraphrase", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "ranking": [ + "figma.figma_get_images", + "figma.figma_get_file", + "figma.figma_get_file_nodes", + "figma.figma_get_comments", + "figma.figma_post_comment", + "docker.docker_images", + "elasticsearch.es_get_doc", + "aws_s3.s3_get_object", + "memory.delete_observations", + "docker.docker_logs" + ], + "hit1": "figma.figma_get_images", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "kind": "verb_product", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "ranking": [ + "twilio.twilio_send_sms", + "twilio.twilio_list_messages", + "twilio.twilio_make_call", + "discord.discord_send_message", + "slack.slack_post_message", + "filesystem.list_directory_with_sizes", + "gmail.gmail_send_email", + "puppeteer.puppeteer_click", + "puppeteer.puppeteer_fill", + "slack.slack_add_reaction" + ], + "hit1": "twilio.twilio_send_sms", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "kind": "verb_product", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "ranking": [ + "okta.okta_get_user", + "okta.okta_deactivate_user", + "okta.okta_create_user", + "okta.okta_list_users", + "linear.create_user", + "linear.get_user", + "linear.update_user", + "linear.delete_user", + "slack.slack_get_user_profile", + "asana.asana_create_task" + ], + "hit1": "okta.okta_get_user", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 656, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query", + "prometheus.prom_query_range", + "prometheus.prom_list_metrics", + "postgres.query", + "postgres.explain_query", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "datadog.datadog_query_metrics", + "notion.notion_query_database" + ], + "hit1": "prometheus.prom_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "kind": "verb_product", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "ranking": [ + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "grafana.grafana_create_annotation", + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "confluence.confluence_search", + "github.search_issues", + "github.search_users", + "github.search_orgs", + "elasticsearch.es_search" + ], + "hit1": "grafana.grafana_search_dashboards", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 690, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "kind": "verb_product", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "ranking": [ + "circleci.circleci_trigger_pipeline", + "circleci.circleci_get_pipeline", + "circleci.circleci_list_pipelines", + "github.actions_run_trigger", + "gitlab.create_pipeline", + "gitlab.get_pipeline", + "gitlab.update_pipeline", + "gitlab.delete_pipeline", + "gitlab.gitlab_create_pipeline", + "hubspot.hubspot_create_deal" + ], + "hit1": "circleci.circleci_trigger_pipeline", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "kind": "identifier", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "ranking": [ + "terraform.tf_plan", + "terraform.tf_apply", + "terraform.tf_state_list", + "postgres.explain_query", + "docker.docker_run", + "github.create_check_run", + "github.get_check_run", + "github.update_check_run", + "github.delete_check_run", + "github.actions_run_trigger" + ], + "hit1": "terraform.tf_plan", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "kind": "verb_product", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "ranking": [ + "asana.asana_get_task", + "asana.asana_create_task", + "asana.asana_update_task", + "asana.asana_create_project", + "asana.asana_list_tasks", + "clickup.clickup_get_task", + "clickup.clickup_create_task", + "clickup.clickup_update_task", + "okta.okta_create_user", + "github.create_issue" + ], + "hit1": "asana.asana_get_task", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "ranking": [ + "trello.trello_create_card", + "trello.trello_add_comment", + "trello.trello_create_board", + "trello.trello_list_cards", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment", + "github.create_package", + "github.create_artifact" + ], + "hit1": "trello.trello_create_card", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 394, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 884, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_get_task", + "clickup.clickup_create_task", + "clickup.clickup_update_task", + "clickup.clickup_list_tasks", + "asana.asana_get_task", + "asana.asana_create_task", + "asana.asana_update_task", + "github.create_gist", + "github.create_milestone", + "github.create_hook" + ], + "hit1": "clickup.clickup_get_task", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 834, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "kind": "verb_product", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "ranking": [ + "confluence.confluence_create_page", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "confluence.confluence_search", + "notion.notion_get_page", + "notion.notion_create_page", + "notion.notion_update_page", + "playwright.browser_take_screenshot", + "notion.notion_get_comments", + "playwright.browser_press_key" + ], + "hit1": "confluence.confluence_create_page", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "kind": "verb_product", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "discord.discord_add_reaction", + "discord.discord_get_channel", + "discord.discord_list_channels", + "discord.discord_list_members", + "slack.slack_post_message", + "gmail.gmail_send_email", + "twilio.twilio_send_sms", + "gmail.gmail_get_email", + "gmail.gmail_trash_email" + ], + "hit1": "discord.discord_send_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_search", + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "confluence.confluence_search", + "github.search_issues", + "github.search_users", + "github.search_orgs", + "github.search_commits" + ], + "hit1": "elasticsearch.es_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "redis.redis_set", + "redis.redis_delete", + "redis.redis_keys", + "playwright.browser_press_key", + "jira.jira_get_issue", + "aws_s3.s3_get_object", + "figma.figma_get_file", + "figma.figma_get_comments", + "figma.figma_get_file_nodes" + ], + "hit1": "redis.redis_get", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 589, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_find", + "mongodb.mongo_insert", + "mongodb.mongo_update", + "mongodb.mongo_delete", + "mongodb.mongo_list_collections" + ], + "hit1": "mongodb.mongo_find", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "kind": "verb_product", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "ranking": [ + "sqlite.sqlite_list_tables", + "sqlite.sqlite_query", + "sqlite.sqlite_describe_table", + "postgres.list_tables", + "mysql.mysql_list_tables", + "supabase.supabase_list_tables", + "postgres.list_schemas", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments" + ], + "hit1": "sqlite.sqlite_list_tables", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 156, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 252, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 514, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "kind": "verb_product", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "ranking": [ + "postgres.describe_table", + "sqlite.sqlite_describe_table", + "postgres.query", + "postgres.list_tables", + "postgres.list_schemas", + "postgres.explain_query", + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update" + ], + "hit1": "postgres.describe_table", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 272, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "github.create_issue", + "github.issue_read", + "github.issue_write", + "github.add_issue_comment", + "github.update_issue_comment", + "github.assign_copilot_to_issue", + "bitbucket.bitbucket_create_issue", + "sentry.sentry_get_issue", + "gitlab.gitlab_create_issue", + "jira.jira_assign_issue" + ], + "hit1": "github.create_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 386, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 596, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1019, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "ranking": [], + "hit1": null, + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 29, + "k3_calls": 10, + "k3_tokens": 2752, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 6, + "k5_tokens": 2680, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 3, + "k10_tokens": 2626, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "kind": "identifier", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "ranking": [ + "filesystem.create_directory", + "filesystem.list_directory", + "filesystem.directory_tree", + "filesystem.list_directory_with_sizes", + "filesystem.get_file_info", + "filesystem.list_allowed_directories", + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "filesystem.write_file" + ], + "hit1": "filesystem.create_directory", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 296, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 607, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "kind": "paraphrase", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "ranking": [ + "sequential_thinking.sequentialthinking" + ], + "hit1": "sequential_thinking.sequentialthinking", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "kind": "paraphrase", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "ranking": [ + "time.convert_time", + "time.get_current_time", + "playwright.browser_wait_for", + "filesystem.list_allowed_directories", + "slack.slack_create_channel", + "google_calendar.gcal_list_events", + "grafana.grafana_create_annotation", + "jira.jira_add_worklog", + "elasticsearch.es_index_doc", + "github.create_milestone" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.8597186998521972, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 783, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "kind": "verb_product", + "gold": [ + "time.convert_time" + ], + "alts": [], + "ranking": [ + "time.convert_time", + "time.get_current_time", + "salesforce.salesforce_convert_lead", + "playwright.browser_wait_for", + "git.git_diff", + "memory.create_relations", + "google_calendar.gcal_list_events", + "grafana.grafana_create_annotation", + "jira.jira_add_worklog" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "kind": "verb_product", + "gold": [ + "github.star_repository" + ], + "alts": [], + "ranking": [ + "github.star_repository", + "github.delete_repository", + "github.fork_repository", + "github.unstar_repository", + "github.create_repository", + "github.list_repository_collaborators", + "github.get_repository_tree", + "github.list_dependabot_alerts", + "github.create_milestone", + "github.get_milestone" + ], + "hit1": "github.star_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 810, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "kind": "resource", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "ranking": [ + "github.fork_repository", + "github.delete_repository", + "github.star_repository", + "github.unstar_repository", + "github.create_repository", + "github.list_repository_collaborators", + "github.get_repository_tree", + "github.list_dependabot_alerts", + "github.list_branches", + "github.list_releases" + ], + "hit1": "github.fork_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 235, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 792, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "kind": "verb_product", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "ranking": [ + "github.get_latest_release", + "github.get_release_by_tag", + "gitlab.get_release", + "gitlab.create_release", + "gitlab.update_release", + "gitlab.delete_release", + "github.get_milestone", + "github.get_hook", + "github.get_environment", + "github.get_package" + ], + "hit1": "github.get_latest_release", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 667, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "kind": "resource", + "gold": [ + "github.create_gist" + ], + "alts": [], + "ranking": [ + "github.create_gist", + "github.update_gist", + "github.get_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment", + "github.create_package", + "github.create_artifact", + "github.create_runner", + "gitlab.create_pipeline" + ], + "hit1": "github.create_gist", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 429, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 912, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "kind": "verb_product", + "gold": [ + "github.search_code" + ], + "alts": [], + "ranking": [ + "github.search_code", + "github.get_code_scanning_alert", + "github.list_code_scanning_alerts", + "github.search_issues", + "github.search_users", + "github.search_orgs", + "github.search_commits", + "github.search_repositories", + "github.search_pull_requests", + "github.create_milestone" + ], + "hit1": "github.search_code", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 419, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 773, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "kind": "verb_product", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "ranking": [ + "github.list_notifications", + "github.mark_all_notifications_read", + "github.list_milestones", + "github.list_hooks", + "github.list_environments", + "github.list_packages", + "github.list_artifacts", + "github.list_runners", + "github.list_check_runs", + "github.create_milestone" + ], + "hit1": "github.list_notifications", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "kind": "verb_product", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "ranking": [ + "github.assign_copilot_to_issue", + "jira.jira_assign_issue", + "linear.linear_assign_issue", + "github.request_copilot_review", + "jira.jira_update_issue", + "jira.jira_transition_issue", + "github.create_issue", + "github.add_issue_comment", + "slack.slack_reply_to_thread", + "github.issue_read" + ], + "hit1": "github.assign_copilot_to_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 434, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 928, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "kind": "resource", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "ranking": [ + "github.list_secret_scanning_alerts", + "github.get_secret_scanning_alert", + "github.list_code_scanning_alerts", + "github.list_dependabot_alerts", + "github.get_code_scanning_alert", + "kubernetes.k8s_get_pod", + "redis.redis_get", + "github.get_discussion", + "github.projects_get", + "github.get_label" + ], + "hit1": "github.list_secret_scanning_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 830, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "ranking": [ + "github.create_or_update_file", + "github.delete_file", + "github.get_file_contents", + "github.get_file_blame", + "filesystem.write_file", + "slack.update_file", + "google_drive.gdrive_get_file", + "slack.create_file", + "google_drive.gdrive_update_file", + "figma.figma_get_file" + ], + "hit1": "github.create_or_update_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 390, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 557, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 922, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "kind": "resource", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "ranking": [ + "github.list_repository_collaborators", + "github.delete_repository", + "github.fork_repository", + "github.star_repository", + "github.unstar_repository", + "github.create_repository", + "github.get_repository_tree", + "github.list_commits", + "trello.trello_list_cards", + "pagerduty.pagerduty_list_oncalls" + ], + "hit1": "github.list_repository_collaborators", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 248, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 813, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "kind": "verb_product", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "ranking": [ + "slack.slack_reply_to_thread", + "slack.slack_get_thread_replies", + "slack.slack_list_channels", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_create_channel", + "slack.slack_search_messages", + "slack.slack_upload_file", + "slack.slack_list_users", + "slack.slack_post_message" + ], + "hit1": "slack.slack_reply_to_thread", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 358, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "kind": "verb_product", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "ranking": [ + "slack.slack_add_reaction", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_create_channel", + "slack.slack_search_messages", + "slack.slack_upload_file", + "slack.slack_list_users", + "slack.slack_post_message", + "slack.slack_get_users", + "slack.slack_list_channels" + ], + "hit1": "slack.slack_add_reaction", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 357, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "kind": "verb_product", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "ranking": [ + "slack.slack_search_messages", + "slack.slack_get_channel_history", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_create_channel", + "slack.slack_upload_file", + "slack.slack_list_users", + "slack.slack_post_message", + "slack.slack_get_users", + "slack.slack_list_channels" + ], + "hit1": "slack.slack_search_messages", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 689, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "kind": "verb_product", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "ranking": [ + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_create_channel", + "slack.slack_get_channel_history", + "slack.slack_set_channel_topic", + "slack.slack_post_message", + "slack.slack_upload_file", + "slack.slack_add_reaction", + "slack.slack_search_messages", + "slack.slack_list_users" + ], + "hit1": "slack.slack_join_channel", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 719, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "kind": "verb_product", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_transition_issue", + "jira.jira_assign_issue", + "jira.jira_get_issue", + "jira.jira_update_issue", + "jira.jira_create_issue", + "jira.jira_add_comment", + "jira.jira_add_worklog", + "jira.jira_list_projects", + "jira.jira_create_sprint", + "jira.jira_search_issues" + ], + "hit1": "jira.jira_transition_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "kind": "identifier", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "ranking": [ + "jira.jira_search_issues", + "jira.jira_assign_issue", + "jira.jira_list_projects", + "jira.jira_create_sprint", + "jira.jira_update_issue", + "jira.jira_get_issue", + "jira.jira_add_comment", + "jira.jira_add_worklog", + "jira.jira_transition_issue", + "jira.jira_create_issue" + ], + "hit1": "jira.jira_search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "kind": "verb_product", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "ranking": [ + "jira.jira_add_worklog", + "jira.jira_add_comment", + "jira.jira_transition_issue", + "jira.jira_assign_issue", + "jira.jira_list_projects", + "jira.jira_create_sprint", + "jira.jira_update_issue", + "jira.jira_get_issue", + "jira.jira_search_issues", + "jira.jira_create_issue" + ], + "hit1": "jira.jira_add_worklog", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 765, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "kind": "verb_product", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "ranking": [ + "linear.linear_list_teams", + "linear.linear_list_projects", + "linear.linear_list_issues", + "linear.linear_update_issue", + "linear.linear_create_project", + "linear.linear_create_issue", + "linear.linear_search_issues", + "linear.linear_create_comment", + "linear.linear_assign_issue", + "linear.linear_get_issue" + ], + "hit1": "linear.linear_list_teams", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 764, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "ranking": [ + "linear.linear_create_comment", + "linear.linear_update_issue", + "linear.linear_create_issue", + "linear.linear_assign_issue", + "linear.linear_get_issue", + "linear.create_comment", + "linear.get_comment", + "linear.update_comment", + "linear.delete_comment", + "linear.linear_list_teams" + ], + "hit1": "linear.linear_create_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 334, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 464, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 760, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "kind": "paraphrase", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "ranking": [ + "filesystem.move_file", + "brave_search.brave_local_search", + "filesystem.read_file", + "filesystem.read_text_file", + "figma.figma_get_file", + "filesystem.read_media_file", + "filesystem.write_file", + "slack.create_file", + "slack.get_file", + "slack.update_file" + ], + "hit1": "filesystem.move_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 214, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "kind": "paraphrase", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "ranking": [ + "filesystem.search_files", + "slack.list_files", + "github.push_files", + "github.get_release_by_tag", + "filesystem.move_file", + "filesystem.list_directory", + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree", + "redis.redis_keys", + "filesystem.read_file" + ], + "hit1": "filesystem.search_files", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 280, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 433, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 705, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "kind": "paraphrase", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "ranking": [ + "filesystem.edit_file", + "playwright.browser_tab_new", + "filesystem.list_directory_with_sizes", + "filesystem.write_file", + "slack.create_file", + "slack.get_file", + "slack.update_file", + "slack.delete_file", + "figma.figma_get_file", + "filesystem.move_file" + ], + "hit1": "filesystem.edit_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 227, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 394, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "ranking": [ + "memory.read_graph", + "memory.delete_relations", + "memory.create_entities", + "memory.create_relations", + "memory.add_observations", + "memory.delete_entities", + "memory.open_nodes", + "memory.search_nodes", + "memory.delete_observations", + "filesystem.read_file" + ], + "hit1": "memory.read_graph", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 259, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 529, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "kind": "verb_product", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "memory.open_nodes", + "figma.figma_get_file_nodes", + "memory.create_entities", + "memory.create_relations", + "memory.add_observations", + "memory.delete_entities", + "memory.delete_observations", + "memory.delete_relations", + "memory.read_graph" + ], + "hit1": "memory.search_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 546, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "kind": "verb_product", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "ranking": [ + "git.git_diff_unstaged", + "git.git_diff", + "git.git_diff_staged", + "git.git_show", + "git.git_status", + "git.git_checkout", + "git.git_branch", + "git.git_commit", + "git.git_add", + "git.git_log" + ], + "hit1": "git.git_diff_unstaged", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 602, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "kind": "identifier", + "gold": [ + "git.git_add" + ], + "alts": [], + "ranking": [ + "filesystem.search_files", + "filesystem.list_directory_with_sizes", + "git.git_add", + "git.git_checkout", + "git.git_show", + "git.git_branch", + "git.git_diff", + "git.git_commit", + "git.git_status", + "github.push_files" + ], + "hit1": "filesystem.search_files", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 213, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 336, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 677, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "kind": "verb_product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "ranking": [ + "shopify.shopify_create_product", + "shopify.shopify_get_product", + "shopify.shopify_update_product", + "shopify.shopify_create_order", + "shopify.shopify_list_products", + "shopify.shopify_list_orders", + "shopify.shopify_get_order", + "shopify.shopify_list_customers", + "stripe.stripe_create_product", + "stripe.stripe_create_price" + ], + "hit1": "shopify.shopify_create_product", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 380, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "kind": "verb_product", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "ranking": [ + "stripe.stripe_cancel_subscription", + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_list_customers", + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "stripe.stripe_list_subscriptions", + "stripe.stripe_create_refund" + ], + "hit1": "stripe.stripe_cancel_subscription", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "kind": "resource", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_payment_intent", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_refund", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment", + "github.create_package", + "github.create_artifact", + "github.create_runner" + ], + "hit1": "stripe.stripe_create_payment_intent", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 897, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_list_oncalls" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "kind": "verb_product", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "ranking": [ + "datadog.datadog_get_monitor", + "datadog.datadog_create_monitor", + "datadog.datadog_list_monitors", + "datadog.datadog_list_logs", + "datadog.datadog_query_metrics", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment", + "github.create_package" + ], + "hit1": "datadog.datadog_get_monitor", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 927, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "kind": "paraphrase", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "ranking": [ + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "sentry.sentry_list_events", + "sentry.sentry_list_projects", + "sentry.sentry_list_issues", + "pagerduty.pagerduty_resolve_incident", + "github.issue_read", + "jira.jira_assign_issue", + "github.issue_write", + "jira.jira_get_issue" + ], + "hit1": "sentry.sentry_get_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 397, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 817, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_apply", + "kubernetes.k8s_delete", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_pod_logs", + "vercel.vercel_get_deployment" + ], + "hit1": "kubernetes.k8s_scale_deployment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 263, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_apply", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_delete", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_scale_deployment", + "kubernetes.k8s_list_services", + "kubernetes.k8s_get_events", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_pod_logs", + "terraform.tf_apply" + ], + "hit1": "kubernetes.k8s_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 253, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "ranking": [ + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_run", + "docker.docker_exec", + "docker.docker_images", + "docker.docker_build", + "docker.docker_ps", + "kubernetes.k8s_pod_logs" + ], + "hit1": "docker.docker_stop", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "kind": "verb_product", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "ranking": [ + "docker.docker_build", + "docker.docker_run", + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_images", + "docker.docker_ps", + "docker.docker_logs", + "docker.docker_exec", + "filesystem.read_media_file" + ], + "hit1": "docker.docker_build", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "kind": "verb_product", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "ranking": [ + "playwright.browser_click", + "puppeteer.puppeteer_click", + "playwright.browser_type", + "playwright.browser_take_screenshot", + "playwright.browser_snapshot", + "playwright.browser_navigate", + "playwright.browser_fill_form", + "playwright.browser_evaluate", + "playwright.browser_tab_list", + "playwright.browser_tab_new" + ], + "hit1": "playwright.browser_click", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 566, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "kind": "paraphrase", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "ranking": [ + "playwright.browser_type", + "playwright.browser_navigate", + "playwright.browser_wait_for", + "playwright.browser_tab_close", + "playwright.browser_tab_list", + "playwright.browser_tab_new", + "playwright.browser_snapshot", + "playwright.browser_click", + "playwright.browser_evaluate", + "playwright.browser_fill_form" + ], + "hit1": "playwright.browser_type", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 554, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_list_projects", + "gitlab.gitlab_list_issues", + "gitlab.gitlab_create_issue", + "gitlab.gitlab_get_issue", + "gitlab.gitlab_get_file", + "gitlab.gitlab_create_pipeline", + "gitlab.gitlab_list_merge_requests", + "gitlab.gitlab_create_merge_request", + "gitlab.gitlab_get_merge_request", + "gitlab.gitlab_merge_merge_request" + ], + "hit1": "gitlab.gitlab_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 398, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 817, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "kind": "verb_product", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "ranking": [ + "bitbucket.bitbucket_create_pr", + "bitbucket.bitbucket_list_prs", + "bitbucket.bitbucket_create_issue", + "bitbucket.bitbucket_list_repos", + "github.merge_pull_request", + "github.pull_request_read", + "github.create_pull_request", + "github.update_pull_request", + "github.update_pull_request_branch", + "github.pull_request_review_write" + ], + "hit1": "bitbucket.bitbucket_create_pr", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 471, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1091, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "kind": "verb_product", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "ranking": [ + "gmail.gmail_create_draft", + "gmail.gmail_search_emails", + "gmail.gmail_list_labels", + "gmail.gmail_send_email", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "gmail.gmail_modify_email", + "github.create_pull_request", + "github.create_gist", + "github.create_milestone" + ], + "hit1": "gmail.gmail_create_draft", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 205, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 352, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 835, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "kind": "verb_product", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_share_file", + "google_drive.gdrive_update_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "gitlab.gitlab_get_file", + "figma.figma_get_file", + "google_calendar.gcal_list_calendars", + "filesystem.write_file" + ], + "hit1": "google_drive.gdrive_share_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "kind": "verb_product", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_list_events", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_list_calendars", + "kubernetes.k8s_get_events", + "sentry.sentry_list_events", + "google_drive.gdrive_list_folder", + "google_drive.gdrive_update_file", + "google_drive.gdrive_share_file" + ], + "hit1": "google_calendar.gcal_list_events", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 784, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "kubernetes.k8s_delete", + "redis.redis_delete", + "aws_s3.s3_delete_object", + "github.delete_repository", + "github.delete_milestone" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 751, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_projects", + "vercel.vercel_list_deployments", + "vercel.vercel_get_deployment", + "vercel.vercel_create_project", + "github.projects_list", + "linear.linear_list_projects", + "jira.jira_list_projects", + "sentry.sentry_list_projects", + "gitlab.gitlab_list_projects", + "github.projects_get" + ], + "hit1": "vercel.vercel_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 301, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "ranking": [ + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update", + "supabase.supabase_invoke_fn", + "supabase.supabase_list_tables", + "elasticsearch.es_get_doc", + "aws_s3.s3_get_object", + "memory.delete_observations", + "docker.docker_logs", + "kubernetes.k8s_pod_logs" + ], + "hit1": "supabase.supabase_select", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "kind": "verb_product", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "figma.figma_get_file", + "figma.figma_get_comments", + "figma.figma_get_images", + "figma.figma_get_file_nodes", + "linear.create_comment", + "linear.get_comment", + "linear.update_comment", + "linear.delete_comment", + "notion.create_comment" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "kind": "verb_product", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "ranking": [ + "okta.okta_list_users", + "okta.okta_get_user", + "okta.okta_create_user", + "okta.okta_deactivate_user", + "linear.list_users", + "github.search_users", + "slack.slack_list_users", + "notion.notion_list_users", + "notion.list_users_tokens", + "slack.slack_get_users" + ], + "hit1": "okta.okta_list_users", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 306, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query_range", + "prometheus.prom_query", + "prometheus.prom_list_metrics", + "postgres.query", + "postgres.explain_query", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "datadog.datadog_query_metrics", + "notion.notion_query_database" + ], + "hit1": "prometheus.prom_query_range", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 680, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "kind": "verb_product", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "ranking": [ + "grafana.grafana_create_annotation", + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment", + "github.create_package", + "github.create_artifact", + "github.create_runner" + ], + "hit1": "grafana.grafana_create_annotation", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 881, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "kind": "identifier", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "ranking": [ + "terraform.tf_apply", + "terraform.tf_plan", + "terraform.tf_state_list", + "kubernetes.k8s_apply" + ], + "hit1": "terraform.tf_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "kind": "verb_product", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "ranking": [ + "asana.asana_list_tasks", + "asana.asana_create_task", + "asana.asana_get_task", + "asana.asana_update_task", + "asana.asana_create_project", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "aws_s3.s3_list_buckets" + ], + "hit1": "asana.asana_list_tasks", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 404, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 682, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "ranking": [ + "trello.trello_add_comment", + "trello.trello_create_card", + "trello.trello_list_cards", + "trello.trello_create_board", + "github.update_issue_comment", + "figma.figma_post_comment", + "linear.create_comment", + "linear.get_comment", + "linear.update_comment", + "linear.delete_comment" + ], + "hit1": "trello.trello_add_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 422, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_update_task", + "clickup.clickup_get_task", + "clickup.clickup_create_task", + "clickup.clickup_list_tasks", + "asana.asana_update_task", + "asana.asana_get_task", + "asana.asana_create_task", + "mongodb.mongo_update", + "github.update_gist", + "github.update_milestone" + ], + "hit1": "clickup.clickup_update_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 795, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "kind": "verb_product", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "ranking": [ + "confluence.confluence_search", + "confluence.confluence_get_page", + "confluence.confluence_create_page", + "confluence.confluence_update_page", + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "github.search_issues", + "github.search_users", + "github.search_orgs", + "elasticsearch.es_search" + ], + "hit1": "confluence.confluence_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 401, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 731, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_deal", + "hubspot.hubspot_create_contact", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_search_contacts", + "github.create_gist", + "github.create_milestone", + "github.create_hook", + "github.create_environment", + "github.create_package" + ], + "hit1": "hubspot.hubspot_create_deal", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 832, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_search", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_create_ticket", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "aws_s3.s3_list_buckets" + ], + "hit1": "zendesk.zendesk_list_tickets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "kind": "verb_product", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "ranking": [ + "discord.discord_list_channels", + "discord.discord_list_members", + "discord.discord_get_channel", + "discord.discord_send_message", + "discord.discord_add_reaction", + "slack.slack_list_channels", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services" + ], + "hit1": "discord.discord_list_channels", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 166, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 625, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "ranking": [ + "redis.redis_set", + "redis.redis_get", + "redis.redis_delete", + "redis.redis_keys", + "playwright.browser_press_key", + "jira.jira_get_issue", + "slack.slack_set_channel_topic", + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "figma.figma_get_file" + ], + "hit1": "redis.redis_set", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert", + "mongodb.mongo_update", + "mongodb.mongo_delete", + "mongodb.mongo_find", + "supabase.supabase_insert", + "mongodb.mongo_list_collections" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "kind": "verb_product", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "ranking": [ + "mysql.mysql_query", + "mysql.mysql_list_tables", + "postgres.query", + "postgres.explain_query", + "salesforce.salesforce_query", + "sqlite.sqlite_query", + "prometheus.prom_query", + "docker.docker_run", + "prometheus.prom_query_range", + "datadog.datadog_query_metrics" + ], + "hit1": "mysql.mysql_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 174, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 289, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 686, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_index_doc", + "elasticsearch.es_search", + "elasticsearch.es_get_doc", + "playwright.browser_tab_close", + "github.create_milestone", + "github.get_milestone", + "github.list_milestones", + "github.update_milestone", + "github.delete_milestone", + "github.create_hook" + ], + "hit1": "elasticsearch.es_index_doc", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 396, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_put_object", + "okta.okta_get_user", + "asana.asana_get_task", + "aws_s3.s3_list_objects", + "elasticsearch.es_search", + "okta.okta_create_user", + "okta.okta_deactivate_user", + "asana.asana_create_task" + ], + "hit1": "aws_s3.s3_get_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 706, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones", + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_dns", + "cloudflare.cf_create_dns", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "aws_s3.s3_list_buckets", + "github.projects_list" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 629, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_create_customer", + "stripe.stripe_create_product", + "stripe.stripe_create_price", + "stripe.stripe_create_refund", + "stripe.stripe_get_customer", + "stripe.stripe_list_customers", + "stripe.stripe_list_invoices", + "stripe.stripe_list_subscriptions", + "stripe.stripe_cancel_subscription" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 632, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "kind": "verb_product", + "gold": [ + "github.actions_list" + ], + "alts": [], + "ranking": [ + "github.actions_list", + "github.actions_get", + "github.actions_run_trigger", + "github.actions_get_job_logs", + "github.list_milestones", + "github.list_hooks", + "github.list_environments", + "github.list_packages", + "github.list_artifacts", + "github.list_runners" + ], + "hit1": "github.actions_list", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 293, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 468, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 845, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "kind": "verb_product", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "ranking": [ + "github.actions_get_job_logs", + "github.actions_get", + "github.actions_list", + "github.actions_run_trigger", + "gitlab.get_job", + "kubernetes.k8s_pod_logs", + "docker.docker_logs", + "gitlab.create_job", + "gitlab.update_job", + "gitlab.delete_job" + ], + "hit1": "github.actions_get_job_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 442, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 822, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "kind": "verb_product", + "gold": [ + "github.create_repository" + ], + "alts": [], + "ranking": [ + "github.create_repository", + "github.delete_repository", + "github.fork_repository", + "github.star_repository", + "github.unstar_repository", + "github.list_repository_collaborators", + "github.get_repository_tree", + "github.create_milestone", + "github.create_hook", + "github.create_environment" + ], + "hit1": "github.create_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 388, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 867, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "kind": "verb_product", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "ranking": [ + "git.git_checkout", + "git.git_create_branch", + "git.git_branch", + "git.git_show", + "github.create_branch", + "git.git_diff", + "git.git_commit", + "git.git_status", + "git.git_add", + "git.git_reset" + ], + "hit1": "git.git_checkout", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 373, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 666, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "kind": "identifier", + "gold": [ + "git.git_log" + ], + "alts": [], + "ranking": [ + "git.git_log", + "git.git_show", + "git.git_diff", + "git.git_status", + "git.git_checkout", + "git.git_branch", + "git.git_commit", + "git.git_add", + "git.git_diff_unstaged", + "git.git_diff_staged" + ], + "hit1": "git.git_log", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 320, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 600, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "kind": "verb_product", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "ranking": [ + "postgres.list_schemas", + "postgres.list_tables", + "postgres.query", + "postgres.describe_table", + "postgres.explain_query", + "clickup.clickup_list_tasks", + "trello.trello_list_cards", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_list_services", + "aws_s3.s3_list_buckets" + ], + "hit1": "postgres.list_schemas", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 158, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 555, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "kind": "verb_product", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "ranking": [ + "memory.open_nodes", + "memory.read_graph", + "memory.search_nodes", + "memory.create_entities", + "memory.delete_relations", + "memory.create_relations", + "memory.add_observations", + "memory.delete_entities", + "memory.delete_observations", + "figma.figma_get_file_nodes" + ], + "hit1": "memory.open_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 260, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 549, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "kind": "paraphrase", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "ranking": [ + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "filesystem.get_file_info", + "filesystem.write_file", + "filesystem.edit_file", + "filesystem.move_file", + "filesystem.list_allowed_directories", + "google_drive.gdrive_get_file", + "gitlab.gitlab_get_file" + ], + "hit1": "filesystem.read_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 462, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 688, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "kind": "paraphrase", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "ranking": [ + "playwright.browser_tab_new", + "filesystem.write_file", + "brave_search.brave_local_search", + "filesystem.edit_file", + "filesystem.read_file", + "filesystem.read_text_file", + "figma.figma_get_file", + "filesystem.read_media_file", + "slack.create_file", + "slack.get_file" + ], + "hit1": "playwright.browser_tab_new", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 366, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 714, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "kind": "verb_product", + "gold": [ + "github.search_issues" + ], + "alts": [], + "ranking": [ + "github.search_issues", + "github.list_issues", + "linear.linear_search_issues", + "jira.jira_search_issues", + "gitlab.gitlab_list_issues", + "linear.linear_list_issues", + "sentry.sentry_list_issues", + "github.search_users", + "github.search_orgs", + "github.search_commits" + ], + "hit1": "github.search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 263, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "kind": "paraphrase", + "gold": [ + "github.get_me" + ], + "alts": [], + "ranking": [ + "github.get_teams", + "linear.get_user", + "okta.okta_get_user", + "okta.okta_deactivate_user", + "slack.slack_get_user_profile", + "linear.create_user", + "linear.update_user", + "linear.delete_user", + "github.get_me", + "github.list_notifications" + ], + "hit1": "github.get_teams", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.1111111111111111, + "ndcg@10": 0.3010299956639812, + "hit_rank": 9, + "k3_calls": 3, + "k3_tokens": 585, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 628, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "cost": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.1066666666666667, + "p95_calls": 1.0, + "success_within_1": 0.9533333333333334, + "success_within_2": 0.9866666666666667, + "success_within_3": 0.9933333333333333, + "mean_tokens": 261.14, + "p95_tokens": 389.09999999999997, + "total_tokens": 39171, + "tokens_per_successful_query": 261.14, + "total_tokens_successful": 39171 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.0533333333333332, + "p95_calls": 1.0, + "success_within_1": 0.9733333333333334, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 398.81333333333333, + "p95_tokens": 565.7999999999997, + "total_tokens": 59822, + "tokens_per_successful_query": 398.81333333333333, + "total_tokens_successful": 59822 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.0133333333333334, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 1.0, + "mean_tokens": 749.76, + "p95_tokens": 987.9499999999989, + "total_tokens": 112464, + "tokens_per_successful_query": 749.76, + "total_tokens_successful": 112464 + } + } +} diff --git a/experiments/mcp-tool-search/results/results.md b/experiments/mcp-tool-search/results/results.md new file mode 100644 index 00000000..04b29070 --- /dev/null +++ b/experiments/mcp-tool-search/results/results.md @@ -0,0 +1,377 @@ +# MCP tool search — reproduction results + +**This is a reproduction, not Zed's unpublished benchmark.** Zed's exact +queries, tools, labels, and 25-approach list are not public. See +[`GAPS.md`](GAPS.md). + +## Headline + +On this 150×536 public reproduction: + +| Result | Number | Note | +|---|---|---| +| Best no-model ranker | `boolean_or` / `hybrid_rrf_bm25_trigram` / `bm25_plain` | R@1 94.7% / 92.0% / 90.7%. CIs overlap. | +| Zed-like weighted BM25 | **85.3% R@1** | Worse than plain BM25 here (CI overlap). Field/verb weights are our guess, not Zed's. | +| Best overall | `hybrid_rrf_bm25_embed` and `hybrid_codedb_weighted_bm25` | **93.3% R@1**, MRR@10 0.96. Embedding needs a model; the codedb hybrid does not. | +| **Embedding vs trigram** | bge-small **92.7%** vs char-trigram **85.3%** vs codedb-trigram **60.7%** | Dense wins R@1; CIs: embed 88–97%, trigram 79–91%. Embed is ~6× slower than in-process trigram. | +| **Where codedb ranks** | `codedb_search` **91.3% R@1** (3rd among shipped-like locals) | Word/symbol/find are the wrong primitive (30–67%). | +| Scaling 536→5k | BM25 and trigram **do not fall off** | Distractors are vendor-distinct; they do not recreate StackOne's lookalike-tool failure. | +| **Agent cost (k=5)** | cheapest **373 tok/q**, 1.01 calls | `hybrid_codedb_weighted_bm25`. All-schemas dump is **38,003** tok/turn (102×). | +| Cost vs recall rank | **different** (29/30 move) | `boolean_or` is #1 recall, #19 cost. Weighted first-page hits beat rank-1. | +| Live agent | **skipped** | No LLM key. `OPENAI_API_KEY=... python live_agent.py` | + +Zed shipped weighted BM25 because it is fast, local, and good enough. On +*this* catalog that conclusion is supportable for **plain** BM25 (0.14 ms +p50, no model). Our guessed field weights did not beat plain BM25; that +is a gap in the reproduction, not a refutation of Zed. + +codedb's own multi-word search (`searchContentRanked` BM25+) already +matches plain BM25 on this task. To *win* tool search it would need +fielded BM25 / verb down-weight / namespace boosts — see GAPS.md. + +--- + +Zed posted on 20 Oct 2026 that they tested 25 approaches on 150 queries +against 536 tools and shipped weighted BM25 with a few normalization +rules ([tweet](https://x.com/zeddotdev/status/2112588120558338261)). +The tweet image is a meme (`fixtures/zed-tweet-2026-10-20.png`), not a +write-up. The Zed blog has no post. `zed-industries/zed` has no public +tool-search fixtures (only an unrelated edit-prediction BM25 retriever). + +Catalog: 536 tools (366 public MCP names + 170 expanded), 51 servers, +150 hand-labelled queries. Frozen in `data/`. + +Metrics are means over 150 queries with 2000-resample bootstrap 95% CIs. +Differences of a few points are usually noise. codedb latencies include +MCP RPC; `hybrid_codedb_*` p50 is RRF-fusion only. + + +## Agent cost (search calls + tokens, not recall@k) + +Recall@k is not the agent metric. An agent calls `search(query)`, gets +the top-k **full tool schemas** (name, description, inputSchema, as an +MCP `tools/list` / `search_tools` hit), and pages until the gold/alt +tool appears. Tokens are `tiktoken` **o200k_base** over the compact JSON +payload actually returned to the model. + +Paging model: offset slices of the saved ranking. +Call *n* returns ranks `[(n-1)k : n k]`. Tools missing from the saved +top-10 are appended in catalog order (pessimistic tail). + +**All-schemas baseline:** `38,003` o200k tokens +per turn to load all 536 tool schemas up front +(170,223 bytes). No search calls. + +### Ranking by tokens burned per successful query (k=5) + +Primary page size is 5 (FastMCP-style). Sorted by mean o200k tokens +returned until gold appears. **This ranking is not the recall@1 ranking.** +29 of 30 approaches change place. `boolean_or` wins recall@1 (94.7%) +and lands 19th on cost: its first page is fatter, and the rare miss +pages deep into the catalog tail. Methods that put gold in the first +page (even at rank 2–5) are cheaper even when they lose rank-1. + +| Cost # | Recall # | Δ | Approach | R@1 | tok/success | mean calls | success@1 | +|---:|---:|---:|---|---:|---:|---:|---:| +| 1 | 3 | +2 | hybrid_codedb_weighted_bm25 | 93.3% | 372.8 | 1.01 | 99.3% | +| 2 | 14 | +12 | hybrid_rrf_weighted_trigram | 86.0% | 376.5 | 1.01 | 99.3% | +| 3 | 13 | +10 | hybrid_rrf_weighted_hashed | 86.0% | 379.0 | 1.01 | 99.3% | +| 4 | 12 | +8 | lexical_rerank_weighted_top20 | 88.0% | 380.4 | 1.01 | 98.7% | +| 5 | 18 | +13 | hashed_ngram_embed | 84.0% | 380.9 | 1.01 | 99.3% | +| 6 | 15 | +9 | bm25_weighted | 85.3% | 382.8 | 1.02 | 98.7% | +| 7 | 17 | +10 | decision_heuristic | 85.3% | 382.8 | 1.02 | 98.7% | +| 8 | 25 | +17 | hybrid_codedb_trigram_bm25 | 70.7% | 384.9 | 1.05 | 95.3% | +| 9 | 10 | +1 | hybrid_rrf_trigram_embed | 90.0% | 388.1 | 1.05 | 99.3% | +| 10 | 4 | -6 | hybrid_rrf_bm25_embed | 93.3% | 389.6 | 1.05 | 99.3% | +| 11 | 2 | -9 | hybrid_codedb_embed | 93.3% | 393.0 | 1.05 | 99.3% | +| 12 | 5 | -7 | embed_bge_small | 92.7% | 393.7 | 1.05 | 99.3% | +| 13 | 11 | -2 | hybrid_rrf_weighted_embed | 88.7% | 397.3 | 1.05 | 99.3% | +| 14 | 19 | +5 | query_rewrite_weighted_bm25 | 83.3% | 398.8 | 1.05 | 97.3% | +| 15 | 6 | -9 | hybrid_rrf_bm25_trigram | 92.0% | 446.9 | 1.20 | 98.7% | +| 16 | 23 | +7 | tfidf | 80.0% | 449.4 | 1.19 | 98.7% | +| 17 | 22 | +5 | bm25_tfidf_hybrid | 82.0% | 450.0 | 1.19 | 98.7% | +| 18 | 8 | -10 | bm25_ident_split | 90.7% | 452.4 | 1.20 | 98.7% | +| 19 | 1 | -18 | boolean_or | 94.7% | 464.9 | 1.11 | 99.3% | +| 20 | 9 | -11 | bm25_plain | 90.7% | 465.3 | 1.23 | 98.0% | +| 21 | 16 | -5 | char_trigram | 85.3% | 503.1 | 1.32 | 98.0% | +| 22 | 7 | -15 | codedb_search | 91.3% | 529.3 | 1.37 | 96.7% | +| 23 | 20 | -3 | bm25_desc_only | 82.7% | 597.3 | 1.52 | 96.0% | +| 24 | 26 | +2 | codedb_word | 67.3% | 748.4 | 1.87 | 91.3% | +| 25 | 21 | -4 | bm25_name_only | 82.0% | 1036.0 | 2.63 | 92.7% | +| 26 | 27 | +1 | codedb_trigram_tokens | 60.7% | 1088.8 | 2.78 | 85.3% | +| 27 | 24 | -3 | boolean_and | 76.7% | 2707.6 | 6.79 | 77.3% | +| 28 | 29 | +1 | codedb_symbol | 30.7% | 3668.0 | 9.29 | 61.3% | +| 29 | 28 | -1 | codedb_find | 48.7% | 6801.5 | 17.26 | 50.7% | +| 30 | 30 | +0 | name_exact_substr | 0.0% | 14978.9 | 38.48 | 0.7% | + +Cheapest: **hybrid_codedb_weighted_bm25** at 373 tokens/query (37,630 less than the all-schemas dump, 99.0% saved). Mean search calls 1.01, first-page hit 99.3%. + +Mean tokens can exceed p95: a few deep-tail misses pull the mean up +while 95% of queries finish in one page. + +### Offline cost, k=3 + +| Approach | mean calls | p95 calls | ≤1 | ≤2 | ≤3 | mean tok | p95 tok | total tok | +|---|---:|---:|---:|---:|---:|---:|---:|---:| +| hybrid_codedb_weighted_bm25 | 1.01 | 1.0 | 98.7% | 100.0% | 100.0% | 236 | 337 | 35,445 | +| hybrid_rrf_weighted_trigram | 1.02 | 1.0 | 98.7% | 99.3% | 100.0% | 237 | 333 | 35,500 | +| hybrid_rrf_weighted_hashed | 1.01 | 1.0 | 98.7% | 100.0% | 100.0% | 239 | 325 | 35,885 | +| lexical_rerank_weighted_top20 | 1.03 | 1.0 | 97.3% | 100.0% | 100.0% | 241 | 373 | 36,149 | +| hashed_ngram_embed | 1.03 | 1.0 | 96.7% | 100.0% | 100.0% | 241 | 340 | 36,135 | +| bm25_weighted | 1.04 | 1.0 | 97.3% | 99.3% | 99.3% | 243 | 350 | 36,435 | +| decision_heuristic | 1.04 | 1.0 | 97.3% | 99.3% | 99.3% | 243 | 350 | 36,435 | +| hybrid_codedb_trigram_bm25 | 1.16 | 2.0 | 90.0% | 95.3% | 98.7% | 268 | 577 | 40,240 | +| hybrid_rrf_trigram_embed | 1.10 | 1.0 | 97.3% | 99.3% | 99.3% | 254 | 329 | 38,082 | +| hybrid_rrf_bm25_embed | 1.09 | 1.0 | 98.7% | 99.3% | 99.3% | 251 | 322 | 37,621 | +| hybrid_codedb_embed | 1.09 | 1.0 | 98.7% | 99.3% | 99.3% | 253 | 346 | 37,948 | +| embed_bge_small | 1.08 | 1.0 | 99.3% | 99.3% | 99.3% | 253 | 316 | 38,022 | +| hybrid_rrf_weighted_embed | 1.08 | 1.0 | 99.3% | 99.3% | 99.3% | 255 | 318 | 38,304 | +| query_rewrite_weighted_bm25 | 1.11 | 1.0 | 95.3% | 98.7% | 99.3% | 261 | 389 | 39,171 | +| hybrid_rrf_bm25_trigram | 1.34 | 1.0 | 98.0% | 98.7% | 99.3% | 314 | 333 | 47,157 | +| tfidf | 1.35 | 1.0 | 97.3% | 98.7% | 99.3% | 319 | 315 | 47,779 | +| bm25_tfidf_hybrid | 1.35 | 1.0 | 97.3% | 98.7% | 99.3% | 319 | 337 | 47,789 | +| bm25_ident_split | 1.34 | 1.0 | 98.0% | 98.7% | 99.3% | 316 | 333 | 47,382 | +| boolean_or | 1.20 | 1.0 | 98.0% | 99.3% | 99.3% | 309 | 393 | 46,369 | +| bm25_plain | 1.40 | 1.0 | 97.3% | 98.0% | 98.7% | 332 | 324 | 49,733 | +| char_trigram | 1.54 | 1.0 | 96.7% | 98.7% | 98.7% | 368 | 330 | 55,230 | +| codedb_search | 1.63 | 1.0 | 96.0% | 97.3% | 97.3% | 400 | 353 | 59,945 | +| bm25_desc_only | 1.90 | 2.0 | 92.7% | 96.7% | 97.3% | 467 | 441 | 70,036 | +| codedb_word | 2.51 | 3.0 | 87.3% | 93.3% | 96.7% | 622 | 668 | 93,371 | +| bm25_name_only | 3.77 | 3.5 | 90.0% | 94.0% | 94.7% | 924 | 976 | 138,610 | +| codedb_trigram_tokens | 4.06 | 30.1 | 80.7% | 87.3% | 90.7% | 1007 | 8227 | 151,069 | +| boolean_and | 10.73 | 59.7 | 77.3% | 77.3% | 77.3% | 2656 | 14505 | 398,471 | +| codedb_symbol | 15.01 | 85.6 | 53.3% | 62.0% | 70.0% | 3683 | 20396 | 552,471 | +| codedb_find | 28.31 | 101.2 | 50.7% | 50.7% | 51.3% | 6935 | 24073 | 1,040,178 | +| name_exact_substr | 63.75 | 116.5 | 0.7% | 0.7% | 1.3% | 15419 | 27302 | 2,312,788 | + +### Offline cost, k=5 + +| Approach | mean calls | p95 calls | ≤1 | ≤2 | ≤3 | mean tok | p95 tok | total tok | +|---|---:|---:|---:|---:|---:|---:|---:|---:| +| hybrid_codedb_weighted_bm25 | 1.01 | 1.0 | 99.3% | 100.0% | 100.0% | 373 | 523 | 55,917 | +| hybrid_rrf_weighted_trigram | 1.01 | 1.0 | 99.3% | 100.0% | 100.0% | 377 | 517 | 56,478 | +| hybrid_rrf_weighted_hashed | 1.01 | 1.0 | 99.3% | 100.0% | 100.0% | 379 | 511 | 56,851 | +| lexical_rerank_weighted_top20 | 1.01 | 1.0 | 98.7% | 100.0% | 100.0% | 380 | 511 | 57,062 | +| hashed_ngram_embed | 1.01 | 1.0 | 99.3% | 100.0% | 100.0% | 381 | 509 | 57,132 | +| bm25_weighted | 1.02 | 1.0 | 98.7% | 99.3% | 100.0% | 383 | 521 | 57,427 | +| decision_heuristic | 1.02 | 1.0 | 98.7% | 99.3% | 100.0% | 383 | 521 | 57,427 | +| hybrid_codedb_trigram_bm25 | 1.05 | 1.0 | 95.3% | 99.3% | 100.0% | 385 | 587 | 57,741 | +| hybrid_rrf_trigram_embed | 1.05 | 1.0 | 99.3% | 99.3% | 99.3% | 388 | 489 | 58,209 | +| hybrid_rrf_bm25_embed | 1.05 | 1.0 | 99.3% | 99.3% | 99.3% | 390 | 536 | 58,441 | +| hybrid_codedb_embed | 1.05 | 1.0 | 99.3% | 99.3% | 99.3% | 393 | 538 | 58,957 | +| embed_bge_small | 1.05 | 1.0 | 99.3% | 99.3% | 99.3% | 394 | 507 | 59,055 | +| hybrid_rrf_weighted_embed | 1.05 | 1.0 | 99.3% | 99.3% | 99.3% | 397 | 492 | 59,589 | +| query_rewrite_weighted_bm25 | 1.05 | 1.0 | 97.3% | 99.3% | 99.3% | 399 | 566 | 59,822 | +| hybrid_rrf_bm25_trigram | 1.20 | 1.0 | 98.7% | 99.3% | 99.3% | 447 | 518 | 67,039 | +| tfidf | 1.19 | 1.0 | 98.7% | 99.3% | 99.3% | 449 | 471 | 67,404 | +| bm25_tfidf_hybrid | 1.19 | 1.0 | 98.7% | 99.3% | 99.3% | 450 | 486 | 67,506 | +| bm25_ident_split | 1.20 | 1.0 | 98.7% | 99.3% | 99.3% | 452 | 515 | 67,853 | +| boolean_or | 1.11 | 1.0 | 99.3% | 99.3% | 99.3% | 465 | 601 | 69,734 | +| bm25_plain | 1.23 | 1.0 | 98.0% | 98.7% | 98.7% | 465 | 478 | 69,795 | +| char_trigram | 1.32 | 1.0 | 98.0% | 98.7% | 98.7% | 503 | 501 | 75,462 | +| codedb_search | 1.37 | 1.0 | 96.7% | 97.3% | 97.3% | 529 | 573 | 79,401 | +| bm25_desc_only | 1.52 | 1.0 | 96.0% | 97.3% | 97.3% | 597 | 598 | 89,594 | +| codedb_word | 1.87 | 2.0 | 91.3% | 96.7% | 96.7% | 748 | 839 | 112,253 | +| bm25_name_only | 2.63 | 2.5 | 92.7% | 94.7% | 95.3% | 1036 | 1115 | 155,394 | +| codedb_trigram_tokens | 2.78 | 18.5 | 85.3% | 92.7% | 92.7% | 1089 | 8117 | 163,319 | +| boolean_and | 6.79 | 35.6 | 77.3% | 77.3% | 77.3% | 2708 | 13944 | 406,143 | +| codedb_symbol | 9.29 | 52.1 | 61.3% | 72.7% | 73.3% | 3668 | 20001 | 550,193 | +| codedb_find | 17.26 | 61.1 | 50.7% | 52.0% | 52.0% | 6801 | 23343 | 1,020,221 | +| name_exact_substr | 38.48 | 70.0 | 0.7% | 1.3% | 2.0% | 14979 | 26505 | 2,246,838 | + +### Offline cost, k=10 + +| Approach | mean calls | p95 calls | ≤1 | ≤2 | ≤3 | mean tok | p95 tok | total tok | +|---|---:|---:|---:|---:|---:|---:|---:|---:| +| hybrid_codedb_weighted_bm25 | 1.00 | 1.0 | 100.0% | 100.0% | 100.0% | 705 | 946 | 105,777 | +| hybrid_rrf_weighted_trigram | 1.00 | 1.0 | 100.0% | 100.0% | 100.0% | 741 | 1001 | 111,088 | +| hybrid_rrf_weighted_hashed | 1.00 | 1.0 | 100.0% | 100.0% | 100.0% | 755 | 1008 | 113,236 | +| lexical_rerank_weighted_top20 | 1.00 | 1.0 | 100.0% | 100.0% | 100.0% | 731 | 960 | 109,649 | +| hashed_ngram_embed | 1.00 | 1.0 | 100.0% | 100.0% | 100.0% | 756 | 942 | 113,413 | +| bm25_weighted | 1.01 | 1.0 | 99.3% | 100.0% | 100.0% | 733 | 953 | 110,019 | +| decision_heuristic | 1.01 | 1.0 | 99.3% | 100.0% | 100.0% | 733 | 953 | 110,019 | +| hybrid_codedb_trigram_bm25 | 1.01 | 1.0 | 99.3% | 100.0% | 100.0% | 716 | 947 | 107,331 | +| hybrid_rrf_trigram_embed | 1.02 | 1.0 | 99.3% | 99.3% | 99.3% | 745 | 970 | 111,823 | +| hybrid_rrf_bm25_embed | 1.02 | 1.0 | 99.3% | 99.3% | 99.3% | 737 | 996 | 110,507 | +| hybrid_codedb_embed | 1.02 | 1.0 | 99.3% | 99.3% | 99.3% | 729 | 979 | 109,423 | +| embed_bge_small | 1.02 | 1.0 | 99.3% | 99.3% | 99.3% | 751 | 979 | 112,672 | +| hybrid_rrf_weighted_embed | 1.02 | 1.0 | 99.3% | 99.3% | 99.3% | 759 | 1008 | 113,793 | +| query_rewrite_weighted_bm25 | 1.01 | 1.0 | 99.3% | 99.3% | 100.0% | 750 | 988 | 112,464 | +| hybrid_rrf_bm25_trigram | 1.09 | 1.0 | 99.3% | 99.3% | 99.3% | 784 | 985 | 117,622 | +| tfidf | 1.09 | 1.0 | 99.3% | 99.3% | 99.3% | 813 | 954 | 121,949 | +| bm25_tfidf_hybrid | 1.09 | 1.0 | 99.3% | 99.3% | 99.3% | 812 | 962 | 121,875 | +| bm25_ident_split | 1.09 | 1.0 | 99.3% | 99.3% | 99.3% | 795 | 1001 | 119,308 | +| boolean_or | 1.05 | 1.0 | 99.3% | 99.3% | 99.3% | 876 | 1188 | 131,331 | +| bm25_plain | 1.11 | 1.0 | 98.7% | 98.7% | 99.3% | 804 | 989 | 120,565 | +| char_trigram | 1.15 | 1.0 | 98.7% | 98.7% | 98.7% | 851 | 994 | 127,618 | +| codedb_search | 1.17 | 1.0 | 97.3% | 97.3% | 98.7% | 844 | 1107 | 126,604 | +| bm25_desc_only | 1.24 | 1.0 | 97.3% | 97.3% | 97.3% | 932 | 1045 | 139,870 | +| codedb_word | 1.40 | 1.0 | 96.7% | 96.7% | 97.3% | 1084 | 1101 | 162,574 | +| bm25_name_only | 1.79 | 1.5 | 94.7% | 95.3% | 95.3% | 1366 | 1302 | 204,854 | +| codedb_trigram_tokens | 1.84 | 9.5 | 92.7% | 92.7% | 92.7% | 1371 | 8033 | 205,675 | +| boolean_and | 3.84 | 18.1 | 77.3% | 78.0% | 78.0% | 3004 | 13725 | 450,537 | +| codedb_symbol | 5.03 | 26.5 | 72.7% | 73.3% | 74.0% | 3851 | 19869 | 577,646 | +| codedb_find | 9.01 | 30.5 | 52.0% | 52.7% | 53.3% | 6907 | 22689 | 1,036,085 | +| name_exact_substr | 19.49 | 35.0 | 1.3% | 4.0% | 7.3% | 14759 | 25789 | 2,213,858 | + +### Live agent loop + +Skipped in this environment: no LLM API key +(`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `AZURE_OPENAI_API_KEY`, +`GOOGLE_API_KEY`, `GEMINI_API_KEY`, `COHERE_API_KEY`, `TOGETHER_API_KEY`, +`GROQ_API_KEY`, `MISTRAL_API_KEY`, `OPENROUTER_API_KEY`). + +Harness is ready: + +```bash +OPENAI_API_KEY=... python live_agent.py +# or +ANTHROPIC_API_KEY=... python live_agent.py +``` + +Default approaches: `bm25_plain`, `bm25_weighted`, `embed_bge_small`, +`hybrid_rrf_bm25_embed`, `codedb_search`, `char_trigram`. Records real +search-call counts, API input/output tokens, whether the named tool is +gold/alt, and wall time. Writes `results/live_agent.json`. + +Raw cost outputs: `results/cost.json`, `results/cost.csv`, +`results/cost_per_query.json`, `results/cost_per_query.csv`. +Per-approach JSON/CSV now include `hit_rank` and `k{3,5,10}_*` fields. + + +## Headline table (n=536 tools, 150 queries) + +| Approach | R@1 | R@3 | R@5 | MRR@10 | nDCG@10 | p50 ms | p95 ms | index s | model/API | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---| +| name_exact_substr | 0.0% | 0.0% | 0.0% | 0.000 | 0.000 | 0.31 | 0.36 | 0.000 | local | +| boolean_and | 76.7% | 77.3% | 77.3% | 0.770 | 0.767 | 0.24 | 0.28 | 0.000 | local | +| boolean_or | 94.7% | 98.0% | 99.3% | 0.964 | 0.959 | 0.30 | 0.37 | 0.000 | local | +| bm25_plain | 90.7% | 97.3% | 98.0% | 0.940 | 0.948 | 0.14 | 0.23 | 0.001 | local | +| bm25_ident_split | 90.7% | 98.0% | 98.7% | 0.942 | 0.952 | 0.14 | 0.26 | 0.001 | local | +| bm25_name_only | 82.0% | 90.0% | 92.7% | 0.870 | 0.882 | 0.10 | 0.14 | 0.001 | local | +| bm25_desc_only | 82.7% | 92.7% | 96.0% | 0.881 | 0.900 | 0.14 | 0.25 | 0.001 | local | +| bm25_weighted | 85.3% | 97.3% | 98.7% | 0.911 | 0.923 | 0.40 | 0.65 | 0.003 | local | +| tfidf | 80.0% | 97.3% | 98.7% | 0.880 | 0.902 | 0.17 | 0.28 | 0.001 | local | +| bm25_tfidf_hybrid | 82.0% | 97.3% | 98.7% | 0.892 | 0.913 | 0.38 | 0.65 | 0.002 | local | +| char_trigram | 85.3% | 96.7% | 98.0% | 0.912 | 0.926 | 0.35 | 0.47 | 0.005 | local | +| hashed_ngram_embed | 84.0% | 96.7% | 99.3% | 0.909 | 0.921 | 4.20 | 4.38 | 0.021 | local | +| hybrid_rrf_bm25_trigram | 92.0% | 98.0% | 98.7% | 0.949 | 0.956 | 0.55 | 0.79 | 0.006 | local | +| hybrid_rrf_weighted_trigram | 86.0% | 98.7% | 99.3% | 0.922 | 0.936 | 0.84 | 1.21 | 0.008 | local | +| hybrid_rrf_weighted_hashed | 86.0% | 98.7% | 99.3% | 0.924 | 0.936 | 4.69 | 4.95 | 0.024 | local | +| decision_heuristic | 85.3% | 97.3% | 98.7% | 0.911 | 0.923 | 0.81 | 1.07 | 0.003 | local | +| query_rewrite_weighted_bm25 | 83.3% | 95.3% | 97.3% | 0.895 | 0.913 | 0.40 | 0.56 | 0.003 | local | +| lexical_rerank_weighted_top20 | 88.0% | 97.3% | 98.7% | 0.929 | 0.940 | 0.58 | 0.82 | 0.003 | local | +| embed_bge_small | 92.7% | 99.3% | 99.3% | 0.957 | 0.960 | 2.14 | 2.64 | 1.960 | model | +| hybrid_rrf_bm25_embed | 93.3% | 98.7% | 99.3% | 0.959 | 0.967 | 2.39 | 3.31 | 1.961 | model | +| hybrid_rrf_weighted_embed | 88.7% | 99.3% | 99.3% | 0.938 | 0.950 | 2.68 | 3.20 | 1.963 | model | +| hybrid_rrf_trigram_embed | 90.0% | 97.3% | 99.3% | 0.940 | 0.950 | 2.59 | 3.03 | 1.965 | model | +| codedb_search | 91.3% | 96.0% | 96.7% | 0.938 | 0.943 | 10.28 | 10.41 | 0.442 | local | +| codedb_trigram_tokens | 60.7% | 80.7% | 85.3% | 0.718 | 0.759 | 30.78 | 41.26 | 0.442 | local | +| codedb_word | 67.3% | 87.3% | 91.3% | 0.783 | 0.819 | 30.81 | 32.20 | 0.442 | local | +| codedb_symbol | 30.7% | 53.3% | 61.3% | 0.432 | 0.497 | 20.62 | 20.85 | 0.442 | local | +| codedb_find | 48.7% | 50.7% | 50.7% | 0.495 | 0.494 | 10.25 | 10.37 | 0.442 | local | +| hybrid_codedb_weighted_bm25 | 93.3% | 98.7% | 99.3% | 0.960 | 0.965 | 0.01 | 0.02 | 0.445 | local | +| hybrid_codedb_trigram_bm25 | 70.7% | 90.0% | 95.3% | 0.809 | 0.845 | 0.01 | 0.01 | 0.445 | local | +| hybrid_codedb_embed | 93.3% | 98.7% | 99.3% | 0.959 | 0.966 | 0.01 | 0.01 | 2.401 | model | + +### Bootstrap 95% CIs for Recall@1 / MRR@10 + +| Approach | Recall@1 | MRR@10 | +|---|---|---| +| name_exact_substr | 0.0% [0.0%, 0.0%] | 0.000 [0.000, 0.000] | +| boolean_and | 76.7% [70.0%, 83.3%] | 0.770 [0.703, 0.840] | +| boolean_or | 94.7% [90.7%, 98.0%] | 0.964 [0.937, 0.987] | +| bm25_plain | 90.7% [86.0%, 95.3%] | 0.940 [0.907, 0.971] | +| bm25_ident_split | 90.7% [86.0%, 95.3%] | 0.942 [0.910, 0.971] | +| bm25_name_only | 82.0% [75.3%, 88.0%] | 0.870 [0.822, 0.916] | +| bm25_desc_only | 82.7% [76.0%, 88.7%] | 0.881 [0.836, 0.922] | +| bm25_weighted | 85.3% [79.3%, 90.7%] | 0.911 [0.874, 0.946] | +| tfidf | 80.0% [73.3%, 86.0%] | 0.880 [0.840, 0.917] | +| bm25_tfidf_hybrid | 82.0% [76.0%, 88.0%] | 0.892 [0.854, 0.927] | +| char_trigram | 85.3% [79.3%, 90.7%] | 0.912 [0.877, 0.946] | +| hashed_ngram_embed | 84.0% [78.0%, 90.0%] | 0.909 [0.874, 0.942] | +| hybrid_rrf_bm25_trigram | 92.0% [87.3%, 96.0%] | 0.949 [0.919, 0.977] | +| hybrid_rrf_weighted_trigram | 86.0% [80.7%, 91.3%] | 0.922 [0.889, 0.953] | +| hybrid_rrf_weighted_hashed | 86.0% [80.0%, 91.3%] | 0.924 [0.893, 0.953] | +| decision_heuristic | 85.3% [79.3%, 90.7%] | 0.911 [0.874, 0.946] | +| query_rewrite_weighted_bm25 | 83.3% [77.3%, 89.3%] | 0.895 [0.855, 0.933] | +| lexical_rerank_weighted_top20 | 88.0% [82.7%, 93.3%] | 0.929 [0.897, 0.960] | +| embed_bge_small | 92.7% [88.0%, 96.7%] | 0.957 [0.929, 0.980] | +| hybrid_rrf_bm25_embed | 93.3% [89.3%, 97.3%] | 0.959 [0.933, 0.983] | +| hybrid_rrf_weighted_embed | 88.7% [83.3%, 93.3%] | 0.938 [0.908, 0.966] | +| hybrid_rrf_trigram_embed | 90.0% [85.3%, 94.7%] | 0.940 [0.908, 0.968] | +| codedb_search | 91.3% [86.7%, 96.0%] | 0.938 [0.901, 0.970] | +| codedb_trigram_tokens | 60.7% [52.7%, 68.7%] | 0.718 [0.658, 0.778] | +| codedb_word | 67.3% [59.3%, 74.7%] | 0.783 [0.728, 0.835] | +| codedb_symbol | 30.7% [24.0%, 38.0%] | 0.432 [0.370, 0.497] | +| codedb_find | 48.7% [40.0%, 56.7%] | 0.495 [0.413, 0.577] | +| hybrid_codedb_weighted_bm25 | 93.3% [89.3%, 97.3%] | 0.960 [0.935, 0.983] | +| hybrid_codedb_trigram_bm25 | 70.7% [63.3%, 78.0%] | 0.809 [0.761, 0.857] | +| hybrid_codedb_embed | 93.3% [89.3%, 97.3%] | 0.959 [0.932, 0.983] | + +## Embedding vs trigram (same queries) + +Head-to-head on the 150-query / 536-tool set. `char_trigram` is an +in-process character-trigram cosine (codedb-like). `codedb_trigram_tokens` +is codedb's real trigram path (one search per token). Dense embedding is +`embed_bge_small` when fastembed is available, else hashed n-grams. + +| Approach | R@1 | R@3 | MRR@10 | nDCG@10 | p50 ms | +|---|---:|---:|---:|---:|---:| +| char_trigram | 85.3% | 96.7% | 0.912 | 0.926 | 0.35 | +| codedb_trigram_tokens | 60.7% | 80.7% | 0.718 | 0.759 | 30.78 | +| embed_bge_small | 92.7% | 99.3% | 0.957 | 0.960 | 2.14 | +| hashed_ngram_embed | 84.0% | 96.7% | 0.909 | 0.921 | 4.20 | +| hybrid_rrf_trigram_embed | 90.0% | 97.3% | 0.940 | 0.950 | 2.59 | +| hybrid_rrf_bm25_trigram | 92.0% | 98.0% | 0.949 | 0.956 | 0.55 | +| hybrid_rrf_weighted_embed | 88.7% | 99.3% | 0.938 | 0.950 | 2.68 | + +## Where codedb ranks + +codedb is a **code-intelligence** engine. Here it is asked to retrieve +MCP tools from a synthetic catalog (one Python file per tool). That is +off-label: no field weights, no verb down-weighting, ranking is for source +files (centrality / call-graph) rather than tool metadata. + +Best codedb-family method by MRR@10: **hybrid_codedb_weighted_bm25** (0.960). + +## Scaling (536 → ~2k → ~5k tools) + +Gold tools stay fixed; distractors reuse the same create/get/list verbs. + +| n tools | Approach | R@1 | R@5 | MRR@10 | p50 ms | +|---:|---|---:|---:|---:|---:| +| 536 | bm25_plain | 90.7% | 98.0% | 0.940 | 0.14 | +| 536 | bm25_weighted | 85.3% | 98.7% | 0.911 | 0.40 | +| 536 | tfidf | 80.0% | 98.7% | 0.880 | 0.16 | +| 536 | char_trigram | 85.3% | 98.0% | 0.912 | 0.35 | +| 536 | hybrid_rrf_weighted_trigram | 86.0% | 99.3% | 0.922 | 0.84 | +| 2000 | bm25_plain | 92.0% | 98.0% | 0.945 | 0.48 | +| 2000 | bm25_weighted | 84.0% | 99.3% | 0.899 | 1.43 | +| 2000 | tfidf | 79.3% | 96.7% | 0.868 | 0.61 | +| 2000 | char_trigram | 85.3% | 97.3% | 0.912 | 1.33 | +| 2000 | hybrid_rrf_weighted_trigram | 86.7% | 99.3% | 0.924 | 2.74 | +| 5000 | bm25_plain | 92.0% | 98.0% | 0.944 | 1.28 | +| 5000 | bm25_weighted | 83.3% | 97.3% | 0.887 | 3.42 | +| 5000 | tfidf | 78.0% | 96.7% | 0.854 | 1.51 | +| 5000 | char_trigram | 85.3% | 97.3% | 0.912 | 3.39 | +| 5000 | hybrid_rrf_weighted_trigram | 86.7% | 98.7% | 0.922 | 6.68 | + +## Approaches we could not run + +- Zed's exact 25-approach list, queries, tools, labels, field weights, and BM25 parameters are not public. +- LLM routing and trained decision models (LightGBM etc.) were not run: no API key / no labelled train split from Zed. +- Live agent loop was not run: no LLM API key. Harness is `python live_agent.py`. +- Cross-encoder rerankers and API embedding models were not run unless those packages/keys are present. +- A second local dense model (gte/nomic) was not downloaded to keep the one-command path small. + + +Full gap list: [`GAPS.md`](GAPS.md). + +## Raw outputs + +Per-approach JSON/CSV lives in `results/` (now includes `hit_rank` and +`k{3,5,10}_*` cost fields). Combined recall table: `results/summary.json`. +Cost tables: `results/cost.json`, `results/cost.csv`, +`results/cost_per_query.json`. Live skip/run: `results/live_agent.json`. + diff --git a/experiments/mcp-tool-search/results/scaling.json b/experiments/mcp-tool-search/results/scaling.json new file mode 100644 index 00000000..90041806 --- /dev/null +++ b/experiments/mcp-tool-search/results/scaling.json @@ -0,0 +1,182 @@ +[ + { + "n_tools": 536, + "name": "bm25_plain", + "recall@1": 0.9066666666666666, + "recall@3": 0.9733333333333334, + "recall@5": 0.98, + "mrr@10": 0.9401851851851851, + "ndcg@10": 0.9478950871964872, + "p50_ms": 0.14209800002618067, + "p95_ms": 0.2306839499738089, + "index_s": 0.0011489039999332817 + }, + { + "n_tools": 536, + "name": "bm25_weighted", + "recall@1": 0.8533333333333334, + "recall@3": 0.9733333333333334, + "recall@5": 0.9866666666666667, + "mrr@10": 0.9111111111111111, + "ndcg@10": 0.9233077080171845, + "p50_ms": 0.4000560000463338, + "p95_ms": 0.6921149499987673, + "index_s": 0.0025080729999444884 + }, + { + "n_tools": 536, + "name": "tfidf", + "recall@1": 0.8, + "recall@3": 0.9733333333333334, + "recall@5": 0.9866666666666667, + "mrr@10": 0.8797222222222223, + "ndcg@10": 0.9022856279855995, + "p50_ms": 0.1646489999984624, + "p95_ms": 0.27000385002793337, + "index_s": 0.0012554109999882712 + }, + { + "n_tools": 536, + "name": "char_trigram", + "recall@1": 0.8533333333333334, + "recall@3": 0.9666666666666667, + "recall@5": 0.98, + "mrr@10": 0.9118888888888889, + "ndcg@10": 0.9257922189326149, + "p50_ms": 0.35120849997838377, + "p95_ms": 0.5566275999910886, + "index_s": 0.003929154000047674 + }, + { + "n_tools": 536, + "name": "hybrid_rrf_weighted_trigram", + "recall@1": 0.86, + "recall@3": 0.9866666666666667, + "recall@5": 0.9933333333333333, + "mrr@10": 0.922074074074074, + "ndcg@10": 0.9362730460311218, + "p50_ms": 0.8366194999780419, + "p95_ms": 1.2119589499718586, + "index_s": 0.006437226999992163 + }, + { + "n_tools": 2000, + "name": "bm25_plain", + "recall@1": 0.92, + "recall@3": 0.9733333333333334, + "recall@5": 0.98, + "mrr@10": 0.945, + "ndcg@10": 0.9509306236046358, + "p50_ms": 0.4807700000242221, + "p95_ms": 1.121455149996109, + "index_s": 0.003890984000008757 + }, + { + "n_tools": 2000, + "name": "bm25_weighted", + "recall@1": 0.84, + "recall@3": 0.9666666666666667, + "recall@5": 0.9933333333333333, + "mrr@10": 0.8985555555555556, + "ndcg@10": 0.9136860902477131, + "p50_ms": 1.4343880000069475, + "p95_ms": 2.8435092499364574, + "index_s": 0.010061115000098653 + }, + { + "n_tools": 2000, + "name": "tfidf", + "recall@1": 0.7933333333333333, + "recall@3": 0.96, + "recall@5": 0.9666666666666667, + "mrr@10": 0.8676825396825397, + "ndcg@10": 0.889823914345432, + "p50_ms": 0.6100300000184689, + "p95_ms": 1.3223620500184547, + "index_s": 0.005720660000065436 + }, + { + "n_tools": 2000, + "name": "char_trigram", + "recall@1": 0.8533333333333334, + "recall@3": 0.9666666666666667, + "recall@5": 0.9733333333333334, + "mrr@10": 0.9116666666666666, + "ndcg@10": 0.9243209355760517, + "p50_ms": 1.3334629999803838, + "p95_ms": 1.7715955999960893, + "index_s": 0.02395692400000371 + }, + { + "n_tools": 2000, + "name": "hybrid_rrf_weighted_trigram", + "recall@1": 0.8666666666666667, + "recall@3": 0.98, + "recall@5": 0.9933333333333333, + "mrr@10": 0.9237777777777777, + "ndcg@10": 0.9351876467817006, + "p50_ms": 2.7353010000297218, + "p95_ms": 4.6674112499999865, + "index_s": 0.03401803900010236 + }, + { + "n_tools": 5000, + "name": "bm25_plain", + "recall@1": 0.92, + "recall@3": 0.9666666666666667, + "recall@5": 0.98, + "mrr@10": 0.9444444444444444, + "ndcg@10": 0.9500461409176629, + "p50_ms": 1.280951500007177, + "p95_ms": 3.186458650043278, + "index_s": 0.010591761000000588 + }, + { + "n_tools": 5000, + "name": "bm25_weighted", + "recall@1": 0.8333333333333334, + "recall@3": 0.9533333333333334, + "recall@5": 0.9733333333333334, + "mrr@10": 0.8866111111111111, + "ndcg@10": 0.902506863602, + "p50_ms": 3.419582000049104, + "p95_ms": 7.11700630001246, + "index_s": 0.02655103000017789 + }, + { + "n_tools": 5000, + "name": "tfidf", + "recall@1": 0.78, + "recall@3": 0.9466666666666667, + "recall@5": 0.9666666666666667, + "mrr@10": 0.8538888888888889, + "ndcg@10": 0.8783669562359429, + "p50_ms": 1.5129959999740095, + "p95_ms": 3.6164074500106835, + "index_s": 0.015511239000034038 + }, + { + "n_tools": 5000, + "name": "char_trigram", + "recall@1": 0.8533333333333334, + "recall@3": 0.9666666666666667, + "recall@5": 0.9733333333333334, + "mrr@10": 0.9116666666666666, + "ndcg@10": 0.9243209355760517, + "p50_ms": 3.3858150000014575, + "p95_ms": 5.894436700049242, + "index_s": 0.0687204070000007 + }, + { + "n_tools": 5000, + "name": "hybrid_rrf_weighted_trigram", + "recall@1": 0.8666666666666667, + "recall@3": 0.9733333333333334, + "recall@5": 0.9866666666666667, + "mrr@10": 0.9222962962962963, + "ndcg@10": 0.934833112045177, + "p50_ms": 6.676154999979644, + "p95_ms": 11.294581150036718, + "index_s": 0.0952714370001786 + } +] diff --git a/experiments/mcp-tool-search/results/summary.json b/experiments/mcp-tool-search/results/summary.json new file mode 100644 index 00000000..8906a35a --- /dev/null +++ b/experiments/mcp-tool-search/results/summary.json @@ -0,0 +1,971 @@ +{ + "reproduction": true, + "zed_fixtures_public": false, + "n_tools": 536, + "n_queries": 150, + "codedb_ran": true, + "methods": [ + { + "name": "name_exact_substr", + "skipped": null, + "notes": "exact/substring on tool name", + "needs_model": false, + "n": 150, + "index_s": 0.0, + "rss_mb": 0.01953125, + "p50_ms": 0.31149599993796073, + "p95_ms": 0.35911225004383596, + "recall@1": 0.0, + "recall@1_lo": 0.0, + "recall@1_hi": 0.0, + "recall@3": 0.0, + "recall@3_lo": 0.0, + "recall@3_hi": 0.0, + "recall@5": 0.0, + "recall@5_lo": 0.0, + "recall@5_hi": 0.0, + "mrr@10": 0.0, + "mrr@10_lo": 0.0, + "mrr@10_hi": 0.0, + "ndcg@10": 0.0, + "ndcg@10_lo": 0.0, + "ndcg@10_hi": 0.0 + }, + { + "name": "boolean_and", + "skipped": null, + "notes": "all query tokens must match", + "needs_model": false, + "n": 150, + "index_s": 0.0, + "rss_mb": 0.0234375, + "p50_ms": 0.24348450006073108, + "p95_ms": 0.27677745001142284, + "recall@1": 0.7666666666666667, + "recall@1_lo": 0.7, + "recall@1_hi": 0.8333333333333334, + "recall@3": 0.7733333333333333, + "recall@3_lo": 0.7066666666666667, + "recall@3_hi": 0.84, + "recall@5": 0.7733333333333333, + "recall@5_lo": 0.7066666666666667, + "recall@5_hi": 0.84, + "mrr@10": 0.77, + "mrr@10_lo": 0.7033333333333334, + "mrr@10_hi": 0.84, + "ndcg@10": 0.7671556541472715, + "ndcg@10_lo": 0.7010483146131327, + "ndcg@10_hi": 0.8352656594616827 + }, + { + "name": "boolean_or", + "skipped": null, + "notes": "token overlap count", + "needs_model": false, + "n": 150, + "index_s": 0.0, + "rss_mb": 0.08984375, + "p50_ms": 0.302081000029375, + "p95_ms": 0.3668339500336514, + "recall@1": 0.9466666666666667, + "recall@1_lo": 0.9066666666666666, + "recall@1_hi": 0.98, + "recall@3": 0.98, + "recall@3_lo": 0.9533333333333334, + "recall@3_hi": 1.0, + "recall@5": 0.9933333333333333, + "recall@5_lo": 0.98, + "recall@5_hi": 1.0, + "mrr@10": 0.9644444444444443, + "mrr@10_lo": 0.9366666666666666, + "mrr@10_hi": 0.9872222222222223, + "ndcg@10": 0.9592042219162568, + "ndcg@10_lo": 0.9358180619000194, + "ndcg@10_hi": 0.9779941382833974 + }, + { + "name": "bm25_plain", + "skipped": null, + "notes": "Okapi BM25, split on non-alnum only", + "needs_model": false, + "n": 150, + "index_s": 0.0009321830000317277, + "rss_mb": 0.046875, + "p50_ms": 0.1412585000366562, + "p95_ms": 0.23318005003716258, + "recall@1": 0.9066666666666666, + "recall@1_lo": 0.86, + "recall@1_hi": 0.9533333333333334, + "recall@3": 0.9733333333333334, + "recall@3_lo": 0.9466666666666667, + "recall@3_hi": 0.9933333333333333, + "recall@5": 0.98, + "recall@5_lo": 0.9533333333333334, + "recall@5_hi": 1.0, + "mrr@10": 0.9401851851851851, + "mrr@10_lo": 0.9066666666666666, + "mrr@10_hi": 0.9707407407407408, + "ndcg@10": 0.9478950871964872, + "ndcg@10_lo": 0.9194780695599083, + "ndcg@10_hi": 0.9739250907995152 + }, + { + "name": "bm25_ident_split", + "skipped": null, + "notes": "BM25 + identifier splitting", + "needs_model": false, + "n": 150, + "index_s": 0.0009321830000317277, + "rss_mb": 0.03515625, + "p50_ms": 0.14297149999720205, + "p95_ms": 0.2555590999463675, + "recall@1": 0.9066666666666666, + "recall@1_lo": 0.86, + "recall@1_hi": 0.9533333333333334, + "recall@3": 0.98, + "recall@3_lo": 0.9533333333333334, + "recall@3_hi": 1.0, + "recall@5": 0.9866666666666667, + "recall@5_lo": 0.9666666666666667, + "recall@5_hi": 1.0, + "mrr@10": 0.9424074074074075, + "mrr@10_lo": 0.91, + "mrr@10_hi": 0.971111111111111, + "ndcg@10": 0.9520969102241539, + "ndcg@10_lo": 0.9253719508213323, + "ndcg@10_hi": 0.9751765482010183 + }, + { + "name": "bm25_name_only", + "skipped": null, + "notes": "BM25 over tool name field", + "needs_model": false, + "n": 150, + "index_s": 0.0006164310000258411, + "rss_mb": 0.03125, + "p50_ms": 0.10278799999241528, + "p95_ms": 0.14317309993998595, + "recall@1": 0.82, + "recall@1_lo": 0.7533333333333333, + "recall@1_hi": 0.88, + "recall@3": 0.9, + "recall@3_lo": 0.8533333333333334, + "recall@3_hi": 0.9466666666666667, + "recall@5": 0.9266666666666666, + "recall@5_lo": 0.8866666666666667, + "recall@5_hi": 0.9666666666666667, + "mrr@10": 0.8695079365079365, + "mrr@10_lo": 0.8217301587301588, + "mrr@10_hi": 0.9156666666666666, + "ndcg@10": 0.8822109879794696, + "ndcg@10_lo": 0.8381211455296853, + "ndcg@10_hi": 0.9216378856737126 + }, + { + "name": "bm25_desc_only", + "skipped": null, + "notes": "BM25 over description field", + "needs_model": false, + "n": 150, + "index_s": 0.0007229240000015125, + "rss_mb": 0.0390625, + "p50_ms": 0.1389104999702795, + "p95_ms": 0.2481368999781352, + "recall@1": 0.8266666666666667, + "recall@1_lo": 0.76, + "recall@1_hi": 0.8866666666666667, + "recall@3": 0.9266666666666666, + "recall@3_lo": 0.8866666666666667, + "recall@3_hi": 0.9666666666666667, + "recall@5": 0.96, + "recall@5_lo": 0.9266666666666666, + "recall@5_hi": 0.9866666666666667, + "mrr@10": 0.8805079365079365, + "mrr@10_lo": 0.8359047619047619, + "mrr@10_hi": 0.9216666666666666, + "ndcg@10": 0.9000526376116909, + "ndcg@10_lo": 0.862655362957362, + "ndcg@10_hi": 0.9347402167331347 + }, + { + "name": "bm25_weighted", + "skipped": null, + "notes": "Zed-like: field BM25 + ident split + verb downweight + name/namespace boosts", + "needs_model": false, + "n": 150, + "index_s": 0.00289805900013107, + "rss_mb": 0.0625, + "p50_ms": 0.4039610000177163, + "p95_ms": 0.6476736999275089, + "recall@1": 0.8533333333333334, + "recall@1_lo": 0.7933333333333333, + "recall@1_hi": 0.9066666666666666, + "recall@3": 0.9733333333333334, + "recall@3_lo": 0.9466666666666667, + "recall@3_hi": 0.9933333333333333, + "recall@5": 0.9866666666666667, + "recall@5_lo": 0.9666666666666667, + "recall@5_hi": 1.0, + "mrr@10": 0.9111111111111111, + "mrr@10_lo": 0.8744444444444444, + "mrr@10_hi": 0.946111111111111, + "ndcg@10": 0.9233077080171845, + "ndcg@10_lo": 0.894350781004159, + "ndcg@10_hi": 0.9506941957337339 + }, + { + "name": "tfidf", + "skipped": null, + "notes": "cosine TF-IDF", + "needs_model": false, + "n": 150, + "index_s": 0.0012128260000281443, + "rss_mb": 0.07421875, + "p50_ms": 0.16994949999116216, + "p95_ms": 0.2750024500073777, + "recall@1": 0.8, + "recall@1_lo": 0.7333333333333333, + "recall@1_hi": 0.86, + "recall@3": 0.9733333333333334, + "recall@3_lo": 0.9466666666666667, + "recall@3_hi": 0.9933333333333333, + "recall@5": 0.9866666666666667, + "recall@5_lo": 0.9666666666666667, + "recall@5_hi": 1.0, + "mrr@10": 0.8797222222222223, + "mrr@10_lo": 0.8402777777777778, + "mrr@10_hi": 0.9172222222222223, + "ndcg@10": 0.9022856279855995, + "ndcg@10_lo": 0.8708873281306948, + "ndcg@10_hi": 0.9323108975923741 + }, + { + "name": "bm25_tfidf_hybrid", + "skipped": null, + "notes": "StackOne-style 20/80 BM25+TF-IDF", + "needs_model": false, + "n": 150, + "index_s": 0.002145009000059872, + "rss_mb": 0.10546875, + "p50_ms": 0.37560699996674884, + "p95_ms": 0.65002184998093, + "recall@1": 0.82, + "recall@1_lo": 0.76, + "recall@1_hi": 0.88, + "recall@3": 0.9733333333333334, + "recall@3_lo": 0.9466666666666667, + "recall@3_hi": 0.9933333333333333, + "recall@5": 0.9866666666666667, + "recall@5_lo": 0.9666666666666667, + "recall@5_hi": 1.0, + "mrr@10": 0.8919444444444444, + "mrr@10_lo": 0.8538888888888889, + "mrr@10_hi": 0.9272222222222223, + "ndcg@10": 0.913323974786248, + "ndcg@10_lo": 0.8835932038147185, + "ndcg@10_hi": 0.941183384439175 + }, + { + "name": "char_trigram", + "skipped": null, + "notes": "character trigram cosine \u2014 local analog of codedb trigrams", + "needs_model": false, + "n": 150, + "index_s": 0.005107007999981761, + "rss_mb": 0.1484375, + "p50_ms": 0.34626800004389224, + "p95_ms": 0.4699199499782477, + "recall@1": 0.8533333333333334, + "recall@1_lo": 0.7933333333333333, + "recall@1_hi": 0.9066666666666666, + "recall@3": 0.9666666666666667, + "recall@3_lo": 0.9333333333333333, + "recall@3_hi": 0.9933333333333333, + "recall@5": 0.98, + "recall@5_lo": 0.9533333333333334, + "recall@5_hi": 1.0, + "mrr@10": 0.9118888888888889, + "mrr@10_lo": 0.8766666666666667, + "mrr@10_hi": 0.9456666666666667, + "ndcg@10": 0.9257922189326149, + "ndcg@10_lo": 0.8958366369759104, + "ndcg@10_hi": 0.9537385986569289 + }, + { + "name": "hashed_ngram_embed", + "skipped": null, + "notes": "fallback sparse hashed n-gram embedding, not neural", + "needs_model": false, + "n": 150, + "index_s": 0.021400491000008515, + "rss_mb": 0.19140625, + "p50_ms": 4.2002120000574905, + "p95_ms": 4.377745999943272, + "recall@1": 0.84, + "recall@1_lo": 0.78, + "recall@1_hi": 0.9, + "recall@3": 0.9666666666666667, + "recall@3_lo": 0.94, + "recall@3_hi": 0.9933333333333333, + "recall@5": 0.9933333333333333, + "recall@5_lo": 0.98, + "recall@5_hi": 1.0, + "mrr@10": 0.9085555555555556, + "mrr@10_lo": 0.8743333333333334, + "mrr@10_hi": 0.9422222222222223, + "ndcg@10": 0.9211576774050081, + "ndcg@10_lo": 0.8934063236877832, + "ndcg@10_hi": 0.9471845982817189 + }, + { + "name": "hybrid_rrf_bm25_trigram", + "skipped": null, + "notes": "RRF of BM25 + char trigram", + "needs_model": false, + "n": 150, + "index_s": 0.0060391910000134885, + "rss_mb": 0.01171875, + "p50_ms": 0.5474685000308455, + "p95_ms": 0.7852027499495757, + "recall@1": 0.92, + "recall@1_lo": 0.8733333333333333, + "recall@1_hi": 0.96, + "recall@3": 0.98, + "recall@3_lo": 0.9533333333333334, + "recall@3_hi": 1.0, + "recall@5": 0.9866666666666667, + "recall@5_lo": 0.9666666666666667, + "recall@5_hi": 1.0, + "mrr@10": 0.9487407407407407, + "mrr@10_lo": 0.9192592592592592, + "mrr@10_hi": 0.9768888888888889, + "ndcg@10": 0.9555953414387945, + "ndcg@10_lo": 0.9313536952974558, + "ndcg@10_hi": 0.9782977920782063 + }, + { + "name": "hybrid_rrf_weighted_trigram", + "skipped": null, + "notes": "RRF of weighted BM25 + char trigram", + "needs_model": false, + "n": 150, + "index_s": 0.008005067000112831, + "rss_mb": 0.01953125, + "p50_ms": 0.8428800000501724, + "p95_ms": 1.2099685999658047, + "recall@1": 0.86, + "recall@1_lo": 0.8066666666666666, + "recall@1_hi": 0.9133333333333333, + "recall@3": 0.9866666666666667, + "recall@3_lo": 0.9666666666666667, + "recall@3_hi": 1.0, + "recall@5": 0.9933333333333333, + "recall@5_lo": 0.98, + "recall@5_hi": 1.0, + "mrr@10": 0.922074074074074, + "mrr@10_lo": 0.8888148148148148, + "mrr@10_hi": 0.9531851851851852, + "ndcg@10": 0.9362730460311218, + "ndcg@10_lo": 0.9105343483940073, + "ndcg@10_hi": 0.9595090861652102 + }, + { + "name": "hybrid_rrf_weighted_hashed", + "skipped": null, + "notes": "RRF of weighted BM25 + hashed embedding", + "needs_model": false, + "n": 150, + "index_s": 0.024298550000139585, + "rss_mb": 0.03125, + "p50_ms": 4.685365999989699, + "p95_ms": 4.952740550021417, + "recall@1": 0.86, + "recall@1_lo": 0.8, + "recall@1_hi": 0.9133333333333333, + "recall@3": 0.9866666666666667, + "recall@3_lo": 0.9666666666666667, + "recall@3_hi": 1.0, + "recall@5": 0.9933333333333333, + "recall@5_lo": 0.98, + "recall@5_hi": 1.0, + "mrr@10": 0.923888888888889, + "mrr@10_lo": 0.8927777777777777, + "mrr@10_hi": 0.9533333333333334, + "ndcg@10": 0.9359694088979469, + "ndcg@10_lo": 0.9119244041164476, + "ndcg@10_hi": 0.9580275511730296 + }, + { + "name": "decision_heuristic", + "skipped": null, + "notes": "rule-based router (identifier \u2192 name match, else weighted BM25); not a trained decision model", + "needs_model": false, + "n": 150, + "index_s": 0.00289805900013107, + "rss_mb": 0.09765625, + "p50_ms": 0.8133175000466508, + "p95_ms": 1.0684915500291936, + "recall@1": 0.8533333333333334, + "recall@1_lo": 0.7933333333333333, + "recall@1_hi": 0.9066666666666666, + "recall@3": 0.9733333333333334, + "recall@3_lo": 0.9466666666666667, + "recall@3_hi": 0.9933333333333333, + "recall@5": 0.9866666666666667, + "recall@5_lo": 0.9666666666666667, + "recall@5_hi": 1.0, + "mrr@10": 0.9111111111111111, + "mrr@10_lo": 0.8744444444444444, + "mrr@10_hi": 0.946111111111111, + "ndcg@10": 0.9233077080171845, + "ndcg@10_lo": 0.894350781004159, + "ndcg@10_hi": 0.9506941957337339 + }, + { + "name": "query_rewrite_weighted_bm25", + "skipped": null, + "notes": "tiny synonym rewrite then weighted BM25 (not an LLM)", + "needs_model": false, + "n": 150, + "index_s": 0.00289805900013107, + "rss_mb": 0.0, + "p50_ms": 0.4016589999196185, + "p95_ms": 0.5639099500399425, + "recall@1": 0.8333333333333334, + "recall@1_lo": 0.7733333333333333, + "recall@1_hi": 0.8933333333333333, + "recall@3": 0.9533333333333334, + "recall@3_lo": 0.9133333333333333, + "recall@3_hi": 0.9866666666666667, + "recall@5": 0.9733333333333334, + "recall@5_lo": 0.9466666666666667, + "recall@5_hi": 0.9933333333333333, + "mrr@10": 0.8954074074074073, + "mrr@10_lo": 0.8554074074074074, + "mrr@10_hi": 0.9333333333333333, + "ndcg@10": 0.9129082726821834, + "ndcg@10_lo": 0.8808897532481432, + "ndcg@10_hi": 0.9425391523882402 + }, + { + "name": "lexical_rerank_weighted_top20", + "skipped": null, + "notes": "lexical overlap rerank of weighted BM25 top-20; not a cross-encoder", + "needs_model": false, + "n": 150, + "index_s": 0.00289805900013107, + "rss_mb": 0.01171875, + "p50_ms": 0.5752700000130062, + "p95_ms": 0.8243450499321623, + "recall@1": 0.88, + "recall@1_lo": 0.8266666666666667, + "recall@1_hi": 0.9333333333333333, + "recall@3": 0.9733333333333334, + "recall@3_lo": 0.9466666666666667, + "recall@3_hi": 0.9933333333333333, + "recall@5": 0.9866666666666667, + "recall@5_lo": 0.9666666666666667, + "recall@5_hi": 1.0, + "mrr@10": 0.928888888888889, + "mrr@10_lo": 0.8966666666666666, + "mrr@10_hi": 0.96, + "ndcg@10": 0.9395892628155625, + "ndcg@10_lo": 0.9144213810520128, + "ndcg@10_hi": 0.9630889375006223 + }, + { + "name": "embed_bge_small", + "skipped": null, + "notes": "fastembed BAAI/bge-small-en-v1.5", + "needs_model": true, + "n": 150, + "index_s": 1.9597746540000571, + "rss_mb": 0.4609375, + "p50_ms": 2.1447405000003528, + "p95_ms": 2.6409478999823923, + "recall@1": 0.9266666666666666, + "recall@1_lo": 0.88, + "recall@1_hi": 0.9666666666666667, + "recall@3": 0.9933333333333333, + "recall@3_lo": 0.98, + "recall@3_hi": 1.0, + "recall@5": 0.9933333333333333, + "recall@5_lo": 0.98, + "recall@5_hi": 1.0, + "mrr@10": 0.9566666666666667, + "mrr@10_lo": 0.928888888888889, + "mrr@10_hi": 0.98, + "ndcg@10": 0.959883370284404, + "ndcg@10_lo": 0.9374330811792257, + "ndcg@10_hi": 0.979001377697449 + }, + { + "name": "hybrid_rrf_bm25_embed", + "skipped": null, + "notes": "RRF BM25 + bge-small", + "needs_model": true, + "n": 150, + "index_s": 1.9607068370000889, + "rss_mb": 0.0, + "p50_ms": 2.387484999985645, + "p95_ms": 3.310145450012669, + "recall@1": 0.9333333333333333, + "recall@1_lo": 0.8933333333333333, + "recall@1_hi": 0.9733333333333334, + "recall@3": 0.9866666666666667, + "recall@3_lo": 0.9666666666666667, + "recall@3_hi": 1.0, + "recall@5": 0.9933333333333333, + "recall@5_lo": 0.98, + "recall@5_hi": 1.0, + "mrr@10": 0.9594444444444444, + "mrr@10_lo": 0.9333333333333333, + "mrr@10_hi": 0.9833333333333333, + "ndcg@10": 0.9671705986683319, + "ndcg@10_lo": 0.9453740714219765, + "ndcg@10_hi": 0.9864568018198906 + }, + { + "name": "hybrid_rrf_weighted_embed", + "skipped": null, + "notes": "RRF weighted BM25 + bge-small", + "needs_model": true, + "n": 150, + "index_s": 1.9626727130001882, + "rss_mb": 0.0, + "p50_ms": 2.6834579999786, + "p95_ms": 3.2046850499682473, + "recall@1": 0.8866666666666667, + "recall@1_lo": 0.8333333333333334, + "recall@1_hi": 0.9333333333333333, + "recall@3": 0.9933333333333333, + "recall@3_lo": 0.98, + "recall@3_hi": 1.0, + "recall@5": 0.9933333333333333, + "recall@5_lo": 0.98, + "recall@5_hi": 1.0, + "mrr@10": 0.9377777777777777, + "mrr@10_lo": 0.9077777777777777, + "mrr@10_hi": 0.9655555555555556, + "ndcg@10": 0.9495127399456762, + "ndcg@10_lo": 0.9253370106437457, + "ndcg@10_hi": 0.9710196028397542 + }, + { + "name": "hybrid_rrf_trigram_embed", + "skipped": null, + "notes": "RRF char-trigram + bge-small", + "needs_model": true, + "n": 150, + "index_s": 1.9648816620000389, + "rss_mb": 0.0, + "p50_ms": 2.585485999929915, + "p95_ms": 3.0305255500081776, + "recall@1": 0.9, + "recall@1_lo": 0.8533333333333334, + "recall@1_hi": 0.9466666666666667, + "recall@3": 0.9733333333333334, + "recall@3_lo": 0.9466666666666667, + "recall@3_hi": 0.9933333333333333, + "recall@5": 0.9933333333333333, + "recall@5_lo": 0.98, + "recall@5_hi": 1.0, + "mrr@10": 0.9398888888888888, + "mrr@10_lo": 0.9082222222222222, + "mrr@10_hi": 0.9683333333333334, + "ndcg@10": 0.9496191211692434, + "ndcg@10_lo": 0.9252086419271226, + "ndcg@10_hi": 0.9720002237819023 + }, + { + "name": "codedb_search", + "skipped": null, + "notes": "codedb_search via MCP (multi-word \u2192 searchContentRanked BM25+; single token \u2192 trigram substring)", + "needs_model": false, + "n": 150, + "index_s": 0.44171834199994464, + "rss_mb": null, + "p50_ms": 10.278972499975225, + "p95_ms": 10.414098049938048, + "recall@1": 0.9133333333333333, + "recall@1_lo": 0.8666666666666667, + "recall@1_hi": 0.96, + "recall@3": 0.96, + "recall@3_lo": 0.9266666666666666, + "recall@3_hi": 0.9866666666666667, + "recall@5": 0.9666666666666667, + "recall@5_lo": 0.9333333333333333, + "recall@5_hi": 0.9933333333333333, + "mrr@10": 0.938, + "mrr@10_lo": 0.9011111111111111, + "mrr@10_hi": 0.97, + "ndcg@10": 0.9427449090658193, + "ndcg@10_lo": 0.9091698969544683, + "ndcg@10_hi": 0.9714351410870373 + }, + { + "name": "codedb_trigram_tokens", + "skipped": null, + "notes": "per-token codedb_search (single-token trigram path) fused with RRF", + "needs_model": false, + "n": 150, + "index_s": 0.44171834199994464, + "rss_mb": null, + "p50_ms": 30.7823924999866, + "p95_ms": 41.255320850029875, + "recall@1": 0.6066666666666667, + "recall@1_lo": 0.5266666666666666, + "recall@1_hi": 0.6866666666666666, + "recall@3": 0.8066666666666666, + "recall@3_lo": 0.74, + "recall@3_hi": 0.8666666666666667, + "recall@5": 0.8533333333333334, + "recall@5_lo": 0.8, + "recall@5_hi": 0.9066666666666666, + "mrr@10": 0.7183042328042328, + "mrr@10_lo": 0.6582407407407408, + "mrr@10_hi": 0.7776190476190475, + "ndcg@10": 0.759147729800791, + "ndcg@10_lo": 0.7073246254501939, + "ndcg@10_hi": 0.809538220808961 + }, + { + "name": "codedb_word", + "skipped": null, + "notes": "codedb_word on distinctive query tokens, RRF", + "needs_model": false, + "n": 150, + "index_s": 0.44171834199994464, + "rss_mb": null, + "p50_ms": 30.814311000028738, + "p95_ms": 32.19671114995322, + "recall@1": 0.6733333333333333, + "recall@1_lo": 0.5933333333333334, + "recall@1_hi": 0.7466666666666667, + "recall@3": 0.8733333333333333, + "recall@3_lo": 0.8133333333333334, + "recall@3_hi": 0.92, + "recall@5": 0.9133333333333333, + "recall@5_lo": 0.8666666666666667, + "recall@5_hi": 0.9533333333333334, + "mrr@10": 0.7833412698412698, + "mrr@10_lo": 0.7279338624338625, + "mrr@10_hi": 0.835026455026455, + "ndcg@10": 0.8190657137179224, + "ndcg@10_lo": 0.773412050894116, + "ndcg@10_hi": 0.8605633185401583 + }, + { + "name": "codedb_symbol", + "skipped": null, + "notes": "codedb_symbol fuzzy on guessed identifier tokens", + "needs_model": false, + "n": 150, + "index_s": 0.44171834199994464, + "rss_mb": null, + "p50_ms": 20.615465000048516, + "p95_ms": 20.850576600042814, + "recall@1": 0.30666666666666664, + "recall@1_lo": 0.24, + "recall@1_hi": 0.38, + "recall@3": 0.5333333333333333, + "recall@3_lo": 0.4533333333333333, + "recall@3_hi": 0.6133333333333333, + "recall@5": 0.6133333333333333, + "recall@5_lo": 0.5333333333333333, + "recall@5_hi": 0.6866666666666666, + "mrr@10": 0.43221693121693117, + "mrr@10_lo": 0.3697010582010582, + "mrr@10_hi": 0.4974232804232804, + "ndcg@10": 0.4972285622400626, + "ndcg@10_lo": 0.43776722351442166, + "ndcg@10_hi": 0.5581462353915159 + }, + { + "name": "codedb_find", + "skipped": null, + "notes": "codedb_find fuzzy filename search (server__name.py files)", + "needs_model": false, + "n": 150, + "index_s": 0.44171834199994464, + "rss_mb": null, + "p50_ms": 10.254846500004078, + "p95_ms": 10.366095449984414, + "recall@1": 0.4866666666666667, + "recall@1_lo": 0.4, + "recall@1_hi": 0.5666666666666667, + "recall@3": 0.5066666666666667, + "recall@3_lo": 0.4266666666666667, + "recall@3_hi": 0.5866666666666667, + "recall@5": 0.5066666666666667, + "recall@5_lo": 0.4266666666666667, + "recall@5_hi": 0.5866666666666667, + "mrr@10": 0.4953968253968254, + "mrr@10_lo": 0.413015873015873, + "mrr@10_hi": 0.5765079365079365, + "ndcg@10": 0.49403029591368397, + "ndcg@10_lo": 0.41143399382506873, + "ndcg@10_hi": 0.5737920498263825 + }, + { + "name": "hybrid_codedb_weighted_bm25", + "skipped": null, + "notes": "RRF codedb_search + weighted BM25", + "needs_model": false, + "n": 150, + "index_s": 0.4446164010000757, + "rss_mb": null, + "p50_ms": 0.006783499998164189, + "p95_ms": 0.015479500012816057, + "recall@1": 0.9333333333333333, + "recall@1_lo": 0.8933333333333333, + "recall@1_hi": 0.9733333333333334, + "recall@3": 0.9866666666666667, + "recall@3_lo": 0.9666666666666667, + "recall@3_hi": 1.0, + "recall@5": 0.9933333333333333, + "recall@5_lo": 0.98, + "recall@5_hi": 1.0, + "mrr@10": 0.9602222222222222, + "mrr@10_lo": 0.9346666666666666, + "mrr@10_hi": 0.9833333333333333, + "ndcg@10": 0.9646616732747915, + "ndcg@10_lo": 0.945318156083469, + "ndcg@10_hi": 0.9824338047871932 + }, + { + "name": "hybrid_codedb_trigram_bm25", + "skipped": null, + "notes": "RRF codedb trigram tokens + weighted BM25", + "needs_model": false, + "n": 150, + "index_s": 0.4446164010000757, + "rss_mb": null, + "p50_ms": 0.0062194999941311835, + "p95_ms": 0.00990849998174781, + "recall@1": 0.7066666666666667, + "recall@1_lo": 0.6333333333333333, + "recall@1_hi": 0.78, + "recall@3": 0.9, + "recall@3_lo": 0.8533333333333334, + "recall@3_hi": 0.9466666666666667, + "recall@5": 0.9533333333333334, + "recall@5_lo": 0.9133333333333333, + "recall@5_hi": 0.9866666666666667, + "mrr@10": 0.8091084656084656, + "mrr@10_lo": 0.7606640211640212, + "mrr@10_hi": 0.8567671957671958, + "ndcg@10": 0.8445062459089498, + "ndcg@10_lo": 0.8062462473957297, + "ndcg@10_hi": 0.882210848512806 + }, + { + "name": "hybrid_codedb_embed", + "skipped": null, + "notes": "RRF codedb_search + embedding", + "needs_model": true, + "n": 150, + "index_s": 2.4014929960000018, + "rss_mb": null, + "p50_ms": 0.006578000011359109, + "p95_ms": 0.014046500018594085, + "recall@1": 0.9333333333333333, + "recall@1_lo": 0.8933333333333333, + "recall@1_hi": 0.9733333333333334, + "recall@3": 0.9866666666666667, + "recall@3_lo": 0.9666666666666667, + "recall@3_hi": 1.0, + "recall@5": 0.9933333333333333, + "recall@5_lo": 0.98, + "recall@5_hi": 1.0, + "mrr@10": 0.9591111111111111, + "mrr@10_lo": 0.9315555555555555, + "mrr@10_hi": 0.9833333333333333, + "ndcg@10": 0.9658210376068275, + "ndcg@10_lo": 0.9431111280939956, + "ndcg@10_hi": 0.98539579120926 + } + ], + "scaling": [ + { + "n_tools": 536, + "name": "bm25_plain", + "recall@1": 0.9066666666666666, + "recall@3": 0.9733333333333334, + "recall@5": 0.98, + "mrr@10": 0.9401851851851851, + "ndcg@10": 0.9478950871964872, + "p50_ms": 0.14209800002618067, + "p95_ms": 0.2306839499738089, + "index_s": 0.0011489039999332817 + }, + { + "n_tools": 536, + "name": "bm25_weighted", + "recall@1": 0.8533333333333334, + "recall@3": 0.9733333333333334, + "recall@5": 0.9866666666666667, + "mrr@10": 0.9111111111111111, + "ndcg@10": 0.9233077080171845, + "p50_ms": 0.4000560000463338, + "p95_ms": 0.6921149499987673, + "index_s": 0.0025080729999444884 + }, + { + "n_tools": 536, + "name": "tfidf", + "recall@1": 0.8, + "recall@3": 0.9733333333333334, + "recall@5": 0.9866666666666667, + "mrr@10": 0.8797222222222223, + "ndcg@10": 0.9022856279855995, + "p50_ms": 0.1646489999984624, + "p95_ms": 0.27000385002793337, + "index_s": 0.0012554109999882712 + }, + { + "n_tools": 536, + "name": "char_trigram", + "recall@1": 0.8533333333333334, + "recall@3": 0.9666666666666667, + "recall@5": 0.98, + "mrr@10": 0.9118888888888889, + "ndcg@10": 0.9257922189326149, + "p50_ms": 0.35120849997838377, + "p95_ms": 0.5566275999910886, + "index_s": 0.003929154000047674 + }, + { + "n_tools": 536, + "name": "hybrid_rrf_weighted_trigram", + "recall@1": 0.86, + "recall@3": 0.9866666666666667, + "recall@5": 0.9933333333333333, + "mrr@10": 0.922074074074074, + "ndcg@10": 0.9362730460311218, + "p50_ms": 0.8366194999780419, + "p95_ms": 1.2119589499718586, + "index_s": 0.006437226999992163 + }, + { + "n_tools": 2000, + "name": "bm25_plain", + "recall@1": 0.92, + "recall@3": 0.9733333333333334, + "recall@5": 0.98, + "mrr@10": 0.945, + "ndcg@10": 0.9509306236046358, + "p50_ms": 0.4807700000242221, + "p95_ms": 1.121455149996109, + "index_s": 0.003890984000008757 + }, + { + "n_tools": 2000, + "name": "bm25_weighted", + "recall@1": 0.84, + "recall@3": 0.9666666666666667, + "recall@5": 0.9933333333333333, + "mrr@10": 0.8985555555555556, + "ndcg@10": 0.9136860902477131, + "p50_ms": 1.4343880000069475, + "p95_ms": 2.8435092499364574, + "index_s": 0.010061115000098653 + }, + { + "n_tools": 2000, + "name": "tfidf", + "recall@1": 0.7933333333333333, + "recall@3": 0.96, + "recall@5": 0.9666666666666667, + "mrr@10": 0.8676825396825397, + "ndcg@10": 0.889823914345432, + "p50_ms": 0.6100300000184689, + "p95_ms": 1.3223620500184547, + "index_s": 0.005720660000065436 + }, + { + "n_tools": 2000, + "name": "char_trigram", + "recall@1": 0.8533333333333334, + "recall@3": 0.9666666666666667, + "recall@5": 0.9733333333333334, + "mrr@10": 0.9116666666666666, + "ndcg@10": 0.9243209355760517, + "p50_ms": 1.3334629999803838, + "p95_ms": 1.7715955999960893, + "index_s": 0.02395692400000371 + }, + { + "n_tools": 2000, + "name": "hybrid_rrf_weighted_trigram", + "recall@1": 0.8666666666666667, + "recall@3": 0.98, + "recall@5": 0.9933333333333333, + "mrr@10": 0.9237777777777777, + "ndcg@10": 0.9351876467817006, + "p50_ms": 2.7353010000297218, + "p95_ms": 4.6674112499999865, + "index_s": 0.03401803900010236 + }, + { + "n_tools": 5000, + "name": "bm25_plain", + "recall@1": 0.92, + "recall@3": 0.9666666666666667, + "recall@5": 0.98, + "mrr@10": 0.9444444444444444, + "ndcg@10": 0.9500461409176629, + "p50_ms": 1.280951500007177, + "p95_ms": 3.186458650043278, + "index_s": 0.010591761000000588 + }, + { + "n_tools": 5000, + "name": "bm25_weighted", + "recall@1": 0.8333333333333334, + "recall@3": 0.9533333333333334, + "recall@5": 0.9733333333333334, + "mrr@10": 0.8866111111111111, + "ndcg@10": 0.902506863602, + "p50_ms": 3.419582000049104, + "p95_ms": 7.11700630001246, + "index_s": 0.02655103000017789 + }, + { + "n_tools": 5000, + "name": "tfidf", + "recall@1": 0.78, + "recall@3": 0.9466666666666667, + "recall@5": 0.9666666666666667, + "mrr@10": 0.8538888888888889, + "ndcg@10": 0.8783669562359429, + "p50_ms": 1.5129959999740095, + "p95_ms": 3.6164074500106835, + "index_s": 0.015511239000034038 + }, + { + "n_tools": 5000, + "name": "char_trigram", + "recall@1": 0.8533333333333334, + "recall@3": 0.9666666666666667, + "recall@5": 0.9733333333333334, + "mrr@10": 0.9116666666666666, + "ndcg@10": 0.9243209355760517, + "p50_ms": 3.3858150000014575, + "p95_ms": 5.894436700049242, + "index_s": 0.0687204070000007 + }, + { + "n_tools": 5000, + "name": "hybrid_rrf_weighted_trigram", + "recall@1": 0.8666666666666667, + "recall@3": 0.9733333333333334, + "recall@5": 0.9866666666666667, + "mrr@10": 0.9222962962962963, + "ndcg@10": 0.934833112045177, + "p50_ms": 6.676154999979644, + "p95_ms": 11.294581150036718, + "index_s": 0.0952714370001786 + } + ] +} diff --git a/experiments/mcp-tool-search/results/tfidf.csv b/experiments/mcp-tool-search/results/tfidf.csv new file mode 100644 index 00000000..00520cfb --- /dev/null +++ b/experiments/mcp-tool-search/results/tfidf.csv @@ -0,0 +1,151 @@ +id,query,kind,hit1,recall@1,recall@3,recall@5,mrr@10,ndcg@10,hit_rank,k3_calls,k3_tokens,k3_success1,k5_calls,k5_tokens,k5_success1,k10_calls,k10_tokens,k10_success1,ranking +q001,Create a GitHub issue,verb_product,gitlab.gitlab_create_issue,0.0,1.0,1.0,0.5000,0.6063,2,1,349,1.0,1,527,1.0,1,947,1.0,gitlab.gitlab_create_issue github.create_issue github.add_issue_comment gitlab.gitlab_get_issue github.update_issue_comment github.issue_read github.issue_write jira.jira_get_issue linear.linear_create_comment jira.jira_update_issue +q002,Create a Jira ticket,verb_product,zendesk.zendesk_get_ticket,0.0,1.0,1.0,0.5000,0.6309,2,1,283,1.0,1,466,1.0,1,951,1.0,zendesk.zendesk_get_ticket jira.jira_create_issue zendesk.zendesk_update_ticket zendesk.zendesk_create_ticket jira.create_filter jira.create_attachment jira.create_board jira.create_sprint jira.create_webhook jira.create_version +q003,Send a Slack message,verb_product,discord.discord_send_message,0.0,1.0,1.0,0.3333,0.5000,3,1,250,1.0,1,427,1.0,1,755,1.0,discord.discord_send_message gmail.gmail_send_email slack.slack_post_message twilio.twilio_send_sms slack.slack_add_reaction slack.get_file slack.delete_file discord.discord_add_reaction slack.slack_get_users slack.update_file +q004,List open pull requests,resource,github.list_pull_requests,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,460,1.0,1,883,1.0,github.list_pull_requests github.search_pull_requests gitlab.gitlab_list_merge_requests github.create_pull_request bitbucket.bitbucket_list_prs memory.open_nodes playwright.browser_tab_list github.pull_request_read github.merge_pull_request github.update_pull_request +q005,Open a pull request on GitHub,verb_product,github.create_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,364,1.0,1,611,1.0,1,1040,1.0,github.create_pull_request github.request_copilot_review github.pull_request_read github.merge_pull_request github.update_pull_request github.pull_request_review_write github.update_pull_request_branch github.search_pull_requests github.list_pull_requests memory.open_nodes +q006,Run a SQL query against postgres,verb_product,postgres.query,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,295,1.0,1,553,1.0,postgres.query mysql.mysql_query sqlite.sqlite_query postgres.explain_query salesforce.salesforce_query postgres.list_schemas postgres.list_tables github.get_check_run postgres.describe_table github.delete_check_run +q007,Read a local file from disk,paraphrase,filesystem.read_file,1.0,1.0,1.0,1.0000,1.0000,1,1,223,1.0,1,428,1.0,1,780,1.0,filesystem.read_file filesystem.read_text_file filesystem.read_media_file gitlab.gitlab_get_file github.delete_file memory.read_graph github.issue_read slack.get_file github.pull_request_read brave_search.brave_local_search +q008,What files are in this folder,paraphrase,google_drive.gdrive_list_folder,0.0,0.0,0.0,0.1250,0.2938,8,3,712,0.0,2,745,0.0,1,723,1.0,google_drive.gdrive_list_folder slack.list_files github.push_files filesystem.search_files github.create_gist google_drive.gdrive_search github.update_gist filesystem.list_directory git.git_add filesystem.directory_tree +q009,Remember this fact in the knowledge graph,paraphrase,memory.read_graph,0.0,1.0,1.0,0.5000,0.4932,2,1,154,1.0,1,262,1.0,1,544,1.0,memory.read_graph memory.create_entities memory.delete_relations memory.create_relations memory.search_nodes memory.open_nodes memory.add_observations memory.delete_entities filesystem.read_file slack.slack_list_channels +q010,Search the web with Brave,verb_product,brave_search.brave_web_search,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,367,1.0,1,710,1.0,brave_search.brave_web_search brave_search.brave_local_search jira.jira_search_issues github.search_users filesystem.list_directory_with_sizes github.search_issues memory.search_nodes github.update_pull_request_branch github.search_orgs github.search_commits +q011,Fetch the contents of a URL,verb_product,fetch.fetch,1.0,1.0,1.0,1.0000,1.0000,1,1,208,1.0,1,372,1.0,1,707,1.0,fetch.fetch filesystem.read_file playwright.browser_navigate github.get_file_contents puppeteer.puppeteer_navigate git.git_add docker.docker_logs git.git_status playwright.browser_tab_new github.get_repository_tree +q012,Show git status,identifier,git.git_show,0.0,1.0,1.0,0.5000,0.6309,2,1,183,1.0,1,299,1.0,1,645,1.0,git.git_show git.git_status git.git_diff_staged git.git_log git.git_diff_unstaged git.git_diff git.git_branch sentry.sentry_update_issue slack.list_files linear.list_users +q013,Commit staged changes,paraphrase,git.git_commit,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,327,1.0,1,663,1.0,git.git_commit git.git_reset git.git_diff_staged github.get_commit git.git_show git.git_log github.push_files +q014,Create a Linear issue,verb_product,linear.linear_create_comment,0.0,0.0,1.0,0.2500,0.4307,4,2,574,0.0,1,463,1.0,1,958,1.0,linear.linear_create_comment linear.linear_get_issue linear.linear_update_issue linear.linear_create_issue linear.linear_assign_issue gitlab.gitlab_create_issue linear.linear_create_project linear.create_comment linear.create_user github.create_issue +q015,Make a task in my project tracker Linear,paraphrase,linear.linear_create_project,0.0,0.0,0.0,0.0000,0.0000,145,49,12448,0.0,29,11769,0.0,15,11810,0.0,linear.linear_create_project clickup.clickup_get_task clickup.clickup_update_task asana.asana_get_task asana.asana_update_task clickup.clickup_create_task asana.asana_create_task github.projects_get mysql.mysql_list_tables mysql.mysql_query +q016,Post a comment on a GitHub issue,verb_product,github.update_issue_comment,0.0,1.0,1.0,0.3333,0.5000,3,1,307,1.0,1,444,1.0,1,732,1.0,github.update_issue_comment figma.figma_post_comment github.add_issue_comment slack.slack_post_message linear.linear_create_comment jira.jira_add_comment linear.get_comment notion.get_comment linear.delete_comment linear.update_comment +q017,Merge the pull request,ambiguous,github.merge_pull_request,1.0,1.0,1.0,1.0000,1.0000,1,1,283,1.0,1,495,1.0,1,1121,1.0,github.merge_pull_request gitlab.gitlab_merge_merge_request gitlab.gitlab_get_merge_request github.update_pull_request_branch gitlab.gitlab_create_merge_request github.pull_request_read github.update_pull_request github.request_copilot_review github.create_pull_request github.pull_request_review_write +q018,Trigger a GitHub Actions workflow,verb_product,github.actions_run_trigger,1.0,1.0,1.0,1.0000,1.0000,1,1,294,1.0,1,458,1.0,1,791,1.0,github.actions_run_trigger github.actions_get github.actions_list github.actions_get_job_logs circleci.circleci_trigger_pipeline github.delete_repository github.list_tags github.get_label git.git_branch github.projects_get +q019,Get Dependabot alerts,resource,github.list_dependabot_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,420,1.0,1,684,1.0,github.list_dependabot_alerts github.get_dependabot_alert github.list_secret_scanning_alerts github.list_code_scanning_alerts slack.get_file linear.get_comment github.get_milestone linear.get_user notion.get_comment gitlab.get_label +q020,List Sentry issues,verb_product,sentry.sentry_list_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,236,1.0,1,410,1.0,1,796,1.0,sentry.sentry_list_issues sentry.sentry_list_projects sentry.sentry_get_issue github.list_issues sentry.sentry_list_events gitlab.gitlab_list_issues sentry.sentry_update_issue github.search_issues linear.linear_search_issues linear.linear_list_issues +q021,Create a Stripe customer,verb_product,stripe.stripe_create_customer,1.0,1.0,1.0,1.0000,1.0000,1,1,186,1.0,1,300,1.0,1,669,1.0,stripe.stripe_create_customer stripe.stripe_get_customer stripe.stripe_create_invoice stripe.stripe_list_invoices stripe.stripe_list_subscriptions stripe.stripe_create_payment_intent stripe.stripe_create_product stripe.stripe_list_customers stripe.stripe_create_refund stripe.stripe_create_price +q022,Refund a Stripe payment,paraphrase,stripe.stripe_create_refund,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,370,1.0,1,647,1.0,stripe.stripe_create_refund stripe.stripe_get_payment_intent stripe.stripe_create_payment_intent stripe.stripe_create_product stripe.stripe_create_customer stripe.stripe_list_customers stripe.stripe_get_customer stripe.stripe_list_invoices stripe.stripe_create_invoice stripe.stripe_list_subscriptions +q023,Send an email with Gmail,verb_product,gmail.gmail_send_email,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,344,1.0,1,711,1.0,gmail.gmail_send_email gmail.gmail_get_email gmail.gmail_trash_email gmail.gmail_list_labels twilio.twilio_send_sms gmail.gmail_create_draft discord.discord_send_message gmail.gmail_modify_email gmail.gmail_search_emails puppeteer.puppeteer_click +q024,Search my Gmail inbox,verb_product,gmail.gmail_search_emails,1.0,1.0,1.0,1.0000,1.0000,1,1,178,1.0,1,315,1.0,1,652,1.0,gmail.gmail_search_emails gmail.gmail_get_email gmail.gmail_list_labels gmail.gmail_create_draft gmail.gmail_trash_email gmail.gmail_send_email mysql.mysql_list_tables gmail.gmail_modify_email mysql.mysql_query github.search_users +q025,Upload a file to Google Drive,paraphrase,google_drive.gdrive_update_file,0.0,0.0,1.0,0.2500,0.4307,4,2,523,0.0,1,415,1.0,1,750,1.0,google_drive.gdrive_update_file google_drive.gdrive_get_file slack.slack_upload_file google_drive.gdrive_create_file google_drive.gdrive_search google_drive.gdrive_share_file google_drive.gdrive_list_folder aws_s3.s3_put_object slack.get_file slack.delete_file +q026,Create a calendar event,resource,google_calendar.gcal_delete_event,0.0,1.0,1.0,0.3333,0.5000,3,1,300,1.0,1,427,1.0,1,962,1.0,google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars github.pull_request_review_write slack.create_file linear.create_comment github.create_milestone linear.create_user +q027,List Kubernetes pods,verb_product,kubernetes.k8s_list_pods,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,324,1.0,1,699,1.0,kubernetes.k8s_list_pods kubernetes.k8s_list_deployments kubernetes.k8s_get_events kubernetes.k8s_list_services kubernetes.k8s_delete kubernetes.k8s_get_pod kubernetes.k8s_pod_logs kubernetes.k8s_apply kubernetes.k8s_scale_deployment jira.jira_list_projects +q028,Get logs from a k8s pod,verb_product,kubernetes.k8s_pod_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,299,1.0,1,455,1.0,1,871,1.0,kubernetes.k8s_pod_logs kubernetes.k8s_get_pod datadog.datadog_list_logs docker.docker_logs github.actions_get_job_logs git.git_log gitlab.gitlab_get_file aws_s3.s3_get_object github.get_tag github.get_file_contents +q029,Run a docker container,verb_product,docker.docker_run,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,345,1.0,1,597,1.0,docker.docker_run docker.docker_exec docker.docker_rm docker.docker_stop docker.docker_logs docker.docker_images docker.docker_ps docker.docker_build github.get_check_run github.delete_check_run +q030,Show docker container logs,verb_product,docker.docker_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,345,1.0,1,645,1.0,docker.docker_logs docker.docker_rm docker.docker_stop docker.docker_exec docker.docker_run kubernetes.k8s_pod_logs git.git_log docker.docker_images git.git_show docker.docker_ps +q031,Navigate the browser to a URL,ambiguous,playwright.browser_navigate,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,294,1.0,1,596,1.0,playwright.browser_navigate puppeteer.puppeteer_navigate playwright.browser_tab_new fetch.fetch playwright.browser_tab_list twilio.twilio_make_call playwright.browser_tab_close playwright.browser_press_key playwright.browser_click playwright.browser_evaluate +q032,Take a screenshot of the page,ambiguous,playwright.browser_take_screenshot,1.0,1.0,1.0,1.0000,1.0000,1,1,226,1.0,1,334,1.0,1,678,1.0,playwright.browser_take_screenshot puppeteer.puppeteer_screenshot notion.notion_get_page playwright.browser_snapshot notion.notion_update_page confluence.confluence_get_page confluence.confluence_update_page notion.notion_create_page git.git_status filesystem.read_file +q033,Create a GitLab merge request,verb_product,gitlab.gitlab_merge_merge_request,0.0,1.0,1.0,0.3333,0.5000,3,1,297,1.0,1,475,1.0,1,939,1.0,gitlab.gitlab_merge_merge_request gitlab.gitlab_get_merge_request gitlab.gitlab_create_merge_request github.merge_pull_request gitlab.gitlab_list_merge_requests gitlab.gitlab_create_issue gitlab.gitlab_create_pipeline github.create_pull_request github.request_copilot_review gitlab.gitlab_list_projects +q034,Search Notion pages,verb_product,notion.notion_search,1.0,1.0,1.0,1.0000,1.0000,1,1,180,1.0,1,283,1.0,1,672,1.0,notion.notion_search notion.notion_list_users notion.get_comment notion.delete_comment github.search_users notion.notion_get_page notion.update_comment notion.create_comment notion.notion_create_database github.search_issues +q035,Create a Notion page,verb_product,notion.notion_get_page,0.0,1.0,1.0,0.3333,0.5000,3,1,230,1.0,1,382,1.0,1,703,1.0,notion.notion_get_page notion.notion_update_page notion.notion_create_page notion.notion_get_comments notion.create_comment notion.notion_create_database confluence.confluence_create_page notion.notion_list_users notion.get_comment notion.delete_comment +q036,Query a Notion database,verb_product,notion.notion_query_database,1.0,1.0,1.0,1.0000,1.0000,1,1,279,1.0,1,386,1.0,1,709,1.0,notion.notion_query_database notion.notion_create_database notion.notion_create_page notion.notion_search notion.notion_list_users notion.get_comment notion.delete_comment notion.notion_get_page notion.update_comment notion.create_comment +q037,Create a Zendesk ticket,verb_product,zendesk.zendesk_get_ticket,0.0,1.0,1.0,0.3333,0.5000,3,1,257,1.0,1,368,1.0,1,875,1.0,zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_create_ticket zendesk.zendesk_search zendesk.zendesk_list_tickets jira.jira_create_issue slack.create_file linear.create_comment github.create_milestone linear.create_user +q038,Create a HubSpot contact,verb_product,hubspot.hubspot_get_contact,0.0,1.0,1.0,0.5000,0.6309,2,1,216,1.0,1,364,1.0,1,899,1.0,hubspot.hubspot_get_contact hubspot.hubspot_create_contact hubspot.hubspot_list_contacts hubspot.hubspot_create_deal hubspot.hubspot_search_contacts github.create_repository github.create_branch github.create_issue github.create_gist github.create_or_update_file +q039,Create a Salesforce lead,verb_product,salesforce.salesforce_convert_lead,0.0,1.0,1.0,0.5000,0.6309,2,1,252,1.0,1,416,1.0,1,796,1.0,salesforce.salesforce_convert_lead salesforce.salesforce_create_lead salesforce.salesforce_create_record salesforce.salesforce_delete_record salesforce.salesforce_update_record salesforce.salesforce_get_record salesforce.salesforce_list_opportunities salesforce.salesforce_query slack.create_file linear.create_comment +q040,Run a SOQL query,identifier,salesforce.salesforce_query,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,359,1.0,1,662,1.0,salesforce.salesforce_query github.get_check_run github.delete_check_run github.update_check_run github.create_check_run postgres.query sqlite.sqlite_query mysql.mysql_query docker.docker_run prometheus.prom_query +q041,List Shopify orders,verb_product,shopify.shopify_list_orders,1.0,1.0,1.0,1.0000,1.0000,1,1,193,1.0,1,338,1.0,1,662,1.0,shopify.shopify_list_orders shopify.shopify_list_customers shopify.shopify_get_product shopify.shopify_update_product shopify.shopify_list_products shopify.shopify_get_order shopify.shopify_create_product shopify.shopify_create_order jira.jira_list_projects slack.list_files +q042,Create a PagerDuty incident,verb_product,pagerduty.pagerduty_get_incident,0.0,1.0,1.0,0.5000,0.6309,2,1,259,1.0,1,387,1.0,1,857,1.0,pagerduty.pagerduty_get_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_oncalls pagerduty.pagerduty_list_incidents slack.create_file linear.create_comment github.create_milestone linear.create_user +q043,Who is on call in PagerDuty,paraphrase,pagerduty.pagerduty_list_oncalls,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,419,1.0,1,772,1.0,pagerduty.pagerduty_list_oncalls pagerduty.pagerduty_get_incident twilio.twilio_make_call pagerduty.pagerduty_list_incidents pagerduty.pagerduty_create_incident slack.slack_create_channel pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_resolve_incident filesystem.list_allowed_directories github.update_issue_comment +q044,Query Datadog metrics,verb_product,datadog.datadog_query_metrics,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,458,1.0,1,797,1.0,datadog.datadog_query_metrics datadog.datadog_list_monitors datadog.datadog_create_monitor datadog.datadog_list_logs datadog.datadog_get_monitor prometheus.prom_list_metrics prometheus.prom_query notion.notion_query_database prometheus.prom_query_range postgres.query +q045,Search Datadog logs,verb_product,datadog.datadog_list_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,458,1.0,1,824,1.0,datadog.datadog_list_logs datadog.datadog_list_monitors datadog.datadog_get_monitor datadog.datadog_create_monitor datadog.datadog_query_metrics kubernetes.k8s_pod_logs github.actions_get_job_logs docker.docker_logs github.search_users git.git_log +q046,List S3 buckets,verb_product,aws_s3.s3_list_buckets,1.0,1.0,1.0,1.0000,1.0000,1,1,170,1.0,1,312,1.0,1,650,1.0,aws_s3.s3_list_buckets jira.jira_list_projects slack.list_files github.list_labels linear.list_users gitlab.list_labels linear.list_labels github.projects_list gitlab.gitlab_list_projects github.list_branches +q047,Upload an object to S3,paraphrase,aws_s3.s3_put_object,1.0,1.0,1.0,1.0000,1.0000,1,1,250,1.0,1,437,1.0,1,905,1.0,aws_s3.s3_put_object aws_s3.s3_delete_object aws_s3.s3_get_object slack.slack_upload_file github.assign_copilot_to_issue gmail.gmail_send_email slack.slack_add_reaction twilio.twilio_send_sms github.create_issue jira.jira_update_issue +q048,Purge Cloudflare cache,verb_product,cloudflare.cf_purge_cache,1.0,1.0,1.0,1.0000,1.0000,1,1,231,1.0,1,340,1.0,1,746,1.0,cloudflare.cf_purge_cache cloudflare.cf_list_zones cloudflare.cf_create_dns cloudflare.cf_list_dns +q049,Create a Cloudflare DNS record,verb_product,cloudflare.cf_create_dns,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,360,1.0,1,757,1.0,cloudflare.cf_create_dns cloudflare.cf_list_dns salesforce.salesforce_create_record cloudflare.cf_list_zones salesforce.salesforce_delete_record salesforce.salesforce_update_record salesforce.salesforce_get_record cloudflare.cf_purge_cache git.git_commit slack.create_file +q050,List Vercel deployments,verb_product,vercel.vercel_list_deployments,1.0,1.0,1.0,1.0000,1.0000,1,1,173,1.0,1,291,1.0,1,618,1.0,vercel.vercel_list_deployments kubernetes.k8s_list_deployments vercel.vercel_list_projects vercel.vercel_create_project vercel.vercel_get_deployment jira.jira_list_projects slack.list_files github.list_labels linear.list_users gitlab.list_labels +q051,Insert a row into Supabase,verb_product,supabase.supabase_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,203,1.0,1,362,1.0,1,665,1.0,supabase.supabase_insert mongodb.mongo_insert supabase.supabase_list_tables supabase.supabase_update supabase.supabase_invoke_fn supabase.supabase_select playwright.browser_type +q052,Export images from Figma,paraphrase,figma.figma_get_images,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,328,1.0,1,777,1.0,figma.figma_get_images docker.docker_images figma.figma_get_file figma.figma_get_file_nodes figma.figma_get_comments figma.figma_post_comment github.delete_file github.create_branch gitlab.gitlab_get_file aws_s3.s3_get_object +q053,Send an SMS with Twilio,verb_product,twilio.twilio_send_sms,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,409,1.0,1,763,1.0,twilio.twilio_send_sms twilio.twilio_list_messages gmail.gmail_send_email twilio.twilio_make_call discord.discord_send_message puppeteer.puppeteer_click filesystem.list_directory_with_sizes puppeteer.puppeteer_fill puppeteer.puppeteer_screenshot github.update_pull_request_branch +q054,Create an Okta user,verb_product,okta.okta_get_user,0.0,1.0,1.0,0.5000,0.6309,2,1,183,1.0,1,329,1.0,1,683,1.0,okta.okta_get_user okta.okta_create_user okta.okta_deactivate_user okta.okta_list_users linear.create_user linear.get_user linear.delete_user linear.update_user slack.slack_get_user_profile github.create_issue +q055,Instant query Prometheus,verb_product,prometheus.prom_query,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,407,1.0,1,679,1.0,prometheus.prom_query prometheus.prom_query_range prometheus.prom_list_metrics notion.notion_query_database datadog.datadog_query_metrics postgres.query sqlite.sqlite_query salesforce.salesforce_query memory.search_nodes github.search_users +q056,Search Grafana dashboards,verb_product,grafana.grafana_search_dashboards,1.0,1.0,1.0,1.0000,1.0000,1,1,217,1.0,1,353,1.0,1,635,1.0,grafana.grafana_search_dashboards grafana.grafana_get_dashboard grafana.grafana_create_annotation github.search_users github.search_issues github.search_orgs github.search_commits linear.linear_search_issues slack.slack_search_messages github.search_pull_requests +q057,Trigger a CircleCI pipeline,verb_product,circleci.circleci_trigger_pipeline,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,324,1.0,1,755,1.0,circleci.circleci_trigger_pipeline circleci.circleci_get_pipeline gitlab.gitlab_create_pipeline gitlab.get_pipeline gitlab.delete_pipeline circleci.circleci_list_pipelines gitlab.update_pipeline gitlab.create_pipeline github.actions_run_trigger hubspot.hubspot_create_deal +q058,Run terraform plan,identifier,terraform.tf_plan,1.0,1.0,1.0,1.0000,1.0000,1,1,172,1.0,1,280,1.0,1,684,1.0,terraform.tf_plan terraform.tf_apply terraform.tf_state_list github.get_check_run github.delete_check_run github.update_check_run github.create_check_run postgres.explain_query docker.docker_run github.actions_get +q059,Create an Asana task,verb_product,asana.asana_get_task,0.0,1.0,1.0,0.5000,0.6309,2,1,267,1.0,1,405,1.0,1,825,1.0,asana.asana_get_task asana.asana_create_task asana.asana_update_task asana.asana_create_project asana.asana_list_tasks clickup.clickup_get_task clickup.clickup_create_task clickup.clickup_update_task github.create_issue okta.okta_create_user +q060,Create a Trello card,verb_product,trello.trello_create_card,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,413,1.0,1,903,1.0,trello.trello_create_card trello.trello_add_comment trello.trello_create_board trello.trello_list_cards slack.create_file linear.create_comment github.create_milestone linear.create_user notion.create_comment gitlab.create_label +q061,Create a ClickUp task,verb_product,clickup.clickup_create_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,362,1.0,1,770,1.0,clickup.clickup_create_task clickup.clickup_get_task clickup.clickup_update_task clickup.clickup_list_tasks asana.asana_get_task asana.asana_update_task asana.asana_create_task playwright.browser_click puppeteer.puppeteer_click slack.create_file +q062,Create a Confluence page,verb_product,confluence.confluence_get_page,0.0,1.0,1.0,0.3333,0.5000,3,1,260,1.0,1,367,1.0,1,755,1.0,confluence.confluence_get_page confluence.confluence_update_page confluence.confluence_create_page confluence.confluence_search notion.notion_get_page notion.notion_create_page notion.notion_update_page playwright.browser_take_screenshot notion.notion_get_comments github.list_commits +q063,Send a Discord message,verb_product,discord.discord_send_message,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,322,1.0,1,740,1.0,discord.discord_send_message discord.discord_add_reaction discord.discord_get_channel discord.discord_list_channels discord.discord_list_members gmail.gmail_send_email twilio.twilio_send_sms slack.slack_post_message github.delete_file gmail.gmail_get_email +q064,Search Elasticsearch,verb_product,elasticsearch.es_search,1.0,1.0,1.0,1.0000,1.0000,1,1,239,1.0,1,375,1.0,1,657,1.0,elasticsearch.es_search elasticsearch.es_get_doc elasticsearch.es_index_doc github.search_users github.search_issues github.search_orgs github.search_commits linear.linear_search_issues slack.slack_search_messages github.search_pull_requests +q065,Get a Redis key,verb_product,redis.redis_get,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,281,1.0,1,579,1.0,redis.redis_get redis.redis_set redis.redis_delete redis.redis_keys jira.jira_get_issue playwright.browser_press_key figma.figma_get_file aws_s3.s3_get_object figma.figma_get_comments jira.jira_update_issue +q066,Find MongoDB documents,verb_product,mongodb.mongo_find,1.0,1.0,1.0,1.0000,1.0000,1,1,258,1.0,1,381,1.0,1,738,1.0,mongodb.mongo_find mongodb.mongo_update mongodb.mongo_delete mongodb.mongo_insert mongodb.mongo_list_collections +q067,List SQLite tables,verb_product,sqlite.sqlite_list_tables,1.0,1.0,1.0,1.0000,1.0000,1,1,147,1.0,1,245,1.0,1,543,1.0,sqlite.sqlite_list_tables sqlite.sqlite_query supabase.supabase_list_tables sqlite.sqlite_describe_table postgres.list_tables mysql.mysql_list_tables jira.jira_list_projects slack.list_files github.list_labels linear.list_users +q068,Describe a postgres table,verb_product,postgres.describe_table,1.0,1.0,1.0,1.0000,1.0000,1,1,215,1.0,1,349,1.0,1,597,1.0,postgres.describe_table sqlite.sqlite_describe_table supabase.supabase_update supabase.supabase_select supabase.supabase_insert postgres.list_schemas postgres.list_tables postgres.query postgres.explain_query +q069,github_create_issue,identifier,gitlab.gitlab_create_issue,0.0,1.0,1.0,0.5000,0.4796,2,1,315,1.0,1,472,1.0,1,900,1.0,gitlab.gitlab_create_issue github.create_issue gitlab.gitlab_get_issue jira.jira_get_issue github.add_issue_comment linear.linear_create_comment jira.jira_update_issue github.issue_read bitbucket.bitbucket_create_issue github.update_issue_comment +q070,createIssue,identifier,gitlab.gitlab_create_issue,0.0,1.0,1.0,0.5000,0.4030,2,1,314,1.0,1,433,1.0,1,864,1.0,gitlab.gitlab_create_issue github.create_issue gitlab.gitlab_get_issue jira.jira_get_issue linear.linear_create_comment jira.jira_update_issue bitbucket.bitbucket_create_issue github.add_issue_comment sentry.sentry_get_issue github.issue_read +q071,list-directory,identifier,filesystem.create_directory,0.0,1.0,1.0,0.5000,0.6309,2,1,234,1.0,1,348,1.0,1,654,1.0,filesystem.create_directory filesystem.list_directory github.get_file_contents filesystem.get_file_info filesystem.list_directory_with_sizes filesystem.directory_tree jira.jira_list_projects slack.list_files github.list_labels linear.list_users +q072,sequential thinking next thought,paraphrase,sequential_thinking.sequentialthinking,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,382,1.0,1,797,1.0,sequential_thinking.sequentialthinking +q073,What time is it in Tokyo,paraphrase,time.get_current_time,1.0,1.0,1.0,1.0000,0.9502,1,1,222,1.0,1,378,1.0,1,699,1.0,time.get_current_time slack.slack_create_channel time.convert_time playwright.browser_wait_for google_calendar.gcal_list_events filesystem.list_allowed_directories jira.jira_add_worklog grafana.grafana_create_annotation slack.get_file slack.delete_file +q074,Convert time between timezones,verb_product,time.convert_time,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,360,1.0,1,704,1.0,time.convert_time salesforce.salesforce_convert_lead time.get_current_time playwright.browser_wait_for google_calendar.gcal_list_events memory.create_relations git.git_diff jira.jira_add_worklog grafana.grafana_create_annotation +q075,Star a GitHub repo,verb_product,github.star_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,201,1.0,1,353,1.0,1,780,1.0,github.star_repository github.delete_repository git.git_branch github.get_label github.list_labels github.list_tags github.list_branches github.search_issues github.get_tag github.delete_file +q076,Fork a repository,resource,github.fork_repository,1.0,1.0,1.0,1.0000,1.0000,1,1,248,1.0,1,386,1.0,1,825,1.0,github.fork_repository github.delete_repository github.create_repository github.star_repository github.unstar_repository github.list_repository_collaborators github.list_labels github.get_repository_tree github.list_branches github.delete_file +q077,Get the latest GitHub release,verb_product,github.get_latest_release,1.0,1.0,1.0,1.0000,1.0000,1,1,201,1.0,1,367,1.0,1,753,1.0,github.get_latest_release gitlab.get_release gitlab.delete_release gitlab.update_release gitlab.create_release github.get_release_by_tag github.get_teams github.get_repository_tree github.get_label github.projects_get +q078,Create a gist,resource,github.get_gist,0.0,1.0,1.0,0.3333,0.5000,3,1,229,1.0,1,421,1.0,1,911,1.0,github.get_gist github.update_gist github.create_gist slack.create_file linear.create_comment github.create_milestone linear.create_user notion.create_comment gitlab.create_label jira.create_filter +q079,Search GitHub code,verb_product,github.search_code,1.0,1.0,1.0,1.0000,1.0000,1,1,283,1.0,1,419,1.0,1,725,1.0,github.search_code github.list_code_scanning_alerts github.get_code_scanning_alert github.search_users github.search_issues github.search_orgs github.search_commits github.search_pull_requests github.search_repositories hubspot.hubspot_search_contacts +q080,List GitHub notifications,verb_product,github.list_notifications,1.0,1.0,1.0,1.0000,1.0000,1,1,192,1.0,1,328,1.0,1,687,1.0,github.list_notifications github.mark_all_notifications_read github.list_labels github.list_tags github.projects_list git.git_branch github.list_branches github.list_issues github.actions_list github.list_releases +q081,Assign Copilot to an issue,verb_product,github.assign_copilot_to_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,267,1.0,1,434,1.0,1,879,1.0,github.assign_copilot_to_issue linear.linear_assign_issue jira.jira_assign_issue github.request_copilot_review jira.jira_update_issue github.add_issue_comment github.create_issue linear.linear_create_comment jira.jira_add_comment gitlab.gitlab_get_issue +q082,Get secret scanning alerts,resource,github.list_secret_scanning_alerts,1.0,1.0,1.0,1.0000,1.0000,1,1,295,1.0,1,470,1.0,1,733,1.0,github.list_secret_scanning_alerts github.get_secret_scanning_alert github.list_code_scanning_alerts github.list_dependabot_alerts github.get_code_scanning_alert slack.get_file linear.get_comment github.get_milestone linear.get_user notion.get_comment +q083,Create or update a file on GitHub,verb_product,github.create_or_update_file,1.0,1.0,1.0,1.0000,0.7602,1,1,333,1.0,1,545,1.0,1,957,1.0,github.create_or_update_file slack.update_file slack.create_file github.update_issue_comment github.get_file_contents slack.get_file filesystem.write_file github.delete_file github.projects_write google_drive.gdrive_update_file +q084,List collaborators on a repo,resource,github.list_repository_collaborators,1.0,1.0,1.0,1.0000,1.0000,1,1,307,1.0,1,462,1.0,1,948,1.0,github.list_repository_collaborators github.list_commits github.update_issue_comment github.get_discussion_comments pagerduty.pagerduty_list_oncalls github.create_pull_request trello.trello_list_cards github.request_copilot_review notion.notion_get_comments github.assign_copilot_to_issue +q085,Reply in a Slack thread,verb_product,slack.slack_reply_to_thread,1.0,1.0,1.0,1.0000,1.0000,1,1,240,1.0,1,346,1.0,1,694,1.0,slack.slack_reply_to_thread slack.slack_get_thread_replies github.get_notification_details slack.get_file slack.delete_file slack.update_file slack.create_file github.dismiss_notification slack.slack_get_users slack.slack_list_users +q086,Add an emoji reaction on Slack,verb_product,slack.slack_add_reaction,1.0,1.0,1.0,1.0000,1.0000,1,1,268,1.0,1,438,1.0,1,802,1.0,slack.slack_add_reaction discord.discord_add_reaction jira.jira_add_comment github.add_issue_comment git.git_add gmail.gmail_modify_email trello.trello_add_comment memory.add_observations jira.jira_add_worklog slack.get_file +q087,Search Slack messages,verb_product,slack.slack_search_messages,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,334,1.0,1,633,1.0,slack.slack_search_messages gmail.gmail_search_emails twilio.twilio_list_messages slack.slack_get_channel_history github.search_users github.search_issues github.search_orgs slack.get_file github.search_commits linear.linear_search_issues +q088,Create a Slack channel,verb_product,slack.slack_create_channel,1.0,1.0,1.0,1.0000,1.0000,1,1,196,1.0,1,339,1.0,1,718,1.0,slack.slack_create_channel slack.slack_join_channel slack.slack_leave_channel slack.slack_get_channel_history slack.slack_post_message slack.slack_set_channel_topic discord.discord_get_channel slack.slack_upload_file slack.create_file slack.slack_get_thread_replies +q089,Transition a Jira issue,verb_product,jira.jira_transition_issue,1.0,1.0,1.0,1.0000,1.0000,1,1,207,1.0,1,342,1.0,1,816,1.0,jira.jira_transition_issue jira.jira_get_issue jira.jira_update_issue jira.jira_assign_issue jira.jira_add_comment jira.jira_create_issue gitlab.gitlab_get_issue jira.jira_add_worklog gitlab.gitlab_create_issue github.add_issue_comment +q090,Search Jira with JQL,identifier,jira.jira_search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,202,1.0,1,322,1.0,1,602,1.0,jira.jira_search_issues github.search_users filesystem.list_directory_with_sizes github.search_issues jira.jira_list_projects github.search_orgs jira.get_filter github.search_commits linear.linear_search_issues slack.slack_search_messages +q091,Add a worklog to Jira,verb_product,jira.jira_add_worklog,1.0,1.0,1.0,1.0000,1.0000,1,1,285,1.0,1,417,1.0,1,839,1.0,jira.jira_add_worklog jira.jira_add_comment github.add_issue_comment git.git_add linear.linear_create_comment memory.add_observations github.add_comment_to_pending_review discord.discord_add_reaction slack.slack_add_reaction jira.jira_list_projects +q092,List Linear teams,verb_product,linear.linear_list_teams,1.0,1.0,1.0,1.0000,1.0000,1,1,137,1.0,1,283,1.0,1,591,1.0,linear.linear_list_teams github.get_teams linear.linear_list_projects linear.list_users linear.list_labels linear.get_comment linear.list_comments linear.linear_create_project linear.get_user linear.delete_comment +q093,Comment on a Linear issue,verb_product,linear.linear_create_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,244,1.0,1,360,1.0,1,798,1.0,linear.linear_create_comment github.update_issue_comment linear.get_comment linear.delete_comment linear.update_comment linear.create_comment github.add_issue_comment jira.jira_add_comment linear.linear_get_issue linear.linear_update_issue +q094,Move a local file,paraphrase,filesystem.move_file,1.0,1.0,1.0,1.0000,1.0000,1,1,224,1.0,1,335,1.0,1,684,1.0,filesystem.move_file filesystem.read_file filesystem.read_text_file filesystem.read_media_file slack.get_file brave_search.brave_local_search slack.delete_file slack.update_file gmail.gmail_trash_email slack.create_file +q095,Recursively search files by pattern,paraphrase,filesystem.search_files,1.0,1.0,1.0,1.0000,1.0000,1,1,212,1.0,1,322,1.0,1,716,1.0,filesystem.search_files redis.redis_keys slack.list_files linear.linear_search_issues google_drive.gdrive_search github.search_repositories github.search_users github.push_files github.search_issues github.search_orgs +q096,Edit a file in place with old and new string,paraphrase,filesystem.edit_file,1.0,1.0,1.0,1.0000,1.0000,1,1,247,1.0,1,353,1.0,1,735,1.0,filesystem.edit_file filesystem.list_directory_with_sizes filesystem.write_file slack.get_file slack.delete_file playwright.browser_tab_new slack.update_file slack.create_file twilio.twilio_make_call github.create_repository +q097,Read the whole knowledge graph,paraphrase,memory.read_graph,1.0,1.0,1.0,1.0000,1.0000,1,1,152,1.0,1,260,1.0,1,561,1.0,memory.read_graph memory.create_entities memory.delete_relations memory.create_relations memory.search_nodes filesystem.read_file memory.open_nodes memory.delete_entities memory.add_observations filesystem.read_text_file +q098,Search memory nodes,verb_product,memory.search_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,202,1.0,1,338,1.0,1,620,1.0,memory.search_nodes memory.open_nodes figma.figma_get_file_nodes github.search_users github.search_issues github.search_orgs github.search_commits linear.linear_search_issues slack.slack_search_messages github.search_pull_requests +q099,Show unstaged git diff,verb_product,git.git_diff_unstaged,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,303,1.0,1,621,1.0,git.git_diff_unstaged git.git_diff git.git_diff_staged git.git_show git.git_status git.git_log git.git_branch git.git_checkout git.git_commit git.git_create_branch +q100,Stage files with git add,identifier,git.git_add,1.0,1.0,1.0,1.0000,1.0000,1,1,220,1.0,1,377,1.0,1,774,1.0,git.git_add filesystem.list_directory_with_sizes slack.list_files salesforce.salesforce_list_opportunities github.add_issue_comment jira.jira_add_comment github.push_files memory.add_observations jira.jira_add_worklog filesystem.search_files +q101,Create a Shopify product,verb_product,shopify.shopify_get_product,0.0,1.0,1.0,0.3333,0.5000,3,1,240,1.0,1,364,1.0,1,702,1.0,shopify.shopify_get_product shopify.shopify_update_product shopify.shopify_create_product stripe.stripe_create_product shopify.shopify_get_order shopify.shopify_list_customers shopify.shopify_create_order shopify.shopify_list_orders shopify.shopify_list_products stripe.stripe_create_price +q102,Cancel a Stripe subscription,verb_product,stripe.stripe_cancel_subscription,1.0,1.0,1.0,1.0000,1.0000,1,1,206,1.0,1,321,1.0,1,610,1.0,stripe.stripe_cancel_subscription stripe.stripe_create_product stripe.stripe_create_customer stripe.stripe_list_customers stripe.stripe_get_customer stripe.stripe_list_invoices stripe.stripe_create_invoice stripe.stripe_list_subscriptions stripe.stripe_get_payment_intent stripe.stripe_create_refund +q103,Create a payment intent,resource,stripe.stripe_get_payment_intent,0.0,1.0,1.0,0.5000,0.6309,2,1,234,1.0,1,426,1.0,1,916,1.0,stripe.stripe_get_payment_intent stripe.stripe_create_payment_intent stripe.stripe_create_refund slack.create_file linear.create_comment github.create_milestone linear.create_user notion.create_comment gitlab.create_label jira.create_filter +q104,Acknowledge a PagerDuty incident,verb_product,pagerduty.pagerduty_acknowledge_incident,1.0,1.0,1.0,1.0000,1.0000,1,1,261,1.0,1,389,1.0,1,717,1.0,pagerduty.pagerduty_acknowledge_incident pagerduty.pagerduty_get_incident pagerduty.pagerduty_create_incident pagerduty.pagerduty_resolve_incident pagerduty.pagerduty_list_oncalls pagerduty.pagerduty_list_incidents +q105,Create a Datadog monitor,verb_product,datadog.datadog_get_monitor,0.0,1.0,1.0,0.5000,0.6309,2,1,259,1.0,1,459,1.0,1,948,1.0,datadog.datadog_get_monitor datadog.datadog_create_monitor datadog.datadog_list_monitors datadog.datadog_list_logs datadog.datadog_query_metrics slack.create_file linear.create_comment github.create_milestone linear.create_user notion.create_comment +q106,Resolve a Sentry issue,paraphrase,sentry.sentry_get_issue,0.0,1.0,1.0,0.5000,0.6309,2,1,251,1.0,1,368,1.0,1,745,1.0,sentry.sentry_get_issue sentry.sentry_update_issue sentry.sentry_list_events pagerduty.pagerduty_resolve_incident sentry.sentry_list_projects sentry.sentry_list_issues gitlab.gitlab_get_issue jira.jira_get_issue jira.jira_update_issue gitlab.gitlab_create_issue +q107,Scale a Kubernetes deployment,verb_product,kubernetes.k8s_scale_deployment,1.0,1.0,1.0,1.0000,1.0000,1,1,230,1.0,1,374,1.0,1,726,1.0,kubernetes.k8s_scale_deployment vercel.vercel_get_deployment kubernetes.k8s_get_events kubernetes.k8s_list_deployments kubernetes.k8s_delete kubernetes.k8s_get_pod kubernetes.k8s_list_services kubernetes.k8s_list_pods kubernetes.k8s_pod_logs kubernetes.k8s_apply +q108,Apply a Kubernetes manifest,verb_product,kubernetes.k8s_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,200,1.0,1,344,1.0,1,727,1.0,kubernetes.k8s_apply terraform.tf_apply kubernetes.k8s_get_events kubernetes.k8s_list_deployments kubernetes.k8s_delete kubernetes.k8s_get_pod kubernetes.k8s_list_services kubernetes.k8s_list_pods kubernetes.k8s_pod_logs kubernetes.k8s_scale_deployment +q109,Stop a docker container,verb_product,docker.docker_stop,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,345,1.0,1,641,1.0,docker.docker_stop docker.docker_rm docker.docker_logs docker.docker_exec docker.docker_run docker.docker_images docker.docker_ps docker.docker_build kubernetes.k8s_pod_logs +q110,Build a docker image,verb_product,docker.docker_build,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,317,1.0,1,601,1.0,docker.docker_build docker.docker_run docker.docker_images docker.docker_ps docker.docker_rm docker.docker_stop docker.docker_logs docker.docker_exec filesystem.read_media_file +q111,Click an element in Playwright,verb_product,playwright.browser_click,1.0,1.0,1.0,1.0000,1.0000,1,1,216,1.0,1,381,1.0,1,654,1.0,playwright.browser_click puppeteer.puppeteer_click playwright.browser_type playwright.browser_take_screenshot clickup.clickup_create_task playwright.browser_snapshot clickup.clickup_list_tasks clickup.clickup_get_task clickup.clickup_update_task playwright.browser_tab_list +q112,Type text into a browser input,paraphrase,playwright.browser_type,1.0,1.0,1.0,1.0000,0.9239,1,1,232,1.0,1,337,1.0,1,647,1.0,playwright.browser_type playwright.browser_wait_for github.projects_list puppeteer.puppeteer_fill playwright.browser_tab_list filesystem.read_text_file playwright.browser_navigate playwright.browser_tab_new playwright.browser_tab_close mongodb.mongo_insert +q113,List GitLab projects,verb_product,gitlab.gitlab_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,171,1.0,1,255,1.0,1,609,1.0,gitlab.gitlab_list_projects github.projects_list jira.jira_list_projects linear.linear_list_projects vercel.vercel_list_projects sentry.sentry_list_projects github.projects_get gitlab.gitlab_list_issues gitlab.gitlab_create_issue gitlab.gitlab_get_issue +q114,Create a Bitbucket pull request,verb_product,bitbucket.bitbucket_create_pr,1.0,1.0,1.0,1.0000,1.0000,1,1,313,1.0,1,456,1.0,1,1091,1.0,bitbucket.bitbucket_create_pr bitbucket.bitbucket_list_prs bitbucket.bitbucket_create_issue github.pull_request_read bitbucket.bitbucket_list_repos github.merge_pull_request github.update_pull_request github.update_pull_request_branch github.create_pull_request github.pull_request_review_write +q115,Create a Gmail draft,verb_product,gmail.gmail_create_draft,1.0,1.0,1.0,1.0000,1.0000,1,1,188,1.0,1,315,1.0,1,844,1.0,gmail.gmail_create_draft gmail.gmail_get_email gmail.gmail_list_labels gmail.gmail_search_emails gmail.gmail_trash_email gmail.gmail_send_email github.create_pull_request gmail.gmail_modify_email slack.create_file linear.create_comment +q116,Share a Google Drive file,verb_product,google_drive.gdrive_share_file,1.0,1.0,1.0,1.0000,1.0000,1,1,254,1.0,1,417,1.0,1,684,1.0,google_drive.gdrive_share_file google_drive.gdrive_update_file google_drive.gdrive_get_file google_drive.gdrive_create_file google_drive.gdrive_search google_drive.gdrive_list_folder slack.get_file slack.delete_file slack.update_file google_calendar.gcal_list_calendars +q117,List Google Calendar events,verb_product,google_calendar.gcal_list_events,1.0,1.0,1.0,1.0000,1.0000,1,1,269,1.0,1,441,1.0,1,753,1.0,google_calendar.gcal_list_events google_calendar.gcal_delete_event google_calendar.gcal_update_event kubernetes.k8s_get_events google_calendar.gcal_create_event google_calendar.gcal_list_calendars sentry.sentry_list_events google_drive.gdrive_search google_drive.gdrive_update_file google_drive.gdrive_get_file +q118,Delete a calendar event,resource,google_calendar.gcal_delete_event,1.0,1.0,1.0,1.0000,1.0000,1,1,300,1.0,1,427,1.0,1,796,1.0,google_calendar.gcal_delete_event google_calendar.gcal_update_event google_calendar.gcal_create_event google_calendar.gcal_list_events google_calendar.gcal_list_calendars github.delete_repository slack.delete_file github.pull_request_review_write linear.delete_comment github.delete_milestone +q119,List Vercel projects,verb_product,vercel.vercel_list_projects,1.0,1.0,1.0,1.0000,1.0000,1,1,175,1.0,1,269,1.0,1,550,1.0,vercel.vercel_list_projects vercel.vercel_create_project vercel.vercel_list_deployments vercel.vercel_get_deployment jira.jira_list_projects github.projects_list gitlab.gitlab_list_projects linear.linear_list_projects sentry.sentry_list_projects github.projects_get +q120,Select rows from Supabase,verb_product,supabase.supabase_select,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,354,1.0,1,815,1.0,supabase.supabase_select supabase.supabase_insert supabase.supabase_update supabase.supabase_list_tables supabase.supabase_invoke_fn github.delete_file github.create_branch gitlab.gitlab_get_file aws_s3.s3_get_object github.get_tag +q121,Post a Figma comment,verb_product,figma.figma_post_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,350,1.0,1,658,1.0,figma.figma_post_comment figma.figma_get_file slack.slack_post_message figma.figma_get_comments figma.figma_get_file_nodes figma.figma_get_images linear.get_comment notion.get_comment linear.delete_comment linear.update_comment +q122,List Okta users,verb_product,okta.okta_list_users,1.0,1.0,1.0,1.0000,1.0000,1,1,177,1.0,1,306,1.0,1,582,1.0,okta.okta_list_users okta.okta_get_user okta.okta_create_user okta.okta_deactivate_user linear.list_users slack.slack_list_users notion.notion_list_users github.search_users slack.slack_get_users notion.get_users_token +q123,Range query Prometheus,verb_product,prometheus.prom_query_range,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,407,1.0,1,679,1.0,prometheus.prom_query_range prometheus.prom_query prometheus.prom_list_metrics notion.notion_query_database datadog.datadog_query_metrics postgres.query sqlite.sqlite_query salesforce.salesforce_query memory.search_nodes github.search_users +q124,Create a Grafana annotation,verb_product,grafana.grafana_create_annotation,1.0,1.0,1.0,1.0000,1.0000,1,1,218,1.0,1,410,1.0,1,900,1.0,grafana.grafana_create_annotation grafana.grafana_search_dashboards grafana.grafana_get_dashboard slack.create_file linear.create_comment github.create_milestone linear.create_user notion.create_comment gitlab.create_label jira.create_filter +q125,Apply terraform,identifier,terraform.tf_apply,1.0,1.0,1.0,1.0000,1.0000,1,1,194,1.0,1,292,1.0,1,698,1.0,terraform.tf_apply kubernetes.k8s_apply terraform.tf_state_list terraform.tf_plan +q126,List Asana tasks,verb_product,asana.asana_list_tasks,1.0,1.0,1.0,1.0000,1.0000,1,1,211,1.0,1,368,1.0,1,708,1.0,asana.asana_list_tasks clickup.clickup_list_tasks asana.asana_create_project asana.asana_get_task asana.asana_create_task asana.asana_update_task jira.jira_list_projects slack.list_files github.list_labels linear.list_users +q127,Comment on a Trello card,verb_product,trello.trello_add_comment,1.0,1.0,1.0,1.0000,1.0000,1,1,241,1.0,1,422,1.0,1,695,1.0,trello.trello_add_comment trello.trello_create_card trello.trello_create_board trello.trello_list_cards github.update_issue_comment linear.get_comment notion.get_comment linear.delete_comment linear.update_comment notion.delete_comment +q128,Update a ClickUp task,verb_product,clickup.clickup_update_task,1.0,1.0,1.0,1.0000,1.0000,1,1,257,1.0,1,399,1.0,1,740,1.0,clickup.clickup_update_task clickup.clickup_get_task clickup.clickup_create_task asana.asana_update_task clickup.clickup_list_tasks asana.asana_get_task playwright.browser_click puppeteer.puppeteer_click asana.asana_create_task slack.update_file +q129,Search Confluence,verb_product,confluence.confluence_search,1.0,1.0,1.0,1.0000,1.0000,1,1,221,1.0,1,362,1.0,1,677,1.0,confluence.confluence_search confluence.confluence_get_page confluence.confluence_update_page confluence.confluence_create_page github.search_users github.search_issues github.search_orgs github.search_commits linear.linear_search_issues slack.slack_search_messages +q130,Create a HubSpot deal,verb_product,hubspot.hubspot_create_deal,1.0,1.0,1.0,1.0000,1.0000,1,1,255,1.0,1,364,1.0,1,899,1.0,hubspot.hubspot_create_deal hubspot.hubspot_get_contact hubspot.hubspot_create_contact hubspot.hubspot_list_contacts hubspot.hubspot_search_contacts github.create_repository github.create_branch github.create_issue github.create_gist github.create_or_update_file +q131,List Zendesk tickets,verb_product,zendesk.zendesk_list_tickets,1.0,1.0,1.0,1.0000,1.0000,1,1,189,1.0,1,367,1.0,1,694,1.0,zendesk.zendesk_list_tickets zendesk.zendesk_search zendesk.zendesk_get_ticket zendesk.zendesk_update_ticket zendesk.zendesk_create_ticket jira.jira_list_projects slack.list_files github.list_labels linear.list_users gitlab.list_labels +q132,List Discord channels,verb_product,discord.discord_list_channels,1.0,1.0,1.0,1.0000,1.0000,1,1,184,1.0,1,300,1.0,1,639,1.0,discord.discord_list_channels slack.slack_list_channels discord.discord_get_channel discord.discord_list_members discord.discord_send_message discord.discord_add_reaction jira.jira_list_projects slack.list_files github.list_labels linear.list_users +q133,Set a Redis key,verb_product,redis.redis_set,1.0,1.0,1.0,1.0000,1.0000,1,1,182,1.0,1,302,1.0,1,595,1.0,redis.redis_set redis.redis_get redis.redis_delete redis.redis_keys slack.slack_set_channel_topic jira.jira_get_issue playwright.browser_press_key jira.jira_update_issue figma.figma_get_file aws_s3.s3_delete_object +q134,Insert MongoDB documents,verb_product,mongodb.mongo_insert,1.0,1.0,1.0,1.0000,1.0000,1,1,256,1.0,1,381,1.0,1,698,1.0,mongodb.mongo_insert mongodb.mongo_update mongodb.mongo_delete mongodb.mongo_find mongodb.mongo_list_collections supabase.supabase_insert +q135,Run a MySQL query,verb_product,mysql.mysql_query,1.0,1.0,1.0,1.0000,1.0000,1,1,179,1.0,1,281,1.0,1,619,1.0,mysql.mysql_query postgres.query sqlite.sqlite_query mysql.mysql_list_tables postgres.explain_query github.get_check_run github.delete_check_run github.update_check_run github.create_check_run salesforce.salesforce_query +q136,Index a document in Elasticsearch,verb_product,elasticsearch.es_index_doc,1.0,1.0,1.0,1.0000,1.0000,1,1,242,1.0,1,344,1.0,1,621,1.0,elasticsearch.es_index_doc elasticsearch.es_get_doc elasticsearch.es_search playwright.browser_tab_close slack.get_file slack.delete_file linear.get_comment slack.update_file github.get_milestone linear.get_user +q137,Get an S3 object,verb_product,aws_s3.s3_get_object,1.0,1.0,1.0,1.0000,1.0000,1,1,249,1.0,1,355,1.0,1,729,1.0,aws_s3.s3_get_object aws_s3.s3_delete_object aws_s3.s3_put_object okta.okta_get_user asana.asana_get_task github.create_issue jira.jira_update_issue asana.asana_create_project okta.okta_create_user slack.get_file +q138,List Cloudflare zones,verb_product,cloudflare.cf_list_zones,1.0,1.0,1.0,1.0000,1.0000,1,1,215,1.0,1,319,1.0,1,685,1.0,cloudflare.cf_list_zones cloudflare.cf_list_dns cloudflare.cf_create_dns cloudflare.cf_purge_cache jira.jira_list_projects slack.list_files github.list_labels linear.list_users gitlab.list_labels linear.list_labels +q139,Create a Stripe invoice,verb_product,stripe.stripe_create_invoice,1.0,1.0,1.0,1.0000,1.0000,1,1,204,1.0,1,319,1.0,1,669,1.0,stripe.stripe_create_invoice stripe.stripe_create_product stripe.stripe_create_customer stripe.stripe_list_customers stripe.stripe_get_customer stripe.stripe_create_refund stripe.stripe_list_invoices stripe.stripe_list_subscriptions stripe.stripe_create_price stripe.stripe_create_payment_intent +q140,List GitHub Actions workflows,verb_product,github.actions_list,1.0,1.0,1.0,1.0000,1.0000,1,1,268,1.0,1,456,1.0,1,812,1.0,github.actions_list github.actions_get github.actions_get_job_logs github.actions_run_trigger github.list_labels github.list_tags github.projects_list git.git_branch github.list_branches github.list_issues +q141,Get GitHub Actions job logs,verb_product,github.actions_get_job_logs,1.0,1.0,1.0,1.0000,1.0000,1,1,237,1.0,1,377,1.0,1,859,1.0,github.actions_get_job_logs gitlab.get_job github.actions_list gitlab.delete_job github.actions_get gitlab.update_job gitlab.create_job kubernetes.k8s_pod_logs github.actions_run_trigger datadog.datadog_list_logs +q142,Create a GitHub repository,verb_product,github.delete_repository,0.0,1.0,1.0,0.5000,0.6309,2,1,268,1.0,1,406,1.0,1,837,1.0,github.delete_repository github.create_repository github.create_branch github.star_repository github.unstar_repository github.create_issue github.list_repository_collaborators github.fork_repository github.list_labels github.list_tags +q143,Checkout a git branch,verb_product,git.git_checkout,1.0,1.0,1.0,1.0000,1.0000,1,1,266,1.0,1,407,1.0,1,1001,1.0,git.git_checkout github.create_branch git.git_create_branch git.git_branch github.update_pull_request_branch gitlab.gitlab_create_merge_request github.delete_file github.list_commits github.create_or_update_file github.push_files +q144,Show git log,identifier,git.git_log,1.0,1.0,1.0,1.0000,1.0000,1,1,194,1.0,1,299,1.0,1,619,1.0,git.git_log git.git_show git.git_status git.git_diff_staged git.git_diff_unstaged git.git_diff git.git_branch git.git_checkout git.git_commit git.git_create_branch +q145,List postgres schemas,verb_product,postgres.list_schemas,1.0,1.0,1.0,1.0000,1.0000,1,1,158,1.0,1,277,1.0,1,604,1.0,postgres.list_schemas postgres.list_tables postgres.query postgres.explain_query postgres.describe_table jira.jira_list_projects slack.list_files github.list_labels linear.list_users gitlab.list_labels +q146,Open nodes in the knowledge graph,verb_product,memory.open_nodes,1.0,1.0,1.0,1.0000,1.0000,1,1,163,1.0,1,260,1.0,1,531,1.0,memory.open_nodes memory.search_nodes memory.read_graph memory.create_entities memory.delete_relations memory.create_relations figma.figma_get_file_nodes memory.add_observations memory.delete_entities playwright.browser_tab_list +q147,Get file metadata from the filesystem,paraphrase,filesystem.get_file_info,1.0,1.0,1.0,1.0000,1.0000,1,1,190,1.0,1,334,1.0,1,769,1.0,filesystem.get_file_info filesystem.read_file google_drive.gdrive_get_file filesystem.read_text_file filesystem.read_media_file slack.get_file gitlab.gitlab_get_file github.delete_file filesystem.write_file github.get_file_contents +q148,Write a new local file,paraphrase,filesystem.write_file,1.0,1.0,1.0,1.0000,1.0000,1,1,234,1.0,1,428,1.0,1,727,1.0,filesystem.write_file filesystem.read_file filesystem.read_text_file filesystem.read_media_file github.issue_write slack.get_file brave_search.brave_local_search playwright.browser_tab_new slack.delete_file slack.update_file +q149,Search GitHub issues,verb_product,github.search_issues,1.0,1.0,1.0,1.0000,1.0000,1,1,263,1.0,1,407,1.0,1,742,1.0,github.search_issues linear.linear_search_issues github.list_issues gitlab.gitlab_list_issues jira.jira_search_issues github.search_users github.search_orgs sentry.sentry_list_issues linear.linear_list_issues github.search_commits +q150,Get the authenticated GitHub user,paraphrase,github.get_teams,0.0,1.0,1.0,0.3333,0.5000,3,1,179,1.0,1,281,1.0,1,606,1.0,github.get_teams github.list_notifications github.get_me linear.get_user linear.delete_user linear.update_user slack.slack_get_user_profile okta.okta_get_user linear.create_user okta.okta_deactivate_user diff --git a/experiments/mcp-tool-search/results/tfidf.json b/experiments/mcp-tool-search/results/tfidf.json new file mode 100644 index 00000000..58cad740 --- /dev/null +++ b/experiments/mcp-tool-search/results/tfidf.json @@ -0,0 +1,6969 @@ +{ + "summary": { + "n": 150, + "index_s": 0.0012128260000281443, + "rss_mb": 0.07421875, + "p50_ms": 0.16994949999116216, + "p95_ms": 0.2750024500073777, + "recall@1": 0.8, + "recall@1_lo": 0.7333333333333333, + "recall@1_hi": 0.86, + "recall@3": 0.9733333333333334, + "recall@3_lo": 0.9466666666666667, + "recall@3_hi": 0.9933333333333333, + "recall@5": 0.9866666666666667, + "recall@5_lo": 0.9666666666666667, + "recall@5_hi": 1.0, + "mrr@10": 0.8797222222222223, + "mrr@10_lo": 0.8402777777777778, + "mrr@10_hi": 0.9172222222222223, + "ndcg@10": 0.9022856279855995, + "ndcg@10_lo": 0.8708873281306948, + "ndcg@10_hi": 0.9323108975923741 + }, + "meta": { + "name": "tfidf", + "needs_model": false, + "needs_api": false, + "notes": "cosine TF-IDF" + }, + "rows": [ + { + "id": "q001", + "query": "Create a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "gitlab.gitlab_create_issue", + "github.create_issue", + "github.add_issue_comment", + "gitlab.gitlab_get_issue", + "github.update_issue_comment", + "github.issue_read", + "github.issue_write", + "jira.jira_get_issue", + "linear.linear_create_comment", + "jira.jira_update_issue" + ], + "hit1": "gitlab.gitlab_create_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6063228553749076, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 349, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 527, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 947, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q002", + "query": "Create a Jira ticket", + "kind": "verb_product", + "gold": [ + "jira.jira_create_issue" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_get_ticket", + "jira.jira_create_issue", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_create_ticket", + "jira.create_filter", + "jira.create_attachment", + "jira.create_board", + "jira.create_sprint", + "jira.create_webhook", + "jira.create_version" + ], + "hit1": "zendesk.zendesk_get_ticket", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 466, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 951, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q003", + "query": "Send a Slack message", + "kind": "verb_product", + "gold": [ + "slack.slack_post_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "gmail.gmail_send_email", + "slack.slack_post_message", + "twilio.twilio_send_sms", + "slack.slack_add_reaction", + "slack.get_file", + "slack.delete_file", + "discord.discord_add_reaction", + "slack.slack_get_users", + "slack.update_file" + ], + "hit1": "discord.discord_send_message", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q004", + "query": "List open pull requests", + "kind": "resource", + "gold": [ + "github.list_pull_requests" + ], + "alts": [ + "github.search_pull_requests" + ], + "ranking": [ + "github.list_pull_requests", + "github.search_pull_requests", + "gitlab.gitlab_list_merge_requests", + "github.create_pull_request", + "bitbucket.bitbucket_list_prs", + "memory.open_nodes", + "playwright.browser_tab_list", + "github.pull_request_read", + "github.merge_pull_request", + "github.update_pull_request" + ], + "hit1": "github.list_pull_requests", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 460, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 883, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q005", + "query": "Open a pull request on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_pull_request" + ], + "alts": [], + "ranking": [ + "github.create_pull_request", + "github.request_copilot_review", + "github.pull_request_read", + "github.merge_pull_request", + "github.update_pull_request", + "github.pull_request_review_write", + "github.update_pull_request_branch", + "github.search_pull_requests", + "github.list_pull_requests", + "memory.open_nodes" + ], + "hit1": "github.create_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 364, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 611, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1040, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q006", + "query": "Run a SQL query against postgres", + "kind": "verb_product", + "gold": [ + "postgres.query" + ], + "alts": [], + "ranking": [ + "postgres.query", + "mysql.mysql_query", + "sqlite.sqlite_query", + "postgres.explain_query", + "salesforce.salesforce_query", + "postgres.list_schemas", + "postgres.list_tables", + "github.get_check_run", + "postgres.describe_table", + "github.delete_check_run" + ], + "hit1": "postgres.query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 295, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 553, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q007", + "query": "Read a local file from disk", + "kind": "paraphrase", + "gold": [ + "filesystem.read_file" + ], + "alts": [ + "filesystem.read_text_file" + ], + "ranking": [ + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "gitlab.gitlab_get_file", + "github.delete_file", + "memory.read_graph", + "github.issue_read", + "slack.get_file", + "github.pull_request_read", + "brave_search.brave_local_search" + ], + "hit1": "filesystem.read_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 223, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 780, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q008", + "query": "What files are in this folder", + "kind": "paraphrase", + "gold": [ + "filesystem.list_directory" + ], + "alts": [ + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree" + ], + "ranking": [ + "google_drive.gdrive_list_folder", + "slack.list_files", + "github.push_files", + "filesystem.search_files", + "github.create_gist", + "google_drive.gdrive_search", + "github.update_gist", + "filesystem.list_directory", + "git.git_add", + "filesystem.directory_tree" + ], + "hit1": "google_drive.gdrive_list_folder", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.125, + "ndcg@10": 0.2938406966301003, + "hit_rank": 8, + "k3_calls": 3, + "k3_tokens": 712, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 1.0, + "k5_calls": 2, + "k5_tokens": 745, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 723, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q009", + "query": "Remember this fact in the knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.add_observations" + ], + "alts": [ + "memory.create_entities" + ], + "ranking": [ + "memory.read_graph", + "memory.create_entities", + "memory.delete_relations", + "memory.create_relations", + "memory.search_nodes", + "memory.open_nodes", + "memory.add_observations", + "memory.delete_entities", + "filesystem.read_file", + "slack.slack_list_channels" + ], + "hit1": "memory.read_graph", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.49320831104542096, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 154, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 262, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 544, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q010", + "query": "Search the web with Brave", + "kind": "verb_product", + "gold": [ + "brave_search.brave_web_search" + ], + "alts": [], + "ranking": [ + "brave_search.brave_web_search", + "brave_search.brave_local_search", + "jira.jira_search_issues", + "github.search_users", + "filesystem.list_directory_with_sizes", + "github.search_issues", + "memory.search_nodes", + "github.update_pull_request_branch", + "github.search_orgs", + "github.search_commits" + ], + "hit1": "brave_search.brave_web_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 710, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q011", + "query": "Fetch the contents of a URL", + "kind": "verb_product", + "gold": [ + "fetch.fetch" + ], + "alts": [], + "ranking": [ + "fetch.fetch", + "filesystem.read_file", + "playwright.browser_navigate", + "github.get_file_contents", + "puppeteer.puppeteer_navigate", + "git.git_add", + "docker.docker_logs", + "git.git_status", + "playwright.browser_tab_new", + "github.get_repository_tree" + ], + "hit1": "fetch.fetch", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 208, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 372, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 707, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q012", + "query": "Show git status", + "kind": "identifier", + "gold": [ + "git.git_status" + ], + "alts": [], + "ranking": [ + "git.git_show", + "git.git_status", + "git.git_diff_staged", + "git.git_log", + "git.git_diff_unstaged", + "git.git_diff", + "git.git_branch", + "sentry.sentry_update_issue", + "slack.list_files", + "linear.list_users" + ], + "hit1": "git.git_show", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q013", + "query": "Commit staged changes", + "kind": "paraphrase", + "gold": [ + "git.git_commit" + ], + "alts": [], + "ranking": [ + "git.git_commit", + "git.git_reset", + "git.git_diff_staged", + "github.get_commit", + "git.git_show", + "git.git_log", + "github.push_files" + ], + "hit1": "git.git_commit", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 327, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 663, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q014", + "query": "Create a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_comment", + "linear.linear_get_issue", + "linear.linear_update_issue", + "linear.linear_create_issue", + "linear.linear_assign_issue", + "gitlab.gitlab_create_issue", + "linear.linear_create_project", + "linear.create_comment", + "linear.create_user", + "github.create_issue" + ], + "hit1": "linear.linear_create_comment", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 574, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 463, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 958, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q015", + "query": "Make a task in my project tracker Linear", + "kind": "paraphrase", + "gold": [ + "linear.linear_create_issue" + ], + "alts": [], + "ranking": [ + "linear.linear_create_project", + "clickup.clickup_get_task", + "clickup.clickup_update_task", + "asana.asana_get_task", + "asana.asana_update_task", + "clickup.clickup_create_task", + "asana.asana_create_task", + "github.projects_get", + "mysql.mysql_list_tables", + "mysql.mysql_query" + ], + "hit1": "linear.linear_create_project", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 0.0, + "mrr@10": 0.0, + "ndcg@10": 0.0, + "hit_rank": 145, + "k3_calls": 49, + "k3_tokens": 12448, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 0.0, + "k3_success3": 0.0, + "k5_calls": 29, + "k5_tokens": 11769, + "k5_found": true, + "k5_success1": 0.0, + "k5_success2": 0.0, + "k5_success3": 0.0, + "k10_calls": 15, + "k10_tokens": 11810, + "k10_found": true, + "k10_success1": 0.0, + "k10_success2": 0.0, + "k10_success3": 0.0 + }, + { + "id": "q016", + "query": "Post a comment on a GitHub issue", + "kind": "verb_product", + "gold": [ + "github.add_issue_comment" + ], + "alts": [], + "ranking": [ + "github.update_issue_comment", + "figma.figma_post_comment", + "github.add_issue_comment", + "slack.slack_post_message", + "linear.linear_create_comment", + "jira.jira_add_comment", + "linear.get_comment", + "notion.get_comment", + "linear.delete_comment", + "linear.update_comment" + ], + "hit1": "github.update_issue_comment", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 307, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 444, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 732, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q017", + "query": "Merge the pull request", + "kind": "ambiguous", + "gold": [ + "github.merge_pull_request" + ], + "alts": [ + "gitlab.gitlab_merge_merge_request" + ], + "ranking": [ + "github.merge_pull_request", + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_get_merge_request", + "github.update_pull_request_branch", + "gitlab.gitlab_create_merge_request", + "github.pull_request_read", + "github.update_pull_request", + "github.request_copilot_review", + "github.create_pull_request", + "github.pull_request_review_write" + ], + "hit1": "github.merge_pull_request", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 495, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1121, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q018", + "query": "Trigger a GitHub Actions workflow", + "kind": "verb_product", + "gold": [ + "github.actions_run_trigger" + ], + "alts": [], + "ranking": [ + "github.actions_run_trigger", + "github.actions_get", + "github.actions_list", + "github.actions_get_job_logs", + "circleci.circleci_trigger_pipeline", + "github.delete_repository", + "github.list_tags", + "github.get_label", + "git.git_branch", + "github.projects_get" + ], + "hit1": "github.actions_run_trigger", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 294, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 791, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q019", + "query": "Get Dependabot alerts", + "kind": "resource", + "gold": [ + "github.list_dependabot_alerts" + ], + "alts": [ + "github.get_dependabot_alert" + ], + "ranking": [ + "github.list_dependabot_alerts", + "github.get_dependabot_alert", + "github.list_secret_scanning_alerts", + "github.list_code_scanning_alerts", + "slack.get_file", + "linear.get_comment", + "github.get_milestone", + "linear.get_user", + "notion.get_comment", + "gitlab.get_label" + ], + "hit1": "github.list_dependabot_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 420, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q020", + "query": "List Sentry issues", + "kind": "verb_product", + "gold": [ + "sentry.sentry_list_issues" + ], + "alts": [], + "ranking": [ + "sentry.sentry_list_issues", + "sentry.sentry_list_projects", + "sentry.sentry_get_issue", + "github.list_issues", + "sentry.sentry_list_events", + "gitlab.gitlab_list_issues", + "sentry.sentry_update_issue", + "github.search_issues", + "linear.linear_search_issues", + "linear.linear_list_issues" + ], + "hit1": "sentry.sentry_list_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 236, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 410, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q021", + "query": "Create a Stripe customer", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_customer" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_customer", + "stripe.stripe_get_customer", + "stripe.stripe_create_invoice", + "stripe.stripe_list_invoices", + "stripe.stripe_list_subscriptions", + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_product", + "stripe.stripe_list_customers", + "stripe.stripe_create_refund", + "stripe.stripe_create_price" + ], + "hit1": "stripe.stripe_create_customer", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 186, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q022", + "query": "Refund a Stripe payment", + "kind": "paraphrase", + "gold": [ + "stripe.stripe_create_refund" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_refund", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_product", + "stripe.stripe_create_customer", + "stripe.stripe_list_customers", + "stripe.stripe_get_customer", + "stripe.stripe_list_invoices", + "stripe.stripe_create_invoice", + "stripe.stripe_list_subscriptions" + ], + "hit1": "stripe.stripe_create_refund", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 370, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q023", + "query": "Send an email with Gmail", + "kind": "verb_product", + "gold": [ + "gmail.gmail_send_email" + ], + "alts": [], + "ranking": [ + "gmail.gmail_send_email", + "gmail.gmail_get_email", + "gmail.gmail_trash_email", + "gmail.gmail_list_labels", + "twilio.twilio_send_sms", + "gmail.gmail_create_draft", + "discord.discord_send_message", + "gmail.gmail_modify_email", + "gmail.gmail_search_emails", + "puppeteer.puppeteer_click" + ], + "hit1": "gmail.gmail_send_email", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 711, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q024", + "query": "Search my Gmail inbox", + "kind": "verb_product", + "gold": [ + "gmail.gmail_search_emails" + ], + "alts": [], + "ranking": [ + "gmail.gmail_search_emails", + "gmail.gmail_get_email", + "gmail.gmail_list_labels", + "gmail.gmail_create_draft", + "gmail.gmail_trash_email", + "gmail.gmail_send_email", + "mysql.mysql_list_tables", + "gmail.gmail_modify_email", + "mysql.mysql_query", + "github.search_users" + ], + "hit1": "gmail.gmail_search_emails", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 178, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 652, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q025", + "query": "Upload a file to Google Drive", + "kind": "paraphrase", + "gold": [ + "google_drive.gdrive_create_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_update_file", + "google_drive.gdrive_get_file", + "slack.slack_upload_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_search", + "google_drive.gdrive_share_file", + "google_drive.gdrive_list_folder", + "aws_s3.s3_put_object", + "slack.get_file", + "slack.delete_file" + ], + "hit1": "google_drive.gdrive_update_file", + "recall@1": 0.0, + "recall@3": 0.0, + "recall@5": 1.0, + "mrr@10": 0.25, + "ndcg@10": 0.43067655807339306, + "hit_rank": 4, + "k3_calls": 2, + "k3_tokens": 523, + "k3_found": true, + "k3_success1": 0.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 415, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 750, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q026", + "query": "Create a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_create_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "github.pull_request_review_write", + "slack.create_file", + "linear.create_comment", + "github.create_milestone", + "linear.create_user" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 962, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q027", + "query": "List Kubernetes pods", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_list_pods" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_list_pods", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_get_events", + "kubernetes.k8s_list_services", + "kubernetes.k8s_delete", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_apply", + "kubernetes.k8s_scale_deployment", + "jira.jira_list_projects" + ], + "hit1": "kubernetes.k8s_list_pods", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q028", + "query": "Get logs from a k8s pod", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_pod_logs" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_get_pod", + "datadog.datadog_list_logs", + "docker.docker_logs", + "github.actions_get_job_logs", + "git.git_log", + "gitlab.gitlab_get_file", + "aws_s3.s3_get_object", + "github.get_tag", + "github.get_file_contents" + ], + "hit1": "kubernetes.k8s_pod_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 299, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 455, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 871, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q029", + "query": "Run a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_run" + ], + "alts": [], + "ranking": [ + "docker.docker_run", + "docker.docker_exec", + "docker.docker_rm", + "docker.docker_stop", + "docker.docker_logs", + "docker.docker_images", + "docker.docker_ps", + "docker.docker_build", + "github.get_check_run", + "github.delete_check_run" + ], + "hit1": "docker.docker_run", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q030", + "query": "Show docker container logs", + "kind": "verb_product", + "gold": [ + "docker.docker_logs" + ], + "alts": [], + "ranking": [ + "docker.docker_logs", + "docker.docker_rm", + "docker.docker_stop", + "docker.docker_exec", + "docker.docker_run", + "kubernetes.k8s_pod_logs", + "git.git_log", + "docker.docker_images", + "git.git_show", + "docker.docker_ps" + ], + "hit1": "docker.docker_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 645, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q031", + "query": "Navigate the browser to a URL", + "kind": "ambiguous", + "gold": [ + "playwright.browser_navigate" + ], + "alts": [ + "puppeteer.puppeteer_navigate" + ], + "ranking": [ + "playwright.browser_navigate", + "puppeteer.puppeteer_navigate", + "playwright.browser_tab_new", + "fetch.fetch", + "playwright.browser_tab_list", + "twilio.twilio_make_call", + "playwright.browser_tab_close", + "playwright.browser_press_key", + "playwright.browser_click", + "playwright.browser_evaluate" + ], + "hit1": "playwright.browser_navigate", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 294, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 596, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q032", + "query": "Take a screenshot of the page", + "kind": "ambiguous", + "gold": [ + "playwright.browser_take_screenshot" + ], + "alts": [ + "puppeteer.puppeteer_screenshot" + ], + "ranking": [ + "playwright.browser_take_screenshot", + "puppeteer.puppeteer_screenshot", + "notion.notion_get_page", + "playwright.browser_snapshot", + "notion.notion_update_page", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "notion.notion_create_page", + "git.git_status", + "filesystem.read_file" + ], + "hit1": "playwright.browser_take_screenshot", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 226, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 678, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q033", + "query": "Create a GitLab merge request", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_create_merge_request" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_merge_merge_request", + "gitlab.gitlab_get_merge_request", + "gitlab.gitlab_create_merge_request", + "github.merge_pull_request", + "gitlab.gitlab_list_merge_requests", + "gitlab.gitlab_create_issue", + "gitlab.gitlab_create_pipeline", + "github.create_pull_request", + "github.request_copilot_review", + "gitlab.gitlab_list_projects" + ], + "hit1": "gitlab.gitlab_merge_merge_request", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 297, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 475, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 939, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q034", + "query": "Search Notion pages", + "kind": "verb_product", + "gold": [ + "notion.notion_search" + ], + "alts": [], + "ranking": [ + "notion.notion_search", + "notion.notion_list_users", + "notion.get_comment", + "notion.delete_comment", + "github.search_users", + "notion.notion_get_page", + "notion.update_comment", + "notion.create_comment", + "notion.notion_create_database", + "github.search_issues" + ], + "hit1": "notion.notion_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 180, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 283, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 672, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q035", + "query": "Create a Notion page", + "kind": "verb_product", + "gold": [ + "notion.notion_create_page" + ], + "alts": [], + "ranking": [ + "notion.notion_get_page", + "notion.notion_update_page", + "notion.notion_create_page", + "notion.notion_get_comments", + "notion.create_comment", + "notion.notion_create_database", + "confluence.confluence_create_page", + "notion.notion_list_users", + "notion.get_comment", + "notion.delete_comment" + ], + "hit1": "notion.notion_get_page", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 703, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q036", + "query": "Query a Notion database", + "kind": "verb_product", + "gold": [ + "notion.notion_query_database" + ], + "alts": [], + "ranking": [ + "notion.notion_query_database", + "notion.notion_create_database", + "notion.notion_create_page", + "notion.notion_search", + "notion.notion_list_users", + "notion.get_comment", + "notion.delete_comment", + "notion.notion_get_page", + "notion.update_comment", + "notion.create_comment" + ], + "hit1": "notion.notion_query_database", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 279, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 709, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q037", + "query": "Create a Zendesk ticket", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_create_ticket" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_create_ticket", + "zendesk.zendesk_search", + "zendesk.zendesk_list_tickets", + "jira.jira_create_issue", + "slack.create_file", + "linear.create_comment", + "github.create_milestone", + "linear.create_user" + ], + "hit1": "zendesk.zendesk_get_ticket", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 875, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q038", + "query": "Create a HubSpot contact", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_contact" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_get_contact", + "hubspot.hubspot_create_contact", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_create_deal", + "hubspot.hubspot_search_contacts", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.create_gist", + "github.create_or_update_file" + ], + "hit1": "hubspot.hubspot_get_contact", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 899, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q039", + "query": "Create a Salesforce lead", + "kind": "verb_product", + "gold": [ + "salesforce.salesforce_create_lead" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_convert_lead", + "salesforce.salesforce_create_lead", + "salesforce.salesforce_create_record", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_update_record", + "salesforce.salesforce_get_record", + "salesforce.salesforce_list_opportunities", + "salesforce.salesforce_query", + "slack.create_file", + "linear.create_comment" + ], + "hit1": "salesforce.salesforce_convert_lead", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 252, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 416, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q040", + "query": "Run a SOQL query", + "kind": "identifier", + "gold": [ + "salesforce.salesforce_query" + ], + "alts": [], + "ranking": [ + "salesforce.salesforce_query", + "github.get_check_run", + "github.delete_check_run", + "github.update_check_run", + "github.create_check_run", + "postgres.query", + "sqlite.sqlite_query", + "mysql.mysql_query", + "docker.docker_run", + "prometheus.prom_query" + ], + "hit1": "salesforce.salesforce_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 359, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q041", + "query": "List Shopify orders", + "kind": "verb_product", + "gold": [ + "shopify.shopify_list_orders" + ], + "alts": [], + "ranking": [ + "shopify.shopify_list_orders", + "shopify.shopify_list_customers", + "shopify.shopify_get_product", + "shopify.shopify_update_product", + "shopify.shopify_list_products", + "shopify.shopify_get_order", + "shopify.shopify_create_product", + "shopify.shopify_create_order", + "jira.jira_list_projects", + "slack.list_files" + ], + "hit1": "shopify.shopify_list_orders", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 193, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 662, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q042", + "query": "Create a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_create_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_oncalls", + "pagerduty.pagerduty_list_incidents", + "slack.create_file", + "linear.create_comment", + "github.create_milestone", + "linear.create_user" + ], + "hit1": "pagerduty.pagerduty_get_incident", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 387, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 857, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q043", + "query": "Who is on call in PagerDuty", + "kind": "paraphrase", + "gold": [ + "pagerduty.pagerduty_list_oncalls" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_list_oncalls", + "pagerduty.pagerduty_get_incident", + "twilio.twilio_make_call", + "pagerduty.pagerduty_list_incidents", + "pagerduty.pagerduty_create_incident", + "slack.slack_create_channel", + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_resolve_incident", + "filesystem.list_allowed_directories", + "github.update_issue_comment" + ], + "hit1": "pagerduty.pagerduty_list_oncalls", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 419, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 772, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q044", + "query": "Query Datadog metrics", + "kind": "verb_product", + "gold": [ + "datadog.datadog_query_metrics" + ], + "alts": [], + "ranking": [ + "datadog.datadog_query_metrics", + "datadog.datadog_list_monitors", + "datadog.datadog_create_monitor", + "datadog.datadog_list_logs", + "datadog.datadog_get_monitor", + "prometheus.prom_list_metrics", + "prometheus.prom_query", + "notion.notion_query_database", + "prometheus.prom_query_range", + "postgres.query" + ], + "hit1": "datadog.datadog_query_metrics", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q045", + "query": "Search Datadog logs", + "kind": "verb_product", + "gold": [ + "datadog.datadog_list_logs" + ], + "alts": [], + "ranking": [ + "datadog.datadog_list_logs", + "datadog.datadog_list_monitors", + "datadog.datadog_get_monitor", + "datadog.datadog_create_monitor", + "datadog.datadog_query_metrics", + "kubernetes.k8s_pod_logs", + "github.actions_get_job_logs", + "docker.docker_logs", + "github.search_users", + "git.git_log" + ], + "hit1": "datadog.datadog_list_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 458, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 824, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q046", + "query": "List S3 buckets", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_list_buckets" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_list_buckets", + "jira.jira_list_projects", + "slack.list_files", + "github.list_labels", + "linear.list_users", + "gitlab.list_labels", + "linear.list_labels", + "github.projects_list", + "gitlab.gitlab_list_projects", + "github.list_branches" + ], + "hit1": "aws_s3.s3_list_buckets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 170, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 312, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 650, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q047", + "query": "Upload an object to S3", + "kind": "paraphrase", + "gold": [ + "aws_s3.s3_put_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_put_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_get_object", + "slack.slack_upload_file", + "github.assign_copilot_to_issue", + "gmail.gmail_send_email", + "slack.slack_add_reaction", + "twilio.twilio_send_sms", + "github.create_issue", + "jira.jira_update_issue" + ], + "hit1": "aws_s3.s3_put_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 250, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 437, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 905, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q048", + "query": "Purge Cloudflare cache", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_purge_cache" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_purge_cache", + "cloudflare.cf_list_zones", + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns" + ], + "hit1": "cloudflare.cf_purge_cache", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 231, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 340, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 746, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q049", + "query": "Create a Cloudflare DNS record", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_create_dns" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_create_dns", + "cloudflare.cf_list_dns", + "salesforce.salesforce_create_record", + "cloudflare.cf_list_zones", + "salesforce.salesforce_delete_record", + "salesforce.salesforce_update_record", + "salesforce.salesforce_get_record", + "cloudflare.cf_purge_cache", + "git.git_commit", + "slack.create_file" + ], + "hit1": "cloudflare.cf_create_dns", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 757, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q050", + "query": "List Vercel deployments", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_deployments" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_deployments", + "kubernetes.k8s_list_deployments", + "vercel.vercel_list_projects", + "vercel.vercel_create_project", + "vercel.vercel_get_deployment", + "jira.jira_list_projects", + "slack.list_files", + "github.list_labels", + "linear.list_users", + "gitlab.list_labels" + ], + "hit1": "vercel.vercel_list_deployments", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 173, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 291, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 618, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q051", + "query": "Insert a row into Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_insert" + ], + "alts": [], + "ranking": [ + "supabase.supabase_insert", + "mongodb.mongo_insert", + "supabase.supabase_list_tables", + "supabase.supabase_update", + "supabase.supabase_invoke_fn", + "supabase.supabase_select", + "playwright.browser_type" + ], + "hit1": "supabase.supabase_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 203, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 665, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q052", + "query": "Export images from Figma", + "kind": "paraphrase", + "gold": [ + "figma.figma_get_images" + ], + "alts": [], + "ranking": [ + "figma.figma_get_images", + "docker.docker_images", + "figma.figma_get_file", + "figma.figma_get_file_nodes", + "figma.figma_get_comments", + "figma.figma_post_comment", + "github.delete_file", + "github.create_branch", + "gitlab.gitlab_get_file", + "aws_s3.s3_get_object" + ], + "hit1": "figma.figma_get_images", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 777, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q053", + "query": "Send an SMS with Twilio", + "kind": "verb_product", + "gold": [ + "twilio.twilio_send_sms" + ], + "alts": [], + "ranking": [ + "twilio.twilio_send_sms", + "twilio.twilio_list_messages", + "gmail.gmail_send_email", + "twilio.twilio_make_call", + "discord.discord_send_message", + "puppeteer.puppeteer_click", + "filesystem.list_directory_with_sizes", + "puppeteer.puppeteer_fill", + "puppeteer.puppeteer_screenshot", + "github.update_pull_request_branch" + ], + "hit1": "twilio.twilio_send_sms", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 409, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 763, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q054", + "query": "Create an Okta user", + "kind": "verb_product", + "gold": [ + "okta.okta_create_user" + ], + "alts": [], + "ranking": [ + "okta.okta_get_user", + "okta.okta_create_user", + "okta.okta_deactivate_user", + "okta.okta_list_users", + "linear.create_user", + "linear.get_user", + "linear.delete_user", + "linear.update_user", + "slack.slack_get_user_profile", + "github.create_issue" + ], + "hit1": "okta.okta_get_user", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 183, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 329, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 683, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q055", + "query": "Instant query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query", + "prometheus.prom_query_range", + "prometheus.prom_list_metrics", + "notion.notion_query_database", + "datadog.datadog_query_metrics", + "postgres.query", + "sqlite.sqlite_query", + "salesforce.salesforce_query", + "memory.search_nodes", + "github.search_users" + ], + "hit1": "prometheus.prom_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 679, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q056", + "query": "Search Grafana dashboards", + "kind": "verb_product", + "gold": [ + "grafana.grafana_search_dashboards" + ], + "alts": [], + "ranking": [ + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "grafana.grafana_create_annotation", + "github.search_users", + "github.search_issues", + "github.search_orgs", + "github.search_commits", + "linear.linear_search_issues", + "slack.slack_search_messages", + "github.search_pull_requests" + ], + "hit1": "grafana.grafana_search_dashboards", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 217, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 635, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q057", + "query": "Trigger a CircleCI pipeline", + "kind": "verb_product", + "gold": [ + "circleci.circleci_trigger_pipeline" + ], + "alts": [], + "ranking": [ + "circleci.circleci_trigger_pipeline", + "circleci.circleci_get_pipeline", + "gitlab.gitlab_create_pipeline", + "gitlab.get_pipeline", + "gitlab.delete_pipeline", + "circleci.circleci_list_pipelines", + "gitlab.update_pipeline", + "gitlab.create_pipeline", + "github.actions_run_trigger", + "hubspot.hubspot_create_deal" + ], + "hit1": "circleci.circleci_trigger_pipeline", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 324, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q058", + "query": "Run terraform plan", + "kind": "identifier", + "gold": [ + "terraform.tf_plan" + ], + "alts": [], + "ranking": [ + "terraform.tf_plan", + "terraform.tf_apply", + "terraform.tf_state_list", + "github.get_check_run", + "github.delete_check_run", + "github.update_check_run", + "github.create_check_run", + "postgres.explain_query", + "docker.docker_run", + "github.actions_get" + ], + "hit1": "terraform.tf_plan", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 172, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 280, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q059", + "query": "Create an Asana task", + "kind": "verb_product", + "gold": [ + "asana.asana_create_task" + ], + "alts": [], + "ranking": [ + "asana.asana_get_task", + "asana.asana_create_task", + "asana.asana_update_task", + "asana.asana_create_project", + "asana.asana_list_tasks", + "clickup.clickup_get_task", + "clickup.clickup_create_task", + "clickup.clickup_update_task", + "github.create_issue", + "okta.okta_create_user" + ], + "hit1": "asana.asana_get_task", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 405, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q060", + "query": "Create a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_create_card" + ], + "alts": [], + "ranking": [ + "trello.trello_create_card", + "trello.trello_add_comment", + "trello.trello_create_board", + "trello.trello_list_cards", + "slack.create_file", + "linear.create_comment", + "github.create_milestone", + "linear.create_user", + "notion.create_comment", + "gitlab.create_label" + ], + "hit1": "trello.trello_create_card", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 413, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 903, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q061", + "query": "Create a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_create_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_create_task", + "clickup.clickup_get_task", + "clickup.clickup_update_task", + "clickup.clickup_list_tasks", + "asana.asana_get_task", + "asana.asana_update_task", + "asana.asana_create_task", + "playwright.browser_click", + "puppeteer.puppeteer_click", + "slack.create_file" + ], + "hit1": "clickup.clickup_create_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 770, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q062", + "query": "Create a Confluence page", + "kind": "verb_product", + "gold": [ + "confluence.confluence_create_page" + ], + "alts": [], + "ranking": [ + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "confluence.confluence_create_page", + "confluence.confluence_search", + "notion.notion_get_page", + "notion.notion_create_page", + "notion.notion_update_page", + "playwright.browser_take_screenshot", + "notion.notion_get_comments", + "github.list_commits" + ], + "hit1": "confluence.confluence_get_page", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 260, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 755, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q063", + "query": "Send a Discord message", + "kind": "verb_product", + "gold": [ + "discord.discord_send_message" + ], + "alts": [], + "ranking": [ + "discord.discord_send_message", + "discord.discord_add_reaction", + "discord.discord_get_channel", + "discord.discord_list_channels", + "discord.discord_list_members", + "gmail.gmail_send_email", + "twilio.twilio_send_sms", + "slack.slack_post_message", + "github.delete_file", + "gmail.gmail_get_email" + ], + "hit1": "discord.discord_send_message", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q064", + "query": "Search Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_search" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_search", + "elasticsearch.es_get_doc", + "elasticsearch.es_index_doc", + "github.search_users", + "github.search_issues", + "github.search_orgs", + "github.search_commits", + "linear.linear_search_issues", + "slack.slack_search_messages", + "github.search_pull_requests" + ], + "hit1": "elasticsearch.es_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 239, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 375, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 657, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q065", + "query": "Get a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_get" + ], + "alts": [], + "ranking": [ + "redis.redis_get", + "redis.redis_set", + "redis.redis_delete", + "redis.redis_keys", + "jira.jira_get_issue", + "playwright.browser_press_key", + "figma.figma_get_file", + "aws_s3.s3_get_object", + "figma.figma_get_comments", + "jira.jira_update_issue" + ], + "hit1": "redis.redis_get", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 579, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q066", + "query": "Find MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_find" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_find", + "mongodb.mongo_update", + "mongodb.mongo_delete", + "mongodb.mongo_insert", + "mongodb.mongo_list_collections" + ], + "hit1": "mongodb.mongo_find", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 258, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 738, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q067", + "query": "List SQLite tables", + "kind": "verb_product", + "gold": [ + "sqlite.sqlite_list_tables" + ], + "alts": [], + "ranking": [ + "sqlite.sqlite_list_tables", + "sqlite.sqlite_query", + "supabase.supabase_list_tables", + "sqlite.sqlite_describe_table", + "postgres.list_tables", + "mysql.mysql_list_tables", + "jira.jira_list_projects", + "slack.list_files", + "github.list_labels", + "linear.list_users" + ], + "hit1": "sqlite.sqlite_list_tables", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 147, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 245, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 543, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q068", + "query": "Describe a postgres table", + "kind": "verb_product", + "gold": [ + "postgres.describe_table" + ], + "alts": [], + "ranking": [ + "postgres.describe_table", + "sqlite.sqlite_describe_table", + "supabase.supabase_update", + "supabase.supabase_select", + "supabase.supabase_insert", + "postgres.list_schemas", + "postgres.list_tables", + "postgres.query", + "postgres.explain_query" + ], + "hit1": "postgres.describe_table", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 349, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 597, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q069", + "query": "github_create_issue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "github.issue_write" + ], + "ranking": [ + "gitlab.gitlab_create_issue", + "github.create_issue", + "gitlab.gitlab_get_issue", + "jira.jira_get_issue", + "github.add_issue_comment", + "linear.linear_create_comment", + "jira.jira_update_issue", + "github.issue_read", + "bitbucket.bitbucket_create_issue", + "github.update_issue_comment" + ], + "hit1": "gitlab.gitlab_create_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.4796249331362629, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 315, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 472, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 900, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q070", + "query": "createIssue", + "kind": "identifier", + "gold": [ + "github.create_issue" + ], + "alts": [ + "jira.jira_create_issue", + "linear.linear_create_issue" + ], + "ranking": [ + "gitlab.gitlab_create_issue", + "github.create_issue", + "gitlab.gitlab_get_issue", + "jira.jira_get_issue", + "linear.linear_create_comment", + "jira.jira_update_issue", + "bitbucket.bitbucket_create_issue", + "github.add_issue_comment", + "sentry.sentry_get_issue", + "github.issue_read" + ], + "hit1": "gitlab.gitlab_create_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.4030302838010049, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 314, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 433, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 864, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q071", + "query": "list-directory", + "kind": "identifier", + "gold": [ + "filesystem.list_directory" + ], + "alts": [], + "ranking": [ + "filesystem.create_directory", + "filesystem.list_directory", + "github.get_file_contents", + "filesystem.get_file_info", + "filesystem.list_directory_with_sizes", + "filesystem.directory_tree", + "jira.jira_list_projects", + "slack.list_files", + "github.list_labels", + "linear.list_users" + ], + "hit1": "filesystem.create_directory", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 348, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 654, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q072", + "query": "sequential thinking next thought", + "kind": "paraphrase", + "gold": [ + "sequential_thinking.sequentialthinking" + ], + "alts": [], + "ranking": [ + "sequential_thinking.sequentialthinking" + ], + "hit1": "sequential_thinking.sequentialthinking", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 382, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 797, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q073", + "query": "What time is it in Tokyo", + "kind": "paraphrase", + "gold": [ + "time.get_current_time" + ], + "alts": [ + "time.convert_time" + ], + "ranking": [ + "time.get_current_time", + "slack.slack_create_channel", + "time.convert_time", + "playwright.browser_wait_for", + "google_calendar.gcal_list_events", + "filesystem.list_allowed_directories", + "jira.jira_add_worklog", + "grafana.grafana_create_annotation", + "slack.get_file", + "slack.delete_file" + ], + "hit1": "time.get_current_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9502344167898356, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 222, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 378, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 699, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q074", + "query": "Convert time between timezones", + "kind": "verb_product", + "gold": [ + "time.convert_time" + ], + "alts": [], + "ranking": [ + "time.convert_time", + "salesforce.salesforce_convert_lead", + "time.get_current_time", + "playwright.browser_wait_for", + "google_calendar.gcal_list_events", + "memory.create_relations", + "git.git_diff", + "jira.jira_add_worklog", + "grafana.grafana_create_annotation" + ], + "hit1": "time.convert_time", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 704, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q075", + "query": "Star a GitHub repo", + "kind": "verb_product", + "gold": [ + "github.star_repository" + ], + "alts": [], + "ranking": [ + "github.star_repository", + "github.delete_repository", + "git.git_branch", + "github.get_label", + "github.list_labels", + "github.list_tags", + "github.list_branches", + "github.search_issues", + "github.get_tag", + "github.delete_file" + ], + "hit1": "github.star_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 201, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 780, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q076", + "query": "Fork a repository", + "kind": "resource", + "gold": [ + "github.fork_repository" + ], + "alts": [], + "ranking": [ + "github.fork_repository", + "github.delete_repository", + "github.create_repository", + "github.star_repository", + "github.unstar_repository", + "github.list_repository_collaborators", + "github.list_labels", + "github.get_repository_tree", + "github.list_branches", + "github.delete_file" + ], + "hit1": "github.fork_repository", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 248, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 386, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 825, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q077", + "query": "Get the latest GitHub release", + "kind": "verb_product", + "gold": [ + "github.get_latest_release" + ], + "alts": [], + "ranking": [ + "github.get_latest_release", + "gitlab.get_release", + "gitlab.delete_release", + "gitlab.update_release", + "gitlab.create_release", + "github.get_release_by_tag", + "github.get_teams", + "github.get_repository_tree", + "github.get_label", + "github.projects_get" + ], + "hit1": "github.get_latest_release", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 201, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q078", + "query": "Create a gist", + "kind": "resource", + "gold": [ + "github.create_gist" + ], + "alts": [], + "ranking": [ + "github.get_gist", + "github.update_gist", + "github.create_gist", + "slack.create_file", + "linear.create_comment", + "github.create_milestone", + "linear.create_user", + "notion.create_comment", + "gitlab.create_label", + "jira.create_filter" + ], + "hit1": "github.get_gist", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 229, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 421, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 911, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q079", + "query": "Search GitHub code", + "kind": "verb_product", + "gold": [ + "github.search_code" + ], + "alts": [], + "ranking": [ + "github.search_code", + "github.list_code_scanning_alerts", + "github.get_code_scanning_alert", + "github.search_users", + "github.search_issues", + "github.search_orgs", + "github.search_commits", + "github.search_pull_requests", + "github.search_repositories", + "hubspot.hubspot_search_contacts" + ], + "hit1": "github.search_code", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 283, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 419, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 725, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q080", + "query": "List GitHub notifications", + "kind": "verb_product", + "gold": [ + "github.list_notifications" + ], + "alts": [], + "ranking": [ + "github.list_notifications", + "github.mark_all_notifications_read", + "github.list_labels", + "github.list_tags", + "github.projects_list", + "git.git_branch", + "github.list_branches", + "github.list_issues", + "github.actions_list", + "github.list_releases" + ], + "hit1": "github.list_notifications", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 192, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 328, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 687, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q081", + "query": "Assign Copilot to an issue", + "kind": "verb_product", + "gold": [ + "github.assign_copilot_to_issue" + ], + "alts": [], + "ranking": [ + "github.assign_copilot_to_issue", + "linear.linear_assign_issue", + "jira.jira_assign_issue", + "github.request_copilot_review", + "jira.jira_update_issue", + "github.add_issue_comment", + "github.create_issue", + "linear.linear_create_comment", + "jira.jira_add_comment", + "gitlab.gitlab_get_issue" + ], + "hit1": "github.assign_copilot_to_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 267, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 434, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 879, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q082", + "query": "Get secret scanning alerts", + "kind": "resource", + "gold": [ + "github.list_secret_scanning_alerts" + ], + "alts": [ + "github.get_secret_scanning_alert" + ], + "ranking": [ + "github.list_secret_scanning_alerts", + "github.get_secret_scanning_alert", + "github.list_code_scanning_alerts", + "github.list_dependabot_alerts", + "github.get_code_scanning_alert", + "slack.get_file", + "linear.get_comment", + "github.get_milestone", + "linear.get_user", + "notion.get_comment" + ], + "hit1": "github.list_secret_scanning_alerts", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 295, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 470, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 733, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q083", + "query": "Create or update a file on GitHub", + "kind": "verb_product", + "gold": [ + "github.create_or_update_file" + ], + "alts": [ + "github.push_files" + ], + "ranking": [ + "github.create_or_update_file", + "slack.update_file", + "slack.create_file", + "github.update_issue_comment", + "github.get_file_contents", + "slack.get_file", + "filesystem.write_file", + "github.delete_file", + "github.projects_write", + "google_drive.gdrive_update_file" + ], + "hit1": "github.create_or_update_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.7601875334318685, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 333, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 545, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 957, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q084", + "query": "List collaborators on a repo", + "kind": "resource", + "gold": [ + "github.list_repository_collaborators" + ], + "alts": [], + "ranking": [ + "github.list_repository_collaborators", + "github.list_commits", + "github.update_issue_comment", + "github.get_discussion_comments", + "pagerduty.pagerduty_list_oncalls", + "github.create_pull_request", + "trello.trello_list_cards", + "github.request_copilot_review", + "notion.notion_get_comments", + "github.assign_copilot_to_issue" + ], + "hit1": "github.list_repository_collaborators", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 307, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 462, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 948, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q085", + "query": "Reply in a Slack thread", + "kind": "verb_product", + "gold": [ + "slack.slack_reply_to_thread" + ], + "alts": [], + "ranking": [ + "slack.slack_reply_to_thread", + "slack.slack_get_thread_replies", + "github.get_notification_details", + "slack.get_file", + "slack.delete_file", + "slack.update_file", + "slack.create_file", + "github.dismiss_notification", + "slack.slack_get_users", + "slack.slack_list_users" + ], + "hit1": "slack.slack_reply_to_thread", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 346, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q086", + "query": "Add an emoji reaction on Slack", + "kind": "verb_product", + "gold": [ + "slack.slack_add_reaction" + ], + "alts": [ + "discord.discord_add_reaction" + ], + "ranking": [ + "slack.slack_add_reaction", + "discord.discord_add_reaction", + "jira.jira_add_comment", + "github.add_issue_comment", + "git.git_add", + "gmail.gmail_modify_email", + "trello.trello_add_comment", + "memory.add_observations", + "jira.jira_add_worklog", + "slack.get_file" + ], + "hit1": "slack.slack_add_reaction", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 438, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 802, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q087", + "query": "Search Slack messages", + "kind": "verb_product", + "gold": [ + "slack.slack_search_messages" + ], + "alts": [], + "ranking": [ + "slack.slack_search_messages", + "gmail.gmail_search_emails", + "twilio.twilio_list_messages", + "slack.slack_get_channel_history", + "github.search_users", + "github.search_issues", + "github.search_orgs", + "slack.get_file", + "github.search_commits", + "linear.linear_search_issues" + ], + "hit1": "slack.slack_search_messages", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 633, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q088", + "query": "Create a Slack channel", + "kind": "verb_product", + "gold": [ + "slack.slack_create_channel" + ], + "alts": [], + "ranking": [ + "slack.slack_create_channel", + "slack.slack_join_channel", + "slack.slack_leave_channel", + "slack.slack_get_channel_history", + "slack.slack_post_message", + "slack.slack_set_channel_topic", + "discord.discord_get_channel", + "slack.slack_upload_file", + "slack.create_file", + "slack.slack_get_thread_replies" + ], + "hit1": "slack.slack_create_channel", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 196, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 339, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 718, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q089", + "query": "Transition a Jira issue", + "kind": "verb_product", + "gold": [ + "jira.jira_transition_issue" + ], + "alts": [], + "ranking": [ + "jira.jira_transition_issue", + "jira.jira_get_issue", + "jira.jira_update_issue", + "jira.jira_assign_issue", + "jira.jira_add_comment", + "jira.jira_create_issue", + "gitlab.gitlab_get_issue", + "jira.jira_add_worklog", + "gitlab.gitlab_create_issue", + "github.add_issue_comment" + ], + "hit1": "jira.jira_transition_issue", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 207, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 342, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 816, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q090", + "query": "Search Jira with JQL", + "kind": "identifier", + "gold": [ + "jira.jira_search_issues" + ], + "alts": [], + "ranking": [ + "jira.jira_search_issues", + "github.search_users", + "filesystem.list_directory_with_sizes", + "github.search_issues", + "jira.jira_list_projects", + "github.search_orgs", + "jira.get_filter", + "github.search_commits", + "linear.linear_search_issues", + "slack.slack_search_messages" + ], + "hit1": "jira.jira_search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 602, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q091", + "query": "Add a worklog to Jira", + "kind": "verb_product", + "gold": [ + "jira.jira_add_worklog" + ], + "alts": [], + "ranking": [ + "jira.jira_add_worklog", + "jira.jira_add_comment", + "github.add_issue_comment", + "git.git_add", + "linear.linear_create_comment", + "memory.add_observations", + "github.add_comment_to_pending_review", + "discord.discord_add_reaction", + "slack.slack_add_reaction", + "jira.jira_list_projects" + ], + "hit1": "jira.jira_add_worklog", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 285, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 839, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q092", + "query": "List Linear teams", + "kind": "verb_product", + "gold": [ + "linear.linear_list_teams" + ], + "alts": [], + "ranking": [ + "linear.linear_list_teams", + "github.get_teams", + "linear.linear_list_projects", + "linear.list_users", + "linear.list_labels", + "linear.get_comment", + "linear.list_comments", + "linear.linear_create_project", + "linear.get_user", + "linear.delete_comment" + ], + "hit1": "linear.linear_list_teams", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 137, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 283, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 591, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q093", + "query": "Comment on a Linear issue", + "kind": "verb_product", + "gold": [ + "linear.linear_create_comment" + ], + "alts": [], + "ranking": [ + "linear.linear_create_comment", + "github.update_issue_comment", + "linear.get_comment", + "linear.delete_comment", + "linear.update_comment", + "linear.create_comment", + "github.add_issue_comment", + "jira.jira_add_comment", + "linear.linear_get_issue", + "linear.linear_update_issue" + ], + "hit1": "linear.linear_create_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 244, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 360, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 798, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q094", + "query": "Move a local file", + "kind": "paraphrase", + "gold": [ + "filesystem.move_file" + ], + "alts": [], + "ranking": [ + "filesystem.move_file", + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "slack.get_file", + "brave_search.brave_local_search", + "slack.delete_file", + "slack.update_file", + "gmail.gmail_trash_email", + "slack.create_file" + ], + "hit1": "filesystem.move_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 224, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 335, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q095", + "query": "Recursively search files by pattern", + "kind": "paraphrase", + "gold": [ + "filesystem.search_files" + ], + "alts": [], + "ranking": [ + "filesystem.search_files", + "redis.redis_keys", + "slack.list_files", + "linear.linear_search_issues", + "google_drive.gdrive_search", + "github.search_repositories", + "github.search_users", + "github.push_files", + "github.search_issues", + "github.search_orgs" + ], + "hit1": "filesystem.search_files", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 212, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 322, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 716, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q096", + "query": "Edit a file in place with old and new string", + "kind": "paraphrase", + "gold": [ + "filesystem.edit_file" + ], + "alts": [], + "ranking": [ + "filesystem.edit_file", + "filesystem.list_directory_with_sizes", + "filesystem.write_file", + "slack.get_file", + "slack.delete_file", + "playwright.browser_tab_new", + "slack.update_file", + "slack.create_file", + "twilio.twilio_make_call", + "github.create_repository" + ], + "hit1": "filesystem.edit_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 247, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 353, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 735, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q097", + "query": "Read the whole knowledge graph", + "kind": "paraphrase", + "gold": [ + "memory.read_graph" + ], + "alts": [], + "ranking": [ + "memory.read_graph", + "memory.create_entities", + "memory.delete_relations", + "memory.create_relations", + "memory.search_nodes", + "filesystem.read_file", + "memory.open_nodes", + "memory.delete_entities", + "memory.add_observations", + "filesystem.read_text_file" + ], + "hit1": "memory.read_graph", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 152, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 260, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 561, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q098", + "query": "Search memory nodes", + "kind": "verb_product", + "gold": [ + "memory.search_nodes" + ], + "alts": [], + "ranking": [ + "memory.search_nodes", + "memory.open_nodes", + "figma.figma_get_file_nodes", + "github.search_users", + "github.search_issues", + "github.search_orgs", + "github.search_commits", + "linear.linear_search_issues", + "slack.slack_search_messages", + "github.search_pull_requests" + ], + "hit1": "memory.search_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 202, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 338, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 620, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q099", + "query": "Show unstaged git diff", + "kind": "verb_product", + "gold": [ + "git.git_diff_unstaged" + ], + "alts": [], + "ranking": [ + "git.git_diff_unstaged", + "git.git_diff", + "git.git_diff_staged", + "git.git_show", + "git.git_status", + "git.git_log", + "git.git_branch", + "git.git_checkout", + "git.git_commit", + "git.git_create_branch" + ], + "hit1": "git.git_diff_unstaged", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 303, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 621, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q100", + "query": "Stage files with git add", + "kind": "identifier", + "gold": [ + "git.git_add" + ], + "alts": [], + "ranking": [ + "git.git_add", + "filesystem.list_directory_with_sizes", + "slack.list_files", + "salesforce.salesforce_list_opportunities", + "github.add_issue_comment", + "jira.jira_add_comment", + "github.push_files", + "memory.add_observations", + "jira.jira_add_worklog", + "filesystem.search_files" + ], + "hit1": "git.git_add", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 220, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 774, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q101", + "query": "Create a Shopify product", + "kind": "verb_product", + "gold": [ + "shopify.shopify_create_product" + ], + "alts": [], + "ranking": [ + "shopify.shopify_get_product", + "shopify.shopify_update_product", + "shopify.shopify_create_product", + "stripe.stripe_create_product", + "shopify.shopify_get_order", + "shopify.shopify_list_customers", + "shopify.shopify_create_order", + "shopify.shopify_list_orders", + "shopify.shopify_list_products", + "stripe.stripe_create_price" + ], + "hit1": "shopify.shopify_get_product", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 240, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 702, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q102", + "query": "Cancel a Stripe subscription", + "kind": "verb_product", + "gold": [ + "stripe.stripe_cancel_subscription" + ], + "alts": [], + "ranking": [ + "stripe.stripe_cancel_subscription", + "stripe.stripe_create_product", + "stripe.stripe_create_customer", + "stripe.stripe_list_customers", + "stripe.stripe_get_customer", + "stripe.stripe_list_invoices", + "stripe.stripe_create_invoice", + "stripe.stripe_list_subscriptions", + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_refund" + ], + "hit1": "stripe.stripe_cancel_subscription", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 206, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 321, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 610, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q103", + "query": "Create a payment intent", + "kind": "resource", + "gold": [ + "stripe.stripe_create_payment_intent" + ], + "alts": [], + "ranking": [ + "stripe.stripe_get_payment_intent", + "stripe.stripe_create_payment_intent", + "stripe.stripe_create_refund", + "slack.create_file", + "linear.create_comment", + "github.create_milestone", + "linear.create_user", + "notion.create_comment", + "gitlab.create_label", + "jira.create_filter" + ], + "hit1": "stripe.stripe_get_payment_intent", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 426, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 916, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q104", + "query": "Acknowledge a PagerDuty incident", + "kind": "verb_product", + "gold": [ + "pagerduty.pagerduty_acknowledge_incident" + ], + "alts": [], + "ranking": [ + "pagerduty.pagerduty_acknowledge_incident", + "pagerduty.pagerduty_get_incident", + "pagerduty.pagerduty_create_incident", + "pagerduty.pagerduty_resolve_incident", + "pagerduty.pagerduty_list_oncalls", + "pagerduty.pagerduty_list_incidents" + ], + "hit1": "pagerduty.pagerduty_acknowledge_incident", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 261, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 389, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 717, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q105", + "query": "Create a Datadog monitor", + "kind": "verb_product", + "gold": [ + "datadog.datadog_create_monitor" + ], + "alts": [], + "ranking": [ + "datadog.datadog_get_monitor", + "datadog.datadog_create_monitor", + "datadog.datadog_list_monitors", + "datadog.datadog_list_logs", + "datadog.datadog_query_metrics", + "slack.create_file", + "linear.create_comment", + "github.create_milestone", + "linear.create_user", + "notion.create_comment" + ], + "hit1": "datadog.datadog_get_monitor", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 259, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 459, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 948, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q106", + "query": "Resolve a Sentry issue", + "kind": "paraphrase", + "gold": [ + "sentry.sentry_update_issue" + ], + "alts": [], + "ranking": [ + "sentry.sentry_get_issue", + "sentry.sentry_update_issue", + "sentry.sentry_list_events", + "pagerduty.pagerduty_resolve_incident", + "sentry.sentry_list_projects", + "sentry.sentry_list_issues", + "gitlab.gitlab_get_issue", + "jira.jira_get_issue", + "jira.jira_update_issue", + "gitlab.gitlab_create_issue" + ], + "hit1": "sentry.sentry_get_issue", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 251, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 745, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q107", + "query": "Scale a Kubernetes deployment", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_scale_deployment" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_scale_deployment", + "vercel.vercel_get_deployment", + "kubernetes.k8s_get_events", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_delete", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_list_services", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_apply" + ], + "hit1": "kubernetes.k8s_scale_deployment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 230, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 374, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 726, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q108", + "query": "Apply a Kubernetes manifest", + "kind": "verb_product", + "gold": [ + "kubernetes.k8s_apply" + ], + "alts": [], + "ranking": [ + "kubernetes.k8s_apply", + "terraform.tf_apply", + "kubernetes.k8s_get_events", + "kubernetes.k8s_list_deployments", + "kubernetes.k8s_delete", + "kubernetes.k8s_get_pod", + "kubernetes.k8s_list_services", + "kubernetes.k8s_list_pods", + "kubernetes.k8s_pod_logs", + "kubernetes.k8s_scale_deployment" + ], + "hit1": "kubernetes.k8s_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 200, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q109", + "query": "Stop a docker container", + "kind": "verb_product", + "gold": [ + "docker.docker_stop" + ], + "alts": [], + "ranking": [ + "docker.docker_stop", + "docker.docker_rm", + "docker.docker_logs", + "docker.docker_exec", + "docker.docker_run", + "docker.docker_images", + "docker.docker_ps", + "docker.docker_build", + "kubernetes.k8s_pod_logs" + ], + "hit1": "docker.docker_stop", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 345, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 641, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q110", + "query": "Build a docker image", + "kind": "verb_product", + "gold": [ + "docker.docker_build" + ], + "alts": [], + "ranking": [ + "docker.docker_build", + "docker.docker_run", + "docker.docker_images", + "docker.docker_ps", + "docker.docker_rm", + "docker.docker_stop", + "docker.docker_logs", + "docker.docker_exec", + "filesystem.read_media_file" + ], + "hit1": "docker.docker_build", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 317, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 601, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q111", + "query": "Click an element in Playwright", + "kind": "verb_product", + "gold": [ + "playwright.browser_click" + ], + "alts": [], + "ranking": [ + "playwright.browser_click", + "puppeteer.puppeteer_click", + "playwright.browser_type", + "playwright.browser_take_screenshot", + "clickup.clickup_create_task", + "playwright.browser_snapshot", + "clickup.clickup_list_tasks", + "clickup.clickup_get_task", + "clickup.clickup_update_task", + "playwright.browser_tab_list" + ], + "hit1": "playwright.browser_click", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 216, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 654, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q112", + "query": "Type text into a browser input", + "kind": "paraphrase", + "gold": [ + "playwright.browser_type" + ], + "alts": [ + "puppeteer.puppeteer_fill" + ], + "ranking": [ + "playwright.browser_type", + "playwright.browser_wait_for", + "github.projects_list", + "puppeteer.puppeteer_fill", + "playwright.browser_tab_list", + "filesystem.read_text_file", + "playwright.browser_navigate", + "playwright.browser_tab_new", + "playwright.browser_tab_close", + "mongodb.mongo_insert" + ], + "hit1": "playwright.browser_type", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 0.9238850086262381, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 232, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 337, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 647, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q113", + "query": "List GitLab projects", + "kind": "verb_product", + "gold": [ + "gitlab.gitlab_list_projects" + ], + "alts": [], + "ranking": [ + "gitlab.gitlab_list_projects", + "github.projects_list", + "jira.jira_list_projects", + "linear.linear_list_projects", + "vercel.vercel_list_projects", + "sentry.sentry_list_projects", + "github.projects_get", + "gitlab.gitlab_list_issues", + "gitlab.gitlab_create_issue", + "gitlab.gitlab_get_issue" + ], + "hit1": "gitlab.gitlab_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 171, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 255, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 609, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q114", + "query": "Create a Bitbucket pull request", + "kind": "verb_product", + "gold": [ + "bitbucket.bitbucket_create_pr" + ], + "alts": [], + "ranking": [ + "bitbucket.bitbucket_create_pr", + "bitbucket.bitbucket_list_prs", + "bitbucket.bitbucket_create_issue", + "github.pull_request_read", + "bitbucket.bitbucket_list_repos", + "github.merge_pull_request", + "github.update_pull_request", + "github.update_pull_request_branch", + "github.create_pull_request", + "github.pull_request_review_write" + ], + "hit1": "bitbucket.bitbucket_create_pr", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 313, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 456, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1091, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q115", + "query": "Create a Gmail draft", + "kind": "verb_product", + "gold": [ + "gmail.gmail_create_draft" + ], + "alts": [], + "ranking": [ + "gmail.gmail_create_draft", + "gmail.gmail_get_email", + "gmail.gmail_list_labels", + "gmail.gmail_search_emails", + "gmail.gmail_trash_email", + "gmail.gmail_send_email", + "github.create_pull_request", + "gmail.gmail_modify_email", + "slack.create_file", + "linear.create_comment" + ], + "hit1": "gmail.gmail_create_draft", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 188, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 315, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 844, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q116", + "query": "Share a Google Drive file", + "kind": "verb_product", + "gold": [ + "google_drive.gdrive_share_file" + ], + "alts": [], + "ranking": [ + "google_drive.gdrive_share_file", + "google_drive.gdrive_update_file", + "google_drive.gdrive_get_file", + "google_drive.gdrive_create_file", + "google_drive.gdrive_search", + "google_drive.gdrive_list_folder", + "slack.get_file", + "slack.delete_file", + "slack.update_file", + "google_calendar.gcal_list_calendars" + ], + "hit1": "google_drive.gdrive_share_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 254, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 417, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 684, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q117", + "query": "List Google Calendar events", + "kind": "verb_product", + "gold": [ + "google_calendar.gcal_list_events" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_list_events", + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "kubernetes.k8s_get_events", + "google_calendar.gcal_create_event", + "google_calendar.gcal_list_calendars", + "sentry.sentry_list_events", + "google_drive.gdrive_search", + "google_drive.gdrive_update_file", + "google_drive.gdrive_get_file" + ], + "hit1": "google_calendar.gcal_list_events", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 269, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 441, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 753, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q118", + "query": "Delete a calendar event", + "kind": "resource", + "gold": [ + "google_calendar.gcal_delete_event" + ], + "alts": [], + "ranking": [ + "google_calendar.gcal_delete_event", + "google_calendar.gcal_update_event", + "google_calendar.gcal_create_event", + "google_calendar.gcal_list_events", + "google_calendar.gcal_list_calendars", + "github.delete_repository", + "slack.delete_file", + "github.pull_request_review_write", + "linear.delete_comment", + "github.delete_milestone" + ], + "hit1": "google_calendar.gcal_delete_event", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 300, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 427, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 796, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q119", + "query": "List Vercel projects", + "kind": "verb_product", + "gold": [ + "vercel.vercel_list_projects" + ], + "alts": [], + "ranking": [ + "vercel.vercel_list_projects", + "vercel.vercel_create_project", + "vercel.vercel_list_deployments", + "vercel.vercel_get_deployment", + "jira.jira_list_projects", + "github.projects_list", + "gitlab.gitlab_list_projects", + "linear.linear_list_projects", + "sentry.sentry_list_projects", + "github.projects_get" + ], + "hit1": "vercel.vercel_list_projects", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 175, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 269, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 550, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q120", + "query": "Select rows from Supabase", + "kind": "verb_product", + "gold": [ + "supabase.supabase_select" + ], + "alts": [], + "ranking": [ + "supabase.supabase_select", + "supabase.supabase_insert", + "supabase.supabase_update", + "supabase.supabase_list_tables", + "supabase.supabase_invoke_fn", + "github.delete_file", + "github.create_branch", + "gitlab.gitlab_get_file", + "aws_s3.s3_get_object", + "github.get_tag" + ], + "hit1": "supabase.supabase_select", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 354, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 815, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q121", + "query": "Post a Figma comment", + "kind": "verb_product", + "gold": [ + "figma.figma_post_comment" + ], + "alts": [], + "ranking": [ + "figma.figma_post_comment", + "figma.figma_get_file", + "slack.slack_post_message", + "figma.figma_get_comments", + "figma.figma_get_file_nodes", + "figma.figma_get_images", + "linear.get_comment", + "notion.get_comment", + "linear.delete_comment", + "linear.update_comment" + ], + "hit1": "figma.figma_post_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 350, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 658, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q122", + "query": "List Okta users", + "kind": "verb_product", + "gold": [ + "okta.okta_list_users" + ], + "alts": [], + "ranking": [ + "okta.okta_list_users", + "okta.okta_get_user", + "okta.okta_create_user", + "okta.okta_deactivate_user", + "linear.list_users", + "slack.slack_list_users", + "notion.notion_list_users", + "github.search_users", + "slack.slack_get_users", + "notion.get_users_token" + ], + "hit1": "okta.okta_list_users", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 177, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 306, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 582, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q123", + "query": "Range query Prometheus", + "kind": "verb_product", + "gold": [ + "prometheus.prom_query_range" + ], + "alts": [], + "ranking": [ + "prometheus.prom_query_range", + "prometheus.prom_query", + "prometheus.prom_list_metrics", + "notion.notion_query_database", + "datadog.datadog_query_metrics", + "postgres.query", + "sqlite.sqlite_query", + "salesforce.salesforce_query", + "memory.search_nodes", + "github.search_users" + ], + "hit1": "prometheus.prom_query_range", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 679, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q124", + "query": "Create a Grafana annotation", + "kind": "verb_product", + "gold": [ + "grafana.grafana_create_annotation" + ], + "alts": [], + "ranking": [ + "grafana.grafana_create_annotation", + "grafana.grafana_search_dashboards", + "grafana.grafana_get_dashboard", + "slack.create_file", + "linear.create_comment", + "github.create_milestone", + "linear.create_user", + "notion.create_comment", + "gitlab.create_label", + "jira.create_filter" + ], + "hit1": "grafana.grafana_create_annotation", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 218, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 410, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 900, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q125", + "query": "Apply terraform", + "kind": "identifier", + "gold": [ + "terraform.tf_apply" + ], + "alts": [], + "ranking": [ + "terraform.tf_apply", + "kubernetes.k8s_apply", + "terraform.tf_state_list", + "terraform.tf_plan" + ], + "hit1": "terraform.tf_apply", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 292, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q126", + "query": "List Asana tasks", + "kind": "verb_product", + "gold": [ + "asana.asana_list_tasks" + ], + "alts": [], + "ranking": [ + "asana.asana_list_tasks", + "clickup.clickup_list_tasks", + "asana.asana_create_project", + "asana.asana_get_task", + "asana.asana_create_task", + "asana.asana_update_task", + "jira.jira_list_projects", + "slack.list_files", + "github.list_labels", + "linear.list_users" + ], + "hit1": "asana.asana_list_tasks", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 211, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 368, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 708, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q127", + "query": "Comment on a Trello card", + "kind": "verb_product", + "gold": [ + "trello.trello_add_comment" + ], + "alts": [], + "ranking": [ + "trello.trello_add_comment", + "trello.trello_create_card", + "trello.trello_create_board", + "trello.trello_list_cards", + "github.update_issue_comment", + "linear.get_comment", + "notion.get_comment", + "linear.delete_comment", + "linear.update_comment", + "notion.delete_comment" + ], + "hit1": "trello.trello_add_comment", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 241, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 422, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 695, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q128", + "query": "Update a ClickUp task", + "kind": "verb_product", + "gold": [ + "clickup.clickup_update_task" + ], + "alts": [], + "ranking": [ + "clickup.clickup_update_task", + "clickup.clickup_get_task", + "clickup.clickup_create_task", + "asana.asana_update_task", + "clickup.clickup_list_tasks", + "asana.asana_get_task", + "playwright.browser_click", + "puppeteer.puppeteer_click", + "asana.asana_create_task", + "slack.update_file" + ], + "hit1": "clickup.clickup_update_task", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 257, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 399, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 740, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q129", + "query": "Search Confluence", + "kind": "verb_product", + "gold": [ + "confluence.confluence_search" + ], + "alts": [], + "ranking": [ + "confluence.confluence_search", + "confluence.confluence_get_page", + "confluence.confluence_update_page", + "confluence.confluence_create_page", + "github.search_users", + "github.search_issues", + "github.search_orgs", + "github.search_commits", + "linear.linear_search_issues", + "slack.slack_search_messages" + ], + "hit1": "confluence.confluence_search", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 221, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 362, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 677, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q130", + "query": "Create a HubSpot deal", + "kind": "verb_product", + "gold": [ + "hubspot.hubspot_create_deal" + ], + "alts": [], + "ranking": [ + "hubspot.hubspot_create_deal", + "hubspot.hubspot_get_contact", + "hubspot.hubspot_create_contact", + "hubspot.hubspot_list_contacts", + "hubspot.hubspot_search_contacts", + "github.create_repository", + "github.create_branch", + "github.create_issue", + "github.create_gist", + "github.create_or_update_file" + ], + "hit1": "hubspot.hubspot_create_deal", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 255, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 364, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 899, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q131", + "query": "List Zendesk tickets", + "kind": "verb_product", + "gold": [ + "zendesk.zendesk_list_tickets" + ], + "alts": [], + "ranking": [ + "zendesk.zendesk_list_tickets", + "zendesk.zendesk_search", + "zendesk.zendesk_get_ticket", + "zendesk.zendesk_update_ticket", + "zendesk.zendesk_create_ticket", + "jira.jira_list_projects", + "slack.list_files", + "github.list_labels", + "linear.list_users", + "gitlab.list_labels" + ], + "hit1": "zendesk.zendesk_list_tickets", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 189, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 367, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 694, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q132", + "query": "List Discord channels", + "kind": "verb_product", + "gold": [ + "discord.discord_list_channels" + ], + "alts": [], + "ranking": [ + "discord.discord_list_channels", + "slack.slack_list_channels", + "discord.discord_get_channel", + "discord.discord_list_members", + "discord.discord_send_message", + "discord.discord_add_reaction", + "jira.jira_list_projects", + "slack.list_files", + "github.list_labels", + "linear.list_users" + ], + "hit1": "discord.discord_list_channels", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 184, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 300, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 639, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q133", + "query": "Set a Redis key", + "kind": "verb_product", + "gold": [ + "redis.redis_set" + ], + "alts": [], + "ranking": [ + "redis.redis_set", + "redis.redis_get", + "redis.redis_delete", + "redis.redis_keys", + "slack.slack_set_channel_topic", + "jira.jira_get_issue", + "playwright.browser_press_key", + "jira.jira_update_issue", + "figma.figma_get_file", + "aws_s3.s3_delete_object" + ], + "hit1": "redis.redis_set", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 182, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 302, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 595, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q134", + "query": "Insert MongoDB documents", + "kind": "verb_product", + "gold": [ + "mongodb.mongo_insert" + ], + "alts": [], + "ranking": [ + "mongodb.mongo_insert", + "mongodb.mongo_update", + "mongodb.mongo_delete", + "mongodb.mongo_find", + "mongodb.mongo_list_collections", + "supabase.supabase_insert" + ], + "hit1": "mongodb.mongo_insert", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 256, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 381, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 698, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q135", + "query": "Run a MySQL query", + "kind": "verb_product", + "gold": [ + "mysql.mysql_query" + ], + "alts": [], + "ranking": [ + "mysql.mysql_query", + "postgres.query", + "sqlite.sqlite_query", + "mysql.mysql_list_tables", + "postgres.explain_query", + "github.get_check_run", + "github.delete_check_run", + "github.update_check_run", + "github.create_check_run", + "salesforce.salesforce_query" + ], + "hit1": "mysql.mysql_query", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 619, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q136", + "query": "Index a document in Elasticsearch", + "kind": "verb_product", + "gold": [ + "elasticsearch.es_index_doc" + ], + "alts": [], + "ranking": [ + "elasticsearch.es_index_doc", + "elasticsearch.es_get_doc", + "elasticsearch.es_search", + "playwright.browser_tab_close", + "slack.get_file", + "slack.delete_file", + "linear.get_comment", + "slack.update_file", + "github.get_milestone", + "linear.get_user" + ], + "hit1": "elasticsearch.es_index_doc", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 242, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 344, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 621, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q137", + "query": "Get an S3 object", + "kind": "verb_product", + "gold": [ + "aws_s3.s3_get_object" + ], + "alts": [], + "ranking": [ + "aws_s3.s3_get_object", + "aws_s3.s3_delete_object", + "aws_s3.s3_put_object", + "okta.okta_get_user", + "asana.asana_get_task", + "github.create_issue", + "jira.jira_update_issue", + "asana.asana_create_project", + "okta.okta_create_user", + "slack.get_file" + ], + "hit1": "aws_s3.s3_get_object", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 249, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 355, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 729, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q138", + "query": "List Cloudflare zones", + "kind": "verb_product", + "gold": [ + "cloudflare.cf_list_zones" + ], + "alts": [], + "ranking": [ + "cloudflare.cf_list_zones", + "cloudflare.cf_list_dns", + "cloudflare.cf_create_dns", + "cloudflare.cf_purge_cache", + "jira.jira_list_projects", + "slack.list_files", + "github.list_labels", + "linear.list_users", + "gitlab.list_labels", + "linear.list_labels" + ], + "hit1": "cloudflare.cf_list_zones", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 215, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 685, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q139", + "query": "Create a Stripe invoice", + "kind": "verb_product", + "gold": [ + "stripe.stripe_create_invoice" + ], + "alts": [], + "ranking": [ + "stripe.stripe_create_invoice", + "stripe.stripe_create_product", + "stripe.stripe_create_customer", + "stripe.stripe_list_customers", + "stripe.stripe_get_customer", + "stripe.stripe_create_refund", + "stripe.stripe_list_invoices", + "stripe.stripe_list_subscriptions", + "stripe.stripe_create_price", + "stripe.stripe_create_payment_intent" + ], + "hit1": "stripe.stripe_create_invoice", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 204, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 319, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 669, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q140", + "query": "List GitHub Actions workflows", + "kind": "verb_product", + "gold": [ + "github.actions_list" + ], + "alts": [], + "ranking": [ + "github.actions_list", + "github.actions_get", + "github.actions_get_job_logs", + "github.actions_run_trigger", + "github.list_labels", + "github.list_tags", + "github.projects_list", + "git.git_branch", + "github.list_branches", + "github.list_issues" + ], + "hit1": "github.actions_list", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 456, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 812, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q141", + "query": "Get GitHub Actions job logs", + "kind": "verb_product", + "gold": [ + "github.actions_get_job_logs" + ], + "alts": [], + "ranking": [ + "github.actions_get_job_logs", + "gitlab.get_job", + "github.actions_list", + "gitlab.delete_job", + "github.actions_get", + "gitlab.update_job", + "gitlab.create_job", + "kubernetes.k8s_pod_logs", + "github.actions_run_trigger", + "datadog.datadog_list_logs" + ], + "hit1": "github.actions_get_job_logs", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 237, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 377, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 859, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q142", + "query": "Create a GitHub repository", + "kind": "verb_product", + "gold": [ + "github.create_repository" + ], + "alts": [], + "ranking": [ + "github.delete_repository", + "github.create_repository", + "github.create_branch", + "github.star_repository", + "github.unstar_repository", + "github.create_issue", + "github.list_repository_collaborators", + "github.fork_repository", + "github.list_labels", + "github.list_tags" + ], + "hit1": "github.delete_repository", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.5, + "ndcg@10": 0.6309297535714575, + "hit_rank": 2, + "k3_calls": 1, + "k3_tokens": 268, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 406, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 837, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q143", + "query": "Checkout a git branch", + "kind": "verb_product", + "gold": [ + "git.git_checkout" + ], + "alts": [], + "ranking": [ + "git.git_checkout", + "github.create_branch", + "git.git_create_branch", + "git.git_branch", + "github.update_pull_request_branch", + "gitlab.gitlab_create_merge_request", + "github.delete_file", + "github.list_commits", + "github.create_or_update_file", + "github.push_files" + ], + "hit1": "git.git_checkout", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 266, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 1001, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q144", + "query": "Show git log", + "kind": "identifier", + "gold": [ + "git.git_log" + ], + "alts": [], + "ranking": [ + "git.git_log", + "git.git_show", + "git.git_status", + "git.git_diff_staged", + "git.git_diff_unstaged", + "git.git_diff", + "git.git_branch", + "git.git_checkout", + "git.git_commit", + "git.git_create_branch" + ], + "hit1": "git.git_log", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 194, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 299, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 619, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q145", + "query": "List postgres schemas", + "kind": "verb_product", + "gold": [ + "postgres.list_schemas" + ], + "alts": [], + "ranking": [ + "postgres.list_schemas", + "postgres.list_tables", + "postgres.query", + "postgres.explain_query", + "postgres.describe_table", + "jira.jira_list_projects", + "slack.list_files", + "github.list_labels", + "linear.list_users", + "gitlab.list_labels" + ], + "hit1": "postgres.list_schemas", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 158, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 277, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 604, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q146", + "query": "Open nodes in the knowledge graph", + "kind": "verb_product", + "gold": [ + "memory.open_nodes" + ], + "alts": [], + "ranking": [ + "memory.open_nodes", + "memory.search_nodes", + "memory.read_graph", + "memory.create_entities", + "memory.delete_relations", + "memory.create_relations", + "figma.figma_get_file_nodes", + "memory.add_observations", + "memory.delete_entities", + "playwright.browser_tab_list" + ], + "hit1": "memory.open_nodes", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 163, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 260, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 531, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q147", + "query": "Get file metadata from the filesystem", + "kind": "paraphrase", + "gold": [ + "filesystem.get_file_info" + ], + "alts": [], + "ranking": [ + "filesystem.get_file_info", + "filesystem.read_file", + "google_drive.gdrive_get_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "slack.get_file", + "gitlab.gitlab_get_file", + "github.delete_file", + "filesystem.write_file", + "github.get_file_contents" + ], + "hit1": "filesystem.get_file_info", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 190, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 334, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 769, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q148", + "query": "Write a new local file", + "kind": "paraphrase", + "gold": [ + "filesystem.write_file" + ], + "alts": [], + "ranking": [ + "filesystem.write_file", + "filesystem.read_file", + "filesystem.read_text_file", + "filesystem.read_media_file", + "github.issue_write", + "slack.get_file", + "brave_search.brave_local_search", + "playwright.browser_tab_new", + "slack.delete_file", + "slack.update_file" + ], + "hit1": "filesystem.write_file", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 234, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 428, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 727, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q149", + "query": "Search GitHub issues", + "kind": "verb_product", + "gold": [ + "github.search_issues" + ], + "alts": [], + "ranking": [ + "github.search_issues", + "linear.linear_search_issues", + "github.list_issues", + "gitlab.gitlab_list_issues", + "jira.jira_search_issues", + "github.search_users", + "github.search_orgs", + "sentry.sentry_list_issues", + "linear.linear_list_issues", + "github.search_commits" + ], + "hit1": "github.search_issues", + "recall@1": 1.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 1.0, + "ndcg@10": 1.0, + "hit_rank": 1, + "k3_calls": 1, + "k3_tokens": 263, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 407, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 742, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + }, + { + "id": "q150", + "query": "Get the authenticated GitHub user", + "kind": "paraphrase", + "gold": [ + "github.get_me" + ], + "alts": [], + "ranking": [ + "github.get_teams", + "github.list_notifications", + "github.get_me", + "linear.get_user", + "linear.delete_user", + "linear.update_user", + "slack.slack_get_user_profile", + "okta.okta_get_user", + "linear.create_user", + "okta.okta_deactivate_user" + ], + "hit1": "github.get_teams", + "recall@1": 0.0, + "recall@3": 1.0, + "recall@5": 1.0, + "mrr@10": 0.3333333333333333, + "ndcg@10": 0.5, + "hit_rank": 3, + "k3_calls": 1, + "k3_tokens": 179, + "k3_found": true, + "k3_success1": 1.0, + "k3_success2": 1.0, + "k3_success3": 1.0, + "k5_calls": 1, + "k5_tokens": 281, + "k5_found": true, + "k5_success1": 1.0, + "k5_success2": 1.0, + "k5_success3": 1.0, + "k10_calls": 1, + "k10_tokens": 606, + "k10_found": true, + "k10_success1": 1.0, + "k10_success2": 1.0, + "k10_success3": 1.0 + } + ], + "cost": { + "3": { + "k": 3, + "n": 150, + "n_found": 150, + "mean_calls": 1.3466666666666667, + "p95_calls": 1.0, + "success_within_1": 0.9733333333333334, + "success_within_2": 0.9866666666666667, + "success_within_3": 0.9933333333333333, + "mean_tokens": 318.52666666666664, + "p95_tokens": 314.54999999999995, + "total_tokens": 47779, + "tokens_per_successful_query": 318.52666666666664, + "total_tokens_successful": 47779 + }, + "5": { + "k": 5, + "n": 150, + "n_found": 150, + "mean_calls": 1.1933333333333334, + "p95_calls": 1.0, + "success_within_1": 0.9866666666666667, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 449.36, + "p95_tokens": 471.09999999999997, + "total_tokens": 67404, + "tokens_per_successful_query": 449.36, + "total_tokens_successful": 67404 + }, + "10": { + "k": 10, + "n": 150, + "n_found": 150, + "mean_calls": 1.0933333333333333, + "p95_calls": 1.0, + "success_within_1": 0.9933333333333333, + "success_within_2": 0.9933333333333333, + "success_within_3": 0.9933333333333333, + "mean_tokens": 812.9933333333333, + "p95_tokens": 954.2999999999998, + "total_tokens": 121949, + "tokens_per_successful_query": 812.9933333333333, + "total_tokens_successful": 121949 + } + } +} diff --git a/experiments/mcp-tool-search/run.sh b/experiments/mcp-tool-search/run.sh new file mode 100755 index 00000000..9ca9f31e --- /dev/null +++ b/experiments/mcp-tool-search/run.sh @@ -0,0 +1,32 @@ +#!/usr/bin/env bash +# One-command reproduction of the MCP tool-search experiment. +set -euo pipefail +ROOT="$(cd "$(dirname "$0")" && pwd)" +cd "$ROOT" + +PYTHON="${PYTHON:-python3}" +if [[ ! -d .venv ]]; then + "$PYTHON" -m venv .venv +fi +# shellcheck disable=SC1091 +source .venv/bin/activate +python -m pip install -q -U pip +python -m pip install -q -r requirements.txt || true +# Dense embeddings are optional; the harness falls back if this fails. +python -c "import fastembed" 2>/dev/null || python -m pip install -q 'fastembed>=0.4' || true + +CODEDB_BIN="${CODEDB_BIN:-$ROOT/bin/codedb}" +if [[ ! -x "$CODEDB_BIN" ]]; then + mkdir -p "$ROOT/bin" + VER="${CODEDB_VERSION:-v0.2.5860}" + URL="https://github.com/justrach/codedb/releases/download/${VER}/codedb-linux-x86_64" + echo "downloading codedb ${VER}…" + curl -fsSL -o "$CODEDB_BIN" "$URL" + chmod +x "$CODEDB_BIN" +fi + +python "$ROOT/lib/catalog.py" +python "$ROOT/harness.py" --codedb-bin "$CODEDB_BIN" "$@" +python "$ROOT/cost_eval.py" +python "$ROOT/live_agent.py" +echo "done. see results.md, results/cost.json, results/live_agent.json"